Performing a Rolling Upgrade on your Cluster
Cloudera Manager's rolling upgrade feature takes advantage of parcels and the HDFS High Availability configuration to enable you to upgrade your cluster software and restart the upgraded services, without taking the entire cluster down.
You must have HDFS High Availability enabled to perform a Rolling Upgrade.
A rolling upgrade involves two steps:
- Download, distribute, and activate the parcel for the new software you want to install.
- Perform a rolling restart to restart the services in your cluster. Note that you can do a rolling restart of individual services, or if you have High Availability enabled, you can perform a restart of the entire cluster. Cloudera Manager will manually fail over your NameNode at the appropriate point in the process so that your cluster will not be without a functional NameNode.
To avoid lots of alerts during the upgrade process, you can enable Maintenance Mode on your cluster before you start the upgrade. You can put the entire cluster into Maintenance Mode using the Enter Maintenance Mode command from the Actions menu on the All Services page. Be sure to Exit Maintenance Mode when you have finished the upgrade in order to re-enable Cloudera Manager alerts.
- Step 1. Ensure High Availability is enabled.
- Step 2. Download, Distribute, and Activate the parcels with the new software
- Step 3. Upgrade your Hive Metastore
- Step 4. (If Upgrading to CDH 4.2) Upgrade the Oozie Sharelib
- Step 5. Restart your Cluster
- Step 6. Remove the previous CDH version packages.
- Step 7. Force an update of the symlinks to point to the newly installed components.
Step 1. Ensure High Availability is enabled.
To enable High Availability see Configuring HDFS High Availability for instructions. You do not need to enable Automatic Failover for rolling restart to work, though you can enable it if you wish. Automatic Failover does not affect the rolling restart operation. If you have JobTracker High Availability configured, Cloudera Manager will also failover the JobTracker during the rolling restart, but this is not a requirement for performing a rolling upgrade.
Step 2. Download, Distribute, and Activate the parcels with the new software
- From the All Hosts page, click the Parcels tab, and download, distribute and activate the parcel(s) you want to
upgrade to. For detailed instructions see Upgrading Using
Parcels. When the parcel has been activated, it is not yet running, but is staged
to become the running version the next time the service is restarted. Note:
If you are doing a major version upgrade (e.g. from CDH3 to CDH4 using parcels), after the distribution phase the button will be labeled Upgrade rather than Activate. In this case, follow the instructions at Upgrading CDH3 to CDH4 in a Cloudera Managed Deployment.
- You are asked if you want to restart the cluster. DO NOT RESTART THE CLUSTER AT THIS TIME.
- Click Close. There are several additional steps you must perform prior to restarting your services.
Step 3. Upgrade your Hive Metastore
If you are upgrading from CDH4.2 to CDH 4.3, you do not need to perform this step. If you are upgrading from an earlier version of CDH to either 4.2 or 4.3, you do need to do this.
- (Strongly recommended) Make a backup copy of your Hive metastore database.
- Follow the instructions at Step 4. "Upgrading Your Metastore" in the
Hive Installation section in CDH4 Installation to run the
metastore upgrade script.
- The upgrade script is located at /opt/cloudera/parcels/<parcel_name>/lib/hive/scripts/metastore/upgrade/<database>. <parcel_name> should be the name of the parcel to which you have upgraded. <database> is the type of database you are running (i.e. mysql, postgres, etc.) For example, if you are installing a CDH 4.2.0 parcel using the default location for the local repository, and using the default database (PostgreSQL) the script will be at: /opt/cloudera/parcels/CDH-4.2.0-1.cdh4.2.0.p0.10-e16.parcel/lib/hive/scripts/metastore/upgrade/postgres
- You must cd to the directory the scripts are in.
- Execute the script in the appropriate DB command shell.
You must know the password for the database; if you installed Cloudera Manager using the default (embedded PostgreSQL) database, the password was displayed on the Database Setup page during the Cloudera Manager installation wizard.
If you do not know the password for your Hive metastore database, you can find it as follows:
- cat /etc/cloudera-scm-server/db.properties This shows you Cloudera Manager's internal database credentials.
- Run the following
psql -p 7432 -U scm scm -c "select s.display_name as hive_service_name, s.name as hive_internal_name, c.value as metastore_password from configs c, services s where attr='hive_metastore_database_password' and c.service_id = s.service_id"
- Use the password shown in the com.cloudera.cmf.db.password property.
- This script will output the passwords for the hive service
hive_service_name | hive_internal_name | metastore_password -------------------+--------------------+-------------------- hive1 | hive1 | lF3Cv2zsvI (1 row)
- For the embedded PostgreSQL database, the database name and user name are both hive.
- If you have multiple instances of Hive, run the upgrade script on each metastore database.
Step 4. (If Upgrading to CDH 4.2) Upgrade the Oozie Sharelib
- In the Cloudera Manager Admin Console, select Oozie from the Services tab. The service should already be stopped.
- From the Actions button choose Install Oozie Sharelib. The commands to perform this function are run.
Step 5. Restart your Cluster
- From the Services tab, select All Services to go to the main Services page.
- From the Actions menu for the Cluster, click Rolling Restart... to proceed with a Rolling Restart. Rolling Restart is available only if High Availability is enabled. Click Restart to perform a normal restart. Services that do not support Rolling Restart will undergo a normal restart, and will not be available during the restart process.
- For a Rolling Restart, a pop-up allows you to chose which services
you want to restart, and presents caveats to be aware of for those services that can
undergo a rolling restart.Note: If you have just upgraded your Cloudera Manager deployment to version 4.6, and are now doing a rolling upgrade of your cluster, you need to ensure that MapReduce is restarted BEFORE the rest of your services, or the restart may fail. This is necessary to ensure the MR configuration changes have taken effect prior to starting your other services.
Further, if you are upgrading from CDH4.1 with Impala to CDH4.2 or 4.3, you must restart MapReduce before Impala restarts (by default Impala is restarted before MapReduce).
The workaround is to perform a restart of MapReduce alone as the first step, then perform a cluster restart of the remaining services.
- For HBase, HDFS, and MapReduce you can configure:
- How many hosts should be included in a batch (the default is one, so individual roles will be started on a single host at a time).
- How long Cloudera Manager should wait before starting the next batch.
- The number of batch failures that will cause the entire rolling restart to fail (this is an advanced feature). Please see the Rolling Restart topic for more information about these choices.
- For HBase, HDFS, and MapReduce you can configure:
- Click Confirm to start the rolling restart.
Step 6. Remove the previous CDH version packages.
If your previous installation of CDH 4 (4.0.x or 4.1.x) was done using packages, you must remove those packages and refresh the symlinks so that clients will run the new software versions.
- Uninstall the CDH and Cloudera Manager packages:
Operating System Command
$ sudo yum remove hadoop hue oozie hbase sqoop pig flume hive impala zookeeper bigtop
$ sudo zypper remove hadoop hue oozie hbase sqoop pig flume hive impala zookeeper bigtop
Ubuntu or Debian
$ sudo apt-get remove hadoop hue oozie hbase sqoop pig flume hive impala zookeeper bigtop
Step 7. Force an update of the symlinks to point to the newly installed components.
There are two ways you can do this:
- Restart all the Cloudera Manager agents, OR
- Disable and re-enable the Create symlinks property through the Cloudera Manager Admin console.
Restarting the agents re-triggers symlink creation at agent start up. Unsetting/resetting the symlink property can be done via the Admin console, but it requires waiting at each step until all the agents have synced (at least one heartbeat -- typically harder to verify) for this to be effective.
Do one of the following:
To restart the Cloudera Manager agents:
On each host:
$ sudo service cloudera-scm-agent restart
To force a re-sync via the Admin Console:
- In the Cloudera Manager Admin Console, from the tab, select .
- In the Parcels section, uncheck the property Create System-Wide Symlinks for Active Parcels and Save your change.
- Now, wait for at least one heartbeat (typically less than 30 seconds), then restore the check to the Create System-Wide Symlinks for Active Parcels property and Save your change.
To Restart Selected Services
- From the Services tab, select the service you want to restart, to go to that Service's Status page.
- From the Actions menu, select Rolling Restart... or Restart (not all services support rolling Restart).
For details of doing a rolling restart, see Rolling Restart.
|<< Previous: Upgrading CDH in a Cloudera Manager Deployment||Next: Using Parcels >>|