This is the documentation for Cloudera Manager 5.1.0.
Documentation for other versions is available at Cloudera Documentation.

Performing a Rolling Upgrade on a Cluster

Required Role:

  Important: This feature is available only with a Cloudera Enterprise license.
For other licenses, the following applies:
  • Cloudera Express - the feature is not available.
  • Cloudera Enterprise Data Hub Edition Trial - the feature will not be available after you end the trial or the trial license expires.
To obtain a license for Cloudera Enterprise, please fill in this form or call 866-843-7207. After you install a Cloudera Enterprise license, the feature will be available.

Cloudera Manager's rolling upgrade feature takes advantage of parcels and the HDFS High Availability configuration to enable you to upgrade your cluster software and restart the upgraded services, without taking the entire cluster down. You must have HDFS High Availability enabled to perform a rolling upgrade.

You can perform a rolling upgrade between minor versions of CDH 4, or between minor versions of CDH 5, except Beta versions, as they become available. It is not possible to perform a rolling upgrade from CDH 4 to CDH 5 because of incompatibilities between the two major versions. Instead, follow the instructions for a full upgrade at Upgrading from CDH 4 to CDH 5 Parcels.

A rolling upgrade involves two steps:
  1. Download, distribute, and activate the parcel for the new software you want to install.
  2. Perform a rolling restart to restart the services in your cluster. You can do a rolling restart of individual services, or if you have High Availability enabled, you can perform a restart of the entire cluster. Cloudera Manager will manually fail over your NameNode at the appropriate point in the process so that your cluster will not be without a functional NameNode.
To avoid lots of alerts during the upgrade process, you can enable maintenance mode on your cluster before you start the upgrade. This will stop email alerts and SNMP traps from being sent, but will not stop checks and configuration validations from being made. Be sure to exit maintenance mode when you have finished the upgrade in order to re-enable Cloudera Manager alerts.

The steps to perform a rolling upgrade of a cluster are as follows.

Step 1. Ensure High Availability is Enabled

To enable High Availability see HDFS High Availability for instructions. You do not need to enable Automatic Failover for rolling restart to work, though you can enable it if you wish. Automatic Failover does not affect the rolling restart operation. If you have JobTracker High Availability configured, Cloudera Manager will fail over the JobTracker during the rolling restart, but this is not a requirement for performing a rolling upgrade.

Step 2. Download, Distribute, and Activate the Parcels with the New Software

  1. From the Hosts page, click the Parcels tab, and download, distribute and activate the parcel(s) you want to upgrade to. For detailed instructions see Parcels. When the parcel has been activated, it is not yet running, but is staged to become the running version the next time the service is restarted.
      Note: If you are doing a major version upgrade (that is, from CDH 4 to CDH 5 using parcels), after the distribution phase the button will be labeled Upgrade rather than Activate. In this case, follow the instructions at Upgrading from CDH 4 to CDH 5 Parcels.
  2. You are asked if you want to restart the cluster. Do not restart the cluster at this time.
  • Click Close. There are several additional steps you must perform prior to restarting your services.

Step 3. Upgrade the Hive Metastore

  Note: If you are upgrading from CDH 4.2 to CDH 4.3, or from CDH 4 to CDH 5, you do not need to perform this step. If you are upgrading from an earlier version of CDH to either 4.2 or 4.3, you do need to do this.
  1. (Strongly recommended) Make a backup copy of your Hive metastore database.
  2. Follow the instructions at Step 4. "Upgrading Your Metastore" in the Hive Installation section in CDH 4 Installation to run the metastore upgrade script.
    • The upgrade script is located at /opt/cloudera/parcels/<parcel_name>/lib/hive/scripts/metastore/upgrade/<database>. <parcel_name> should be the name of the parcel to which you have upgraded. <database> is the type of database you are running (that is, MySQL, PostgreSQL, and so on) For example, if you are installing a CDH 4.2.0 parcel using the default location for the local repository, and using the default database (PostgreSQL) the script will be at: /opt/cloudera/parcels/CDH-4.2.0-1.cdh4.2.0.p0.10-e16.parcel/lib/hive/scripts/metastore/upgrade/postgres
    • You must cd to the directory the scripts are in.
    • Execute the script in the appropriate DB command shell.
      Important: You must know the password for the database; if you installed Cloudera Manager using the default (embedded PostgreSQL) database, the password was displayed on the Database Setup page during the Cloudera Manager installation wizard. If you do not know the password for your Hive metastore database, you can find it as follows:
    • cat /etc/cloudera-scm-server/db.properties This shows you Cloudera Manager's internal database credentials.
    • Run the following command:
      psql -p 7432 -U scm scm -c "select s.display_name as hive_service_name, s.name as hive_internal_name, c.value as metastore_password from configs c, services s 
      where attr='hive_metastore_database_password' and c.service_id = s.service_id"
    • Use the password shown in the com.cloudera.cmf.db.password property.
    • This script will output the passwords for the hive service metastore as follows:
       hive_service_name | hive_internal_name | metastore_password
      -------------------+--------------------+--------------------
       hive1             | hive1              | lF3Cv2zsvI
      (1 row)
    • For the embedded PostgreSQL database, the database name and user name are both hive.
  3. If you have multiple instances of Hive, run the upgrade script on each metastore database.

Step 4. (If Upgrading to CDH 4.2) Upgrade the Oozie Sharelib

  1. In the Cloudera Manager Admin Console, go to the Oozie service. The service should already be stopped.
  2. From the Actions button choose Install Oozie Sharelib. The commands to perform this function are run.

Step 5. Restart the Cluster

  1. On the Home page, click to the right of the cluster name and select Rolling Restart to proceed with a Rolling Restart. Rolling restart is available only if High Availability is enabled. Click Restart to perform a normal restart. Services that do not support Rolling Restart will undergo a normal restart, and will not be available during the restart process.
  2. For a rolling restart, a pop-up allows you to chose which services you want to restart, and presents caveats to be aware of for those services that can undergo a rolling restart.
      Note: If you have just upgraded your Cloudera Manager deployment to 4.6, and are now doing a rolling upgrade of your cluster, you need to ensure that MapReduce is restarted before the rest of your services, or the restart may fail. This is necessary to ensure the MapReduce configuration changes are propagated.

    Further, if you are upgrading from CDH 4.1 with Impala to CDH 4.2 or 4.3, you must restart MapReduce before Impala restarts (by default Impala is restarted before MapReduce).

    The workaround is to perform a restart of MapReduce alone as the first step, then perform a cluster restart of the remaining services.

  3. Click Confirm to start the rolling restart.

Step 6. Remove the Previous CDH Version Packages

If your previous installation of CDH was done using packages, you must remove those packages on all hosts on which you installed the parcels and refresh the symlinks so that clients will run the new software versions. This will definitely be the case if you are running a version of CDH prior to CDH 4.1.3, since parcels were not available with those releases.
  1. Uninstall the CDH packages.
    Operating System Command
    RHEL $ sudo yum remove bigtop-jsvc bigtop-utils bigtop-tomcat hue-common sqoop2-client hbase-solr-doc solr-doc
    SLES $ sudo zypper remove bigtop-jsvc bigtop-utils bigtop-tomcat hue-common sqoop2-client hbase-solr-doc solr-doc
    Ubuntu or Debian $ sudo apt-get purge bigtop-jsvc bigtop-utils bigtop-tomcat hue-common sqoop2-client hbase-solr-doc solr-doc
  2. If you were previously using packages rather than parcels prior to this upgrade, you must restart all the Cloudera Manager Agents to force an update of the symlinks to point to the newly installed components. On each host:
    $ sudo service cloudera-scm-agent restart