This is the documentation for Cloudera Manager 4.8.3.
Documentation for other versions is available at Cloudera Documentation.

The MapReduce Service

Configuring High Availability for the JobTracker

Starting with Cloudera Manager 4.6 you can use Cloudera Manager to configure CDH4.3 or later for JobTracker High Availability (HA). Job Tracker High Availability is not supported for CDH3 clusters, or with CDH4 prior to CDH4.2.

  Note: Cloudera Manager supports JobTracker HA with CDH4.3. Although it is possible to configure JobTracker HA with CDH4.2, it is not recommended. Rolling restart, decommissioning of TaskTrackers, and rolling upgrade of MapReduce from CDH 4.2 to CDH4.3 are not supported when JobTracker HA is enabled.

JobTracker High Availability is available for MapReduce v1. It is not available for YARN (MR v2).

A JobTracker HA cluster is configured with an Active and a Standby JobTracker. Only one JobTracker can be active at any point in time. JT High Availability depends on maintaining a log of all namespace modifications in a location available to both NameNodes, so that in the event of a failure the Standby NameNode has up-to-date information about the edits and location of blocks in the cluster.

Cloudera Manager supports automatic failover of the JobTracker. It does not provide a mechanism to manually force a failover through the Cloudera Manager user interface.

See the Configuring High Availability for the JobTracker in the CDH High Availability Guide for a more detailed introduction to JobTracker High Availability.

  Important:
  • Enabling or Disabling High Availability will shut down your HDFS service, and the services that depend on it–MapReduce, YARN, and HBase. Therefore, you should not do this while you have jobs running on your cluster. Further, once HDFS has been restored, the services that depend upon it must be restarted, and the client configurations for HDFS must be redeployed.
  • Enabling or Disabling High Availability will cause the previous monitoring history to become unavailable.

Enabling JobTracker High Availability

After you have installed MapReduce on your CDH4.2 cluster, the Enable High Availability workflow leads you through adding a second (Standby) JobTracker and configuring JournalNodes.

  1. From the Services tab, select your MapReduce service.
  2. Pull down the Actions menu and select Enable High Availability. This starts the wizard for enabling HA. (The menu option does not appear if this is a CDH3 version of the MapReduce service.)
  3. The next screen shows the hosts that are eligible to run a standby JobTracker. Select the host where you want the standby JobTracker to be installed, and click Continue. The host where the current JobTracker is running is not available as a choice.
  4. Enter a directory location on the local filesystem for each JobTracker host. These directories will be used to store job configuration data.
    • You may enter more than one directory, though it is not required. The names/paths do not need to be the same on both JobTrackers.
    • The directories you specify must exist, and should be empty, and must have the appropriate permissions.
    • If the directories are not empty, Cloudera Manager will not delete the contents; however, in that case the data should be in sync across the edits directories of the JournalNodes and should have the same version data as the NameNodes.
  5. Optionally use the checkbox under Advanced Options to force initialize the ZooKeeper ZNode for autofailover.
  6. Click Continue.

    Cloudera Manager proceeds to execute the set of commands that stops the MapReduce service, adds a Standby JobTracker and Failover controller, initializes the JobTracker High Availability state in ZooKeeper, creates the JobStatus directory, restarts MapReduce and redeploys the relevant client configurations.

Disabling JobTracker High Availability

  1. From the Services tab, select your MapReduce service.
  2. Pull down the Actions menu and select Disable High Availability....
  3. Select which JobTracker (host) you want to remain as the single JobTracker, and click Continue.

    Cloudera Manager proceeds to execute the set of commands that stops the MapReduce service, removes the Standby JobTracker and the Failover Controller, restarts the MapReduce service and redeploys client configurations.

MapReduce Scheduler Configuration

The MapReduce service is configured by default to use the FairScheduler. You can change scheduler type to FIFO or Capacity Scheduler. You can also modify the Fair Scheduler and Capacity Scheduler configuration. For more information about these schedulers, see Fair Scheduler or Capacity Scheduler.

You can modify the task scheduler configuration from the JobTracker Role Group:

  1. From the MapReduce service page, under the Configuration tab, select View and Edit.
  2. Under the JobTracker Default Group, go to the Classes category.
  3. By default, the Task Scheduler (mapred.jobtracker.taskScheduler) is set to the FairScheduler (org.apache.hadoop.mapred.FairScheduler).

    To change this, click anywhere in the Value field—this displays the list of scheduler types. Select the one you want to use and then Save Changes.

  4. The actual configuration of the Fair Scheduler or the Capacity Scheduler (allocation, pool configuration, pre-emption and so on) can be managed under the Jobs category of the JobTracker Role Group. Make your changes as appropriate and click Save Changes.
  5. Refresh the JobTracker to have the new configuration take effect.
    1. Go to the MapReduce service page.
    2. Click the Instances tab.
    3. Click the jobtracker role.
    4. Select Actions for Selected > Restart.

The MapReduce service is configured by default to use the FairScheduler. You can change scheduler type to FIFO or Capacity Scheduler. You can also modify the Fair Scheduler and Capacity Scheduler configuration. For more information about these schedulers, see Fair Scheduler or Capacity Scheduler.

Configuring the MapReduce Service to Save Job History in HDFS

Normally job history is saved on the host on which the JobTracker is running. To enable Map Reduce job history to be saved to HDFS:
  1. Create a folder in HDFS to contain the history information. When creating the folder in HDFS, set the owner and group to mapred:hadoop with permission setting 775.
  2. In the Cloudera Manager Admin Console, select the MapReduce service.
  3. Select Configuration > View and Edit.
  4. Expand the JobTracker (Default) category and click the Paths subcategory.
  5. Set the Completed Job History Location property to the location that you created in step 1.
  6. Click Save Changes.
  7. Restart the MapReduce service.
The Job Tracker writes information about every job that completes to the specified location. By default the information is retained for 7 days. Files older than 7 days are deleted when the Job History Files Cleaner runs. To change the retention duration:
  1. Select the JobTracker (Default) category.
  2. Set the Job History Files Maximum Age property (mapreduce.jobhistory.max-age-ms) to the length of time (in milliseconds, seconds, minutes, or hours) that you want job history files to be kept.
  3. Restart the MapReduce service.
The Job History Files Cleaner runs at regular intervals to check for job history files that are ready to be deleted. By default, the interval is 24 hours. To change the frequency with which the Job History Files Cleaner runs:
  1. Select the JobTracker (Default) category.
  2. Set the Job History Files Cleaner Interval property (mapreduce.jobhistory.cleaner.interval) to the desired frequency (in milliseconds, seconds, minutes, or hours).
  3. Restart the MapReduce service.

Configuring Client Overrides

A configuration property qualified with (Client Override) is a server-side setting that ignores whatever value a client might try to set for that property. It performs the same role as its unqualified counterpart, and applies the configuration to the service with the setting <final>true</final>.

For example, if you set the Map task heap property to 1 GB in the job configuration code and the service's heap property qualified with (Client Override) is set to 500 MB, then 500 MB is applied despite what you've asked the job to use.