Known Issues and Workarounds in Cloudera Manager 5
The following sections describe the current known issues in Cloudera Manager 5.
Reset non-default HDFS File Block Storage Location Timeout value after upgrade from CDH 4 to CDH 5
If HDFS File Block Storage Location Timeout (dfs.client.file-block-storage-locations.timeout, dfs.client.file-block-storage-locations.timeout.millis) was set to a non-default value on a CDH 4 cluster, when upgrading that cluster to CDH 5 the value will not be updated to the new minimum required by Impala, 10 seconds. This will cause Impala daemons not to start after upgrade.
Workaround: Reset this config to the default, or set it to a value >= 10 seconds.
Secure CDH 4.1 clusters can't have Hue and Impala share the same Hive
Secure CDH 4.1 clusters can't have Hue and Impala share the same Hive. If Hive has "Bypass Hive Metastore Server" disabled, then Hue will not be able to talk to Hive. If Hive has "Bypass Hive Metastore" enabled, then Impala will have a validation error.
Workaround: Upgrade to CDH 4.2.
If installing CDH 4 packages, the Impala 1.3.0 option does not work because Impala 1.3 is not yet released for CDH 4.
If installing CDH 4 packages, the Impala 1.3.0 option listed in the install wizard does not work because Impala 1.3.0 is not yet released for CDH 4.
Workaround: Install using parcels (where the unreleased version of Impala does not appear), or select a different version of Impala when installing with packages.
When updating dynamic resource pools, Cloudera Manager updates roles but may fail to update role information displayed in the UI
When updating dynamic resource pools, Cloudera Manager automatically refreshes the affected roles, but they sometimes get marked incorrectly as running with outdated configurations and requiring a refresh.
Workaround: Invoke the Refresh Cluster command from the cluster actions drop-down menu.
User should be prompted to add the AMON role when adding MapReduce to a CDH 5 cluster
When the MapReduce service is added to a CDH 5 cluster, the user is not asked to add the AMON role. Then, an error displays when the user tries to view MapReduce activities.
Workaround: Manually add the AMON role after adding the MapReduce service.
Replace YARN_HOME with HADOOP_YARN_HOME during upgrade
If yarn.application.classpath was set to a non-default value on a CDH 4 cluster, and that cluster is upgraded to CDH 5, the classpath is not updated to reflect that $YARN_HOME was replaced with $HADOOP_YARN_HOME. This will cause YARN jobs to fail.
Workaround: Reset yarn.application.classpath to the default, then re-apply your classpath customizations if needed.
Enterprise license expiration alert not displayed until Cloudera Manager Server is restarted
When an enterprise license expires, the expiration notification banner is not displayed until the Cloudera Manager Server has been restarted (although the expired enterprise features stop working immediately upon license expiration, as expected).
Erroneous warning displayed on the HBase configuration page on CDH 4.1 in Cloudera Manager 5.0.0
An erroneous "Failed parameter validation" warning is displayed on the HBase configuration page on CDH 4.1 in Cloudera Manager 5.0.0
Workaround: Use CDH4.2 or higher, or ignore the warning.
Cluster installation with CDH 4.1 and Impala fails
In Cloudera Manager 5.0, installing a new cluster through the wizard with CDH 4.1 and Impala fails with the following error message, "dfs.client.use.legacy.blockreader.local is not enabled."
- Use CDH 4.2 or higher, or
- Install all desired services except Impala in your initial cluster setup. From the home page, use the dropdown menu near the cluster name and select Configure CDH Version. Confirm the version, then add Impala.
MR1 to MR2 import fails on a secure cluster
When running the MR1 to MR2 import on a secure cluster, YARN jobs will fail to find container-executor.cfg.
Workaround: Restart YARN after the import.
After upgrade from CDH 4 to CDH 5, Oozie is missing workflow extension schemas
After an upgrade from CDH 4 to CDH 5, Oozie does not pick up the new workflow extension schemas automatically. User will need to update oozie.service.SchemaService.wf.ext.schemas manually and add the schemas added in CDH 5: shell-action-0.3.xsd, sqoop-action-0.4.xsd, distcp-action-0.2.xsd, oozie-sla-0.1.xsd, oozie-sla-0.2.xsd. Note: None of the existing jobs will be affected by this bug, only new workflows that require new schemas.
Workaround: Add the new workflow extension schemas to Oozie manually by editing oozie.service.SchemaService.wf.ext.schemas.
HDFS NFS gateway works only on RHEL and similar systems
Because of bug in native versions of portmap/rpcbind, the HDFS NFS gateway does not work on SLES, Ubuntu, or Debian systems. It does work on supported versions of RHEL- compatible systems on which rpcbind-0.2.0-10.el6 or later is installed. (See
- On Red Hat and similar systems, make sure rpcbind-0.2.0-10.el6 or later is installed.
- On SLES, Debian, and Ubuntu systems, you can use the gateway by running rpcbind in insecure mode, using the -i option, but keep in mind that this allows anyone from a remote host to bind to the portmap.
HDFS replication does not work from CDH 5 to CDH 4 with different realms
HDFS replication does not work from CDH 5 to CDH 4 with different realms. This is because authentication fails for services in a non-default realm via the WebHdfs API due to a JDK bug. This has been fixed in JDK6-u34 (b03)) and in JDK7.
Workaround: Use JDK 7 or upgrade JDK6 to at least version u34.
Upgrade from Cloudera Manager 5.0.0 beta 1 or beta 2 to Cloudera Manager 5.0.0 requires assistance from Cloudera Support
Contact Cloudera Support before upgrading from Cloudera Manager 5.0.0 beta 1 or beta 2 to Cloudera Manager 5.0.0.
Workaround: Contact Cloudera Support.
Failure of HDFS Replication between clusters with YARN
HDFS replication between clusters in different Kerberos realms fails when using YARN if the target cluster is CDH 5.
Workaround: Use MapReduce (MRv1) instead of YARN.
Empty box appears over the list of results after adding a tag to a file
When tags are added to an entity, in some cases a white box remains after pressing Enter.
Workaround: Refresh the page to remove the artifact.
Upgrade of secure cluster requires installation of JCE policy files
When upgrading a secure cluster via Cloudera Manager, the upgrade initially fails due to the JDK not having Java Cryptography Extension (JCE) unlimited strength policy files. This is because Cloudera Manager installs a copy of the Java 7 JDK during the upgrade, which does not include the unlimited strength policy files. To ensure that unlimited strength functionality continues to work, install the unlimited strength JCE policy files immediately after completing the Cloudera Manager Upgrade Wizard and before taking any other actions in Cloudera Manager.
Workaround: Install the unlimited strength JCE policy files immediately after completing the Cloudera Manager Upgrade Wizard and before taking any other action in Cloudera Manager.
The Details page for MapReduce jobs displays the wrong id for YARN-based replications
The Details link for MapReduce jobs is wrong for YARN-based replications.
Workaround: Find the job id in the link and then go to the YARN Applications page and look for the job there.
The Spark Upload Jar command fails in a secure cluster
The Spark Upload Jar command fails in a secure cluster.
Workaround: To run Spark on YARN, manually upload the Spark assembly jar to HDFS /user/spark/share/lib. The Spark assembly jar is located on the local filesystem, typically in /usr/lib/spark/assembly/lib or /opt/cloudera/parcels/CDH/lib/spark/assembly/lib.
Configurations for decommissioned roles not migrated from MapReduce to YARN
When the Import MapReduce Configuration wizard is used to import MapReduce configurations to YARN, decommissioned roles in the MapReduce service do not cause the corresponding imported roles to be marked as decommissioned in YARN.
Workaround: Delete or decommission the roles in YARN after running the import.
The HDFS command Roll Edits does not work in the UI when HDFS is federated
The HDFS command Roll Edits does not work in the Cloudera Manager UI when HDFS is federated because the command doesn't know which nameservice to use.
Workaround: Use the API, not the Cloudera Manager UI, to execute the Roll Edits command.
Cloudera Manager reports a confusing version number if you have oozie-client, but not oozie installed on a CDH 4.4 node
In CDH versions before 4.4, the metadata identifying Oozie was placed in the client, rather than the server package. Consequently, if the client package is not installed, but the server is, Cloudera Manager will report Oozie has been present but as coming from CDH 3 instead of CDH 4.
Workaround: Either install the oozie-client package, or upgrade to at least CDH 4.4. Parcel based installations are unaffected.
The Sqoop Upgrade command in Cloudera Manager may report success even when the upgrade fails
- Click the Sqoop service and then the Instances tab.
- Click the Sqoop server role then the Commands tab.
- Click the stdout link and scan for the Sqoop Upgrade command.
- In the All Recent Commands page, select the stdout link for latest Sqoop Upgrade command.
The command history has an option to select the number of commands, but doesn't always return the number you request
Hue doesn't support YARN ResourceManager High Availability
- Go to the Hue service.
- Select .
- Click .
- In the Hue Server Advanced Configuration Snippet (Safety Valve) for hue_safety_valve_server.ini field, add the following:
[hadoop] [[ yarn_clusters ]] [[[default]]] resourcemanager_host=<hostname of active ResourceManager> resourcemanager_api_url=http://<hostname of active resource manager>:<web port of active resource manager> proxy_api_url=http://<hostname of active resource manager>:<web port of active resource manager>The default web port of Resource Manager is 8088.
- Click Save Changes to have these configurations take effect.
- Restart the Hue service.
Cloudera Manager doesn't work with CDH 5.0.0 Beta 1
When you upgrade from Cloudera Manager 5.0.0 Beta 1 with CDH 5.0.0 Beta 1 to Cloudera Manager 5.0.0 Beta 2, Cloudera Manager won't work with CDH 5.0.0 Beta 1 and there's no notification of that fact.
Workaround: None. Do a new installation of CDH 5.0.0 Beta 2.
On CDH 4.1 secure clusters managed by Cloudera Manager 4.8.1 and higher, the Impala Catalog server needs advanced configuration snippet update
Impala queries fail on CDH 4.1 when Hive "Bypass Hive Metastore Server" option is selected.
Workaround: Add the following to Impala catalog server advanced configuration snippet for hive-site.xml, replacing <Hive_Metastore_Server_Host> with the host name of your Hive Metastore Server:
<property> <name>hive.metastore.local</name> <value>false</value> </property> <property> <name>hive.metastore.uris</name> <value>thrift://<Hive_Metastore_Server_Host>:9083</value> </property>
Rolling Upgrade to CDH 5 is not supported.
Rolling upgrade between CDH 4 and CDH 5 is not supported. Incompatibilities between major versions means rolling restarts are not possible. In addition, rolling upgrade will not be supported from CDH 5.0.0 Beta 1 to any later releases, and may not be supported between any future beta versions of CDH 5 and the General Availability release of CDH 5.
Error reading .zip file created with the Collect Diagnostic Data command.
After collecting Diagnostic Data and using the Download Diagnostic Data button to download the created zip file to the local system, the zip file cannot be opened using the FireFox browser on a Macintosh. This is because the zip file is created as a Zip64 file, and the unzip utility included with Macs does not support Zip64. The zip utility must be version 6.0 or later. You can determine the zip version with unzip -v.
Workaround: Update the unzip utility to a version that supports Zip64.
Enabling wildcarding in a secure environment causes NameNode to fail to start.
In a secure cluster, you cannot use a wildcard for the NameNode's RPC or HTTP bind address, or the NameNode will fail to start. For example, dfs.namenode.http-address must be a real, routable address and port, not 0.0.0.0.<port>. In Cloudera Manager, the "Bind NameNode to Wildcard Address" property must not be enabled. This should affect you only if you are running a secure cluster and your NameNode needs to bind to multiple local addresses.
Workaround: Disable the "Bind NameNode to Wildcard Address" property found under the Configuration tab for the NameNode role group.
After JobTracker failover, complete jobs from the previous active JobTracker are not visible.
When a JobTracker failover occurs and a new JobTracker becomes active, the new JobTracker UI does not show the completed jobs from the previously active JobTracker (that is now the standby JobTracker). For these jobs the "Job Details" link does not work.
After JobTracker failover, information about rerun jobs is not updated in Activity Monitor.
When a JobTracker failover occurs while there are running jobs, jobs are restarted by the new active JobTracker by default. For the restarted jobs the Activity Monitor will not update the following: 1) The start time of the restarted job will remain the start time of the original job. 2) Any Map or Reduce task that had finished before the failure happened will not be updated with information about the corresponding task that was rerun by the new active JobTracker.
Installing on AWS, you must use private EC2 hostnames.
When installing on an AWS instance, and adding hosts using their public names, the installation will fail when the hosts fail to heartbeat.
Use the Back button in the wizard to return to the original screen, where it prompts for a license.
Rerun the wizard, but choose "Use existing hosts" instead of searching for hosts. Now those hosts show up with their internal EC2 names.
Continue through the wizard and the installation should succeed.
After removing and then re-adding a service, the alternatives settings are incorrect.
After deleting a service, the alternatives settings are not cleaned up. If you then re-add the service, it will be given a new instance name, and a new set of configurations settings are added. However, because both the new and old (deleted) instances have the same alternatives priority, the original one will be used rather than the newer one.
Workaround: The simplest way to fix this is:
- Go to the Configuration tab for the new service instance in Cloudera Manager
- Search for "alternatives".
- Raise the priority value and save the setting.
- Redeploy the client configuration.
Cloudera Manager does not support encrypted shuffle.
Encrypted shuffle has been introduced in CDH 4.1, but it is not currently possible to enable it through Cloudera Manager.
If HDFS uses Quorum-based Storage without HA enabled, the SecondaryNameNode cannot checkpoint.
If HDFS is set up in non-HA mode, but with Quorum-based storage configured, the dfs.namenode.edits.dir is automatically configured to the Quorum-based Storage URI. However, the SecondaryNameNode cannot currently read the edits from a Quorum-based Storage URI, and will be unable to do a checkpoint.
Workaround: Add to the NameNode's advanced configuration snippet the dfs.namenode.edits.dir property with both the value of the Quorum-based Storage URI as well as a local directory, and restart the NameNode. For example,
<property> <name>dfs.namenode.edits.dir</name> <value>qjournal://jn1HostName:8485;jn2HostName:8485;jn3HostName:8485/journalhdfs1,file:///dfs/edits</value> </property>
Changing the rack configuration may temporarily cause mis-replicated blocks to be reported.
A rack re-configuration will cause HDFS to report mis-replicated blocks until HDFS rebalances the system, which may take some time. This is a normal side-effect of changing the configuration.
Starting HDFS with HA and Automatic Failover enabled, one of the NameNodes might not start.
When starting an HDFS service with High Availability and Automatic Failover enabled, one of the NameNodes might might not start up.
Workaround: To fix this, start the NameNode that failed to start up after the remaining HDFS roles start up.
Cannot use '/' as a mount point with a Federated HDFS Nameservice.
A Federated HDFS Service doesn't support nested mount points, so it is impossible to mount anything at '/'. Because of this issue, the root directory will always be read-only, and any client application that requires a writeable root directory will fail.
- In the CDH 4 HDFS Service > Configuration tab of the Cloudera Manager Admin Console, search for "nameservice".
- In the Mountpoints field, change the mount point from "/" to a list of mount points that are in the namespace that the Nameservice will manage. (You can enter this as a comma-separated list - for example, "/hbase, /tmp, /user" or by clicking the plus icon to add each mount point in its own field.) You can determine the list of mount points by running the command hadoop fs -ls / from the CLI on the NameNode host.
Historical disk usage reports do not work with federated HDFS.
(CDH 4 only) Activity monitoring does not work on YARN activities.
Activity monitoring is not supported for YARN in CDH 4.
HDFS monitoring configuration applies to all Nameservices
The monitoring configurations at the HDFS level apply to all Nameservices. So, if there are two federated Nameservices, it's not possible to disable a check on one but not the other. Likewise, it's not possible to have different thresholds for the two Nameservices.
Supported and Unsupported Replication Scenarios and Limitations
Restoring snapshot of a file to an empty directory does not overwrite the directory
Restoring the snapshot of an HDFS file to an HDFS path that is an empty HDFS directory (using the Restore As action) will result in the restored file present inside the HDFS directory instead of overwriting the empty HDFS directory.
HDFS Snapshot appears to fail if policy specifies duplicate directories.
In an HDFS snapshot policy, if a directory is specified more than once, the snapshot appears to fail with an error message on the Snapshot page. However, in the HDFS Browser, the snapshot is shown as having been created successfully.
Workaround: Remove the duplicate directory specification from the policy.
Hive replication fails if "Force Overwrite" is not set.
The Force Overwrite option, if checked, forces overwriting data in the target metastore if there are incompatible changes detected. For example, if the target metastore was modified and a new partition was added to a table, this option would force deletion of that partition, overwriting the table with the version found on the source. If the Force Overwrite option is not set, recurring replications may fail.
Workaround: Set the Force Overwrite option.
During HDFS replication, tasks may fail due to DataNode timeouts.
In CDH 4.2, during an HDFS replication job (using Cloudera Manager's Backup and Data Recovery product) individual tasks in the Replication job may fail due to DataNode timeouts. If enough of these timeouts occur, the replication task may slow down, and the entire replication job could time out and fail.