Installing Cloudera Manager and CDH on EC2
The following procedure leads you through setting up Cloudera Manager, CDH and Impala on a cluster of Amazon Web Services (AWS) EC2 instances.
- The Cloudera Manager installation wizard launches the EC2 version of the wizard when Cloudera Manager is started on EC2.
- The EC2 wizard installs Cloudera Manager Free Edition. If you have a license for the Enterprise Edition, you can upgrade from the Cloudera Manager Admin Console after installation. (See the Upgrade from Cloudera Manager Free Edition 4 to Cloudera Manager 4 for details.)
- The resulting installation will use the embedded PostgreSQL data base; there is no option for setting up other databases.
- This wizard will install and start all the latest Cloudera Manager-managed CDH 4 services as well as Impala.
Step 1: Set up an Amazon Web Services (AWS) EC2 instance for the Cloudera Manager server.
- Log into the AWS console.
- Go to EC2.
- Create a security group:
- In the left menu, select Security Groups.
- Click Create Security Group.
- When prompted, enter a name and description, and click OK.
- Select the group you created in the list of groups.
- In the bottom panel, go to the Inbound tab.
- Authorize TCP ports 7180, 7182, 7183 and 7432.
- Authorize ICMP (Echo Reply).
- Create (or import) an SSH key pair:
- In the left menu, select Key Pairs.
- Click Create Key Pair.
- When prompted, enter a key pair name and click OK
- Your private key < keypair-name >.pem will be downloaded automatically. AWS does not store the private keys – if you lose this file, you won't be able to SSH into instances you provision with this key pair.
- Launch an EC2 instance:
- Select the Ubuntu12.04 AMI or other Operating System supported by Cloudera Manager. See Requirements for Cloudera Manager.
- Use the key pair and security group you prepared in the previous steps.
- Select the Instance Type: Cloudera recommends using at least "M1 Large" instances.
- Look at the instance details, and copy the public hostname.
- Then SSH into the instance:
$ ssh -i <private-key-file> <username>@ec2-xx-xx-xx-xx.compute-1.amazonaws.com
The <username> is usually "ubuntu" on Ubuntu systems, and "ec2-user" on most other Linux images on EC2.
- Download the Cloudera Manager installer:
$ wget http://archive.cloudera.com/cm4/installer/latest/cloudera-manager-installer.bin
- Execute the installer:
$ sudo su $ chmod +x cloudera-manager-installer.bin $ ./cloudera-manager-installer.bin
- When the installer finishes, navigate to http://<public hostname>:7180 and log into the Cloudera Manager Admin console.
- (optional) Configure TLS encryption. (See Configuring TLS Security for Cloudera Manager).
Step 2: Use the cloud wizard to provision new cloud instances and install Cloudera Manager and CDH packages.
- Log into the Cloudera Manager admin console on your EC2
- The initial user name and password are admin.
- The Welcome Page appears.
- Please note the Warning that is presented.
Instances provisioned on AWS EC2 by this wizard are instance store-based, so all data will be lost when an instance is stopped or terminated.
- Provide the instance specifications:
- Choose your OS: currently supported are Ubuntu 12.04 or Centos 6.3.
- Alternatively, you may use a custom AMI:
- Make sure the AMI is in the same region as Cloudera Manager server!
- Specify the username Cloudera Manager should use to SSH in. This is usually "ubuntu" on Ubuntu systems, and "ec2-user" on most other Linux images on EC2.
- Choose the type of EC2 instances you want to
- Note that instances not matching the minimum requirements were deliberately removed from the list.
- Specify the number of instances you wish to provision.
- Specify the group name (string).
- This string will be included in the name of your instances and the security group and key pair, which will be created by Cloudera Manager.
- Provide credentials:
- Enter the AWS Access and Secret key. To create new
ones, follow these instructions:
- Go to https://console.aws.amazon.com/iam/.
- Click Users.
- Check the box next to the desired user, scroll down and click Manage Access Keys.
- Copy the new keys and paste them to the inputs.
- Choose the instance authentication method: Either:
- Let Cloudera Manager create a new SSH key pair for your instances. You will be able to download the private key later to SSH into the new instances. — OR —
- Import and upload your own key:
- In the console, go to Key Pairs.
- Click Import Key.
- Select your private key file, specify the name and click Yes, Import.
- Enter the AWS Access and Secret key. To create new ones, follow these instructions:
- Review the Installation Settings:
- You may go back if you want to correct any information you provided in the previous steps.
- Once the instances are provisioned, you must terminate them if you need to modify the installation settings.
- Click Start Installation.
- Provision new instances. Once instances are provisioned:
- Download the private SSH key if you chose to create one.
- Remember to terminate the instances (through the AWS EC2 console).
- The wizard leads you through the installation steps:
- Install Cloudera Manager and CDH packages.
- Run the Host Inspector.
- Start all Services.
How to Terminate your EC2 instances
Cloudera Manager will only terminate instances if the installation fails. Terminate the instances manually when you are done using the CDH cluster.
- Sign into the AWS EC2 console.
- In the left menu, select Instances.
- Select the instances you want to terminate. You may use the string you entered as "group name" to filter the instances provisioned by Cloudera Manager.
- From Actions select Terminate.