In the previous Cloudera documentation, we have seen the entire setup using the Cloudera version 5.x.x. This blog will not have the details about the Cloudera manager but will only cover the configuration steps of Cloudera manager v6.1.1. For more details on Cloudera, you can refer our previous blog here. So let’s Start with Setup of Cloudera Manager v6.1.1.
Consider it to be a 2 node cluster just for an example. Server configurations will be like. Make Sure to change the Hostname of your Nodes. We are doing this setup with respect to centos/redhat7.
cloudera.devopsage.com: Cloudera manager node.
node1.devopsage.com : node 1
node2.devopsage.com : node 2
Change the hostname of the server by using below command.
# hostnamectl set-hostname cloudera.devopsage.com
# sudo vim /etc/sysconfig/network
HOSTNAME=cloudera.devopsage.com
Also make sure to have a local DNS entries in the /etc/hosts file
node1_IP node1.example.com node1
node2_IP node2.example.com node2
Create a local DNS Entries on all the Server and then try to ping each other from the local domain to check whether its working or not.
If you do not make entry to /etc/hosts then you may observe an error while installing the Cloudera agents to the nodes probably the error would be like “Installation failed. Failed to receive a heartbeat from an agent.”
You also need to check a few things like your firewall should be disabled, Disabling SELinux etc. Check here for more details.
Let’s Start with the Installation and Configuration Steps,
Configure NTP On All the servers
# yum install ntp
# sudo systemctl start ntpd
# sudo systemctl enable ntpd
# ntpdate -u --> check for ntp server in the config /etc/ntp.conf
# hwclock --systohc --> Synchronize the hardware clock to system clock.
# sysctl -w vm.swappiness=1
# echo never > /sys/kernel/mm/transparent_hugepage/defrag
# echo never > /sys/kernel/mm/transparent_hugepage/enabled
Install & Configure MySQL for Cloudera.
# wget http://repo.mysql.com/mysql-community-release-el7-5.noarch.rpm
# sudo rpm -ivh mysql-community-release-el7-5.noarch.rpm
# sudo yum update
# sudo yum install mysql-server
Get the MySQL Config file and paste it to /etc/my.cnf. Copy from here.
Configuring Mysql Server.
# sudo vim /etc/my.cnf
# sudo systemctl enable mysqld
# sudo systemctl start mysqld
# sudo mysql_secure_installation
Enter current password for root (enter for none): OK, successfully used password, moving on…
Set root password? [Y/n] Y
New password: Re-enter new password:
Remove anonymous users? [Y/n] Y
Disallow root login remotely? [Y/n] N
Remove test database and access to it [Y/n] Y
Reload privilege tables now? [Y/n] Y
All done!
Install MySQL JDBC Driver
# wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.46.tar.gz
# tar zxvf mysql-connector-java-5.1.46.tar.gz
# sudo mkdir -p /usr/share/java/
# cd mysql-connector-java-5.1.46
# sudo cp mysql-connector-java-5.1.46-bin.jar /usr/share/java/mysql-connector-java.jar
Add Repository for Cloudera Manager
# wget https://archive.cloudera.com/cm6/6.1.1/redhat7/yum/cloudera-manager.repo -P /etc/yum.repos.d/
# sudo rpm --import https://archive.cloudera.com/cm6/6.1.0/redhat7/yum/RPM-GPG-KEY-cloudera
# yum update -y
# yum repolist
Install Java Development Kit – JDK
# sudo yum install oracle-j2sdk1.8
# export JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera/
# export JRE_HOME=/usr/java/jdk1.8.0_181-cloudera/jre
# export PATH=$PATH:$JAVA_HOME/bin
Note: Add this to .bashrc file. For installing JDK on other nodes you can use Cloudera manager later.
Install Cloudera Manager Server
# sudo yum install cloudera-manager-daemons cloudera-manager-agent cloudera-manager-server
Note: It will download and install around 1.1GB of packages. Wait for it to download and install.
Setup databases and users used by Hadoop Components.
# mysql -u root -p
# CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
# GRANT ALL ON scm.* TO 'scm_user'@'%' IDENTIFIED BY 'secure_pass';
# create database hive DEFAULT CHARACTER SET utf8;
# create database oozie DEFAULT CHARACTER SET utf8;
# create database hue DEFAULT CHARACTER SET utf8;
# create database rman DEFAULT CHARACTER SET utf8;
# GRANT ALL PRIVILEGES ON hive.* TO 'hive-user' IDENTIFIED BY 'HiveyPassword@123#';
# GRANT ALL PRIVILEGES ON oozie.* TO 'oozie_user' IDENTIFIED BY 'OoziePassword@123#';
# GRANT ALL PRIVILEGES ON hue.* TO 'hue_user' IDENTIFIED BY 'HuePassword@123#';
# GRANT ALL PRIVILEGES ON rman.* TO 'rman_user' IDENTIFIED BY 'RmanPassword@123#';
# flush privileges;
# show databases;
Setup Cloudera Manager Database
# sudo /opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm scm_user
Enter password:
If using a remote database, then use below command,
# sudo /opt/cloudera/cm/schema/scm_prepare_database.sh mysql -h --scm-host cloudera.devopsage.com scm scm_user
Start Cloudera SCM Server
# sudo systemctl start cloudera-scm-server
# sudo systemctl enable cloudera-scm-server
# sudo tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log
When you see the below log, that means you Cloudera server is ready. It may take a couple of mins to start
INFO WebServerImpl:com.cloudera.server.cmf.WebServerImpl: Started Jetty server.
Cluster Setup
Take the public Ip of your Cloudera manager server and log in. default username and password will be admin/admin. You can change your password later from the settings.
http://Cloudera-manager-IP:7180 --> wait for a couple of minutes if you are not able to connect. Check logs.
For the Cluster Setup (adding nodes, installing Cloudera agents, installing JDK on node servers, assigning roles etc.) refer our previous blog here
You Can Also Refer.
https://www.devopsage.com/how-to-setup-hadoop-cluster-using-cloudera-manager/
References:
Cloudera Official Documentation
Link: https://www.cloudera.com/documentation/enterprise/6/latest/topics/installation.html