来源：https://blog.csdn.net/ddyyll988/article/details/59057584
标签：Hadoop，Cloudera, Manager, 集群，部署
收藏：株野
CDH install
[detail installing PDF]($ sudo ./cloudera-manager-installer.bin)
officialcreate virtual machine
eth0: hostonly
eth1: bridge
configure network and host
set ethX name and write down hardware address
/etc/udev/rules.d/70-persistent-net.rules
/etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0HWADDR=08:00:27:0F:B6:C1TYPE=EthernetUUID=be125edc-4dda-4478-964b-285409fdc735ONBOOT=yesNM_CONTROLLED=yesBOOTPROTO=staticIPADDR=192.168.56.2
/etc/sysconfig/network-scripts/ifcfg-eth1
DEVICE=eth1HWADDR=08:00:27:9D:D9:E6TYPE=EthernetUUID=be125edc-4dda-4478-964b-285409fdc735ONBOOT=yesNM_CONTROLLED=yesBOOTPROTO=dhcp
set host name and gateway
$ vi /etc/sysconfig/networkNETWORKING=yesHOSTNAME=master#GATEWAY=192.168.1.1
hosts
$vi /etc/hosts 192.168.56.2 master192.168.56.3 slave1
install perl
yum install perlcreate user
sudo useradd -m clouderapasswd cloudera
add to sudoer
visudo
无密码sudo权限
cloudera ALL=(root)NOPASSWD:ALL
trun off firewalls
disable the selinux at all nodes
vi /etc/selinux/configSELINUX=disabled #carfore! not SELINUXTYPE!!!close iptables firewall
chkconfig iptables offchkconfig ip6tables offservice iptables stopTime
date #view the time#set all node's time zone to shanghaicp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime#update timeyum install ntpntpdate pool.ntp.org#update time manuallydate -s 02/25/17date -s 19:21:00hwclock -w #white time to BIOSchkconfig ntpd oninstall JDK(alternative)
remove open jdk
java -versionrpm -qa | grep javarpm -e --nodeps java-1.5.0-gcj-1.5.0.0-29.1.el6.x86_64rpm -e --nodeps java-1.6.0-openjdk-1.6.0.0-1.66.1.13.0.el6.x86_64rpm -e --nodeps java-1.6.0-openjdk-devel-1.6.0.0-1.66.1.13.0.el6.x86_64
install JDK
supported versionCDH5 -> jdk-7u55-linux-x64
jdk7jdk8mkdir /usr/java/jdk-versioncd /usr/java/jdk-versiontar zxvf jdk.1.7.0_nn.tar.gzln -s /usr/java/jdk.1.7.0_nn /usr/java/default#or rpm -ivh jdk-7u55-linux-x64.rpm
vi /home/cloudera/.bash_profile
export JAVA_HOME=/usr/java/defaultexport CLASSPATH=.:$JAVA_HOME/lib:$CLASSPATH export PATH=$PATH:$JAVA_HOME/bin:$JAVA_HOME/jre/bin source /home/hadoop/.bash_profilerm /usr/java/default.tar.gzjava -version
Ubuntu:
sudo apt-get -o Dpkg::Options::=--force-confdef -o Dpkg::Options::=--force-confold -y install oracle-j2sdk1.7
这步如果跳过的话之后在安装的时候就勾上要装JDK
clone virtual machine
clone and configure:
network configure
delete /etc/udev/rules.d/70-persistent-net.rules and reboot, or delete primary eth0 & eth1 change eth3 ð4 to eth0 & eth1
or you can
remove modprobe: modprobe -r e1000
reload modprobe: modprobe e1000
change name and write down hardware address in
/etc/udev/rules.d/70-persistent-net.rules
change DEVICE, HWADDR, IPADDR in
/etc/sysconfig/network-scripts/ifcfg-eth0
restart network: service network restart
change host name:
in /etc/sysconfig/network
HOSTNAME=
write host file
/etc/hosts
generate ssh key
cd /root/.sshssh-keygen
then send the id_rsa.pub from all nodes(including master) to master
ssh-copy-id master#or send manuallycat ~/.ssh/id_rsa.pub | ssh hadoop@CDHNode1 'cat >> ~/.ssh/authorized_keys'
finally sent this authorized_keys from master to all of the nodes
scp -r authorized_keys cloudera@slave1...
理解rsa认证：
我有一对玉佩，我送给你其中一个玉佩，我自己拿着另一个玉佩，等我下次到你家的时候我给你家的保安看我这个玉佩就知道是我来了而不是小偷。
mysql
hadoop 本身需要数据库来维护，选择mysql来做这个事
yum install -y mysql-server mysql mysql-devel#/usr/bin/mysqladmin -u root 'old_password' 'new-password'#SET PASSWORD = Password('password');chkconfig mysqld onservice mysqld startcp /usr/my.cnf /etc/my.cnf
vi /etc/my.cnf
#bind-address = 127.0.0.1max_connections = 400 #(db_number*100+50)#hivecreate database hive DEFAULT CHARSET utf8 COLLATE utf8_general_ci;grant all privileges on hive.* to 'hive'@'master' identified by 'qwerqwer' with grant option;#activity monitorcreate database amon DEFAULT CHARSET utf8 COLLATE utf8_general_ci;grant all privileges on amon.* to 'amon'@'master' identified by 'qwerqwer' with grant option;#Huecreate database hue DEFAULT CHARSET utf8 COLLATE utf8_general_ci;grant all privileges on hue.* to 'hue'@'master' identified by 'qwerqwer' with grant option;#Oozie Servercreate database oozie_server DEFAULT CHARSET utf8 COLLATE utf8_general_ci;grant all privileges on oozie_server.* to 'oozie_server'@'master' identified by 'qwerqwer' with grant option;#smoncreate database smon DEFAULT CHARSET utf8 COLLATE utf8_general_ci;grant all privileges on smon.* to 'smon'@'master' identified by 'qwerqwer' with grant option;#hmoncreate database hmon DEFAULT CHARSET utf8 COLLATE utf8_general_ci;grant all privileges on hmon.* to 'hmon'@'master' identified by 'qwerqwer' with grant option;#hmoncreate database hmon DEFAULT CHARSET utf8 COLLATE utf8_general_ci;grant all privileges on hmon.* to 'hmon'@'master' identified by 'qwerqwer' with grant option;#rootgrant all privileges on *.* to 'root'@'master' identified by 'qwerqwer' with grant option;grant SELECT on *.* to 'root'@'%' identified by 'qwerqwer' with grant option;#show grantsshow grants;JDBC
yum install mysql-connector-java -ydownload cloudera maneger
download CM$ wget http://archive.cloudera.com/cm5/installer/latest/cloudera-manager-installer.bin$ chmod u+x cloudera-manager-installer.bin$ sudo ./cloudera-manager-installer.bin#and next next ...
or
#preparesu -c 'rpm -Uvh http://download.fedoraproject.org/pub/epel/5/i386/epel-release-5-4.noarch.rpm'#centoswget https://archive.cloudera.com/cm5/redhat/5/x86_64/cm/cloudera-manager.reposudo cp cloudera-manager.repo /etc/yum.repos.d/#ubuntuwget https://archive.cloudera.com/cm5/debian/squeeze/amd64/cm/cloudera.listsudo cp cloudera-manager.repo /etc/apt/sources.list.d/ sudo apt-get updatedo the next step in browser
open http://192.168.your.ip:7180 in browser
default account and password is admin, admin
specify hosts for your CDH cluster installation
Add your hostnames as defined in your /etc/hosts file of all servers.
master slave1 slave2
provide SSH login credentials
upload id_rsa.pub from base server
or
use same username password
on all the nodes
choose the services that you want to start on your cluster
CDH5
All Service
choose mysql as the database
Host name: master
Database type: mysql
Database name:
use name:
password:
start and stop
sudo service cloudera-scm-server startsudo service cloudera-scm-server stopsudo service cloudera-scm-server restartsudo service cloudera-scm-agent startsudo service cloudera-scm-agent stopsudo service cloudera-scm-agent restartsudo service cloudera-scm-agent statusadd to startup
in master
chkconfig --add cloudera-scm-serverchkconfig cloudera-scm-server onchkconfig --list cloudera-scm-serverchkconfig httpd on
in slaves
chkconfig --add cloudera-scm-agentchkconfig cloudera-scm-agent onchkconfig --list cloudera-scm-agenttest installing
testing the installRunning a MapReduce Job
Parcel installation
sudo -u hdfs hadoop jar /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar pi 10 100
Package installation
sudo -u hdfs hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar pi 10 100
View the results of running the job by selecting Activities > MAPREDUCE-1 Jobs from the top navigation bar:
