Hadoop_VmImage_Allhadoop_Echosystems

This link is all Hadoop_Echosystmes(Hadoop, Hbase, Hive, Pig, Zookeeper, Mahout, Flume, Sqoop).

This Image will start hadoop automatic  when you will open image in vmware.


Download this image from below link:

ftp://www.madmurk.com/pub/Hadoop.zip

USERNAME : tony
PASSWORD : 123

If you don't have vmplayer download vmplayer also blow link:

ftp://www.madmurk.com/pub/VMware-player-6.0.1-1379776.exe

README.TXT

Hadoop_single

=============

Hadoop MultiNode Install shell script

Cloudera’s Distribution including Apache Hadoop 4.4.0

Yarn version MultiNode Installation Shell script



Pleace download this two script of Multinode_Install.sh and Slave_Install.sh.



Just do before start run the script run this commands in namenode bellow:



ssh-keygen -t rsa

ssh-copy-id -i  ~/.ssh/id_rsa.pub $USER@tony.com

ssh-copy-id -i  ~/.ssh/id_rsa.pub $USER@tony1.com

ssh-copy-id -i  ~/.ssh/id_rsa.pub $USER@tony2.com



And type Bellow command in your ubuntu michine:



bash Multinode_Install.sh



Then it will ask NameNode ip address and hostname of NameNode give to example of bellow given type:



192.168.0.1 tony.com



Then it will ask number of DataNode You want to install in Your cluster give to example of bellow given type:

3



Then it will ask DataNode ip address and hostname of DataNodes give to example of bellow given type:



Note if you want to use NameNode as a DataNode give NameNode ipaddress and hostname include DataNode ipaddress and hostname:



192.168.0.1 tony.com

192.168.0.2 tony1.com

192.168.0.3 tony2.com



Set the environment path and hadoop home in /etc/environment

In path : /usr/local/had/hadoop/bin:/usr/local/hadoop/sbin

HADOOP_HOME="/usr/local/had/hadoop"



If hav any problem send mail to tony.ntw@gmail.com

Slave_Install.sh

#!/bin/bash
tar xzf hadoop-2.0.0-cdh4.4.0.tar.gz
mv hadoop-2.0.0-cdh4.4.0 hadoop
master=$(cat /etc/hosts | cut -f2 -s | head -1)
slave=$(cat /etc/hosts | cut -f2 -s | sed -n '2p')
cat pas.txt | sudo -S  mkdir /usr/local/had
cat pas.txt | sudo -S  mkdir /hadoop
cat pas.txt | sudo -S chown $USER:$GROUP /usr/local/had
cat pas.txt | sudo -S chown $USER:$GROUP /hadoop
sed 's/echo\ "This script is Deprecated. Instead use start-dfs.sh and mr-jobhistory-daemon.sh"/#echo\ "This script is Deprecated. Instead use start-dfs.sh and mr-jobhistory-daemon.sh"/g' hadoop/sbin/start-all.sh -i
echo 'if [ -f "${YARN_HOME}"/sbin/mr-jobhistory-daemon.sh ]; then' >> hadoop/sbin/start-all.sh
echo  '"${YARN_HOME}"/sbin/mr-jobhistory-daemon.sh --config $HADOOP_CONF_DIR start historyserver' >> hadoop/sbin/start-all.sh
echo 'fi' >> hadoop/sbin/start-all.sh
sed 's/echo\ "This script is Deprecated. Instead use stop-dfs.sh and stop-yarn.sh"/#echo\ "This script is Deprecated. Instead use stop-dfs.sh and stop-yarn.sh"/g' hadoop/sbin/stop-all.sh -i
echo 'if [ -f "${YARN_HOME}"/sbin/mr-jobhistory-daemon.sh ]; then' >> hadoop/sbin/stop-all.sh
echo  '"${YARN_HOME}"/sbin/mr-jobhistory-daemon.sh --config $HADOOP_CONF_DIR stop historyserver' >> hadoop/sbin/stop-all.sh
echo 'fi' >> hadoop/sbin/stop-all.sh
sed "s/<\/configuration>/<property>\n<name>fs.default.name<\/name>\n<value>hdfs:\/\/$master:8020<\/value>\n<\/property>\n<property>\n<name>hadoop.tmp.dir<\/name>\n<value>\/hadoop\/datastore-hadoop<\/value>\n<\/property>\n<\/configuration>/g" -i.bak hadoop/etc/hadoop/core-site.xml
cp hadoop/etc/hadoop/mapred-site.xml.template hadoop/etc/hadoop/mapred-site.xml
sed "s/<\/configuration>/<property>\n<name>mapreduce.framework.name<\/name>\n<value>yarn<\/value>\n<\/property>\n<\/configuration>/g" -i.bak hadoop/etc/hadoop/mapred-site.xml
sed "s/<\/configuration>/<property>\n<name>dfs.replication<\/name>\n<value>1<\/value>\n<\/property>\n<property>\n<name>dfs.permissions<\/name>\n<value>false<\/value>\n<\/property>\n<!-- Immediately exit safemode as soon as one DataNode checks in. On a multi-node cluster, these configurations must be removed.-->\n<property>\n<name>dfs.safemode.extension<\/name>\n<value>0<\/value>\n<\/property>\n<property>\n<name>dfs.safemode.min.datanodes<\/name>\n<value>1<\/value>\n<\/property>\n<\/configuration>/g" -i.bak hadoop/etc/hadoop/hdfs-site.xml
sed "s/<\/configuration>/<property>\n<name>yarn.resourcemanager.resource-tracker.address<\/name>\n<value>$master:8031<\/value>\n<\/property>\n<property>\n<name>yarn.resourcemanager.address<\/name>\n<value>$master:8032<\/value>\n<\/property>\n<property>\n<name>yarn.resourcemanager.scheduler.address<\/name>\n<value>$master:8030<\/value>\n<\/property>\n<property>\n<name>yarn.resourcemanager.admin.address<\/name>\n<value>$master:8033<\/value>\n<\/property>\n<property>\n<name>yarn.resourcemanager.webapp.address<\/name>\n<value>$master:8088<\/value>\n<\/property>\n<property>\n<name>yarn.nodemanager.aux-services<\/name>\n<value>mapreduce.shuffle<\/value>\n<\/property>\n<property>\n<name>yarn.nodemanager.aux-services.mapreduce_shuffle.class<\/name>\n<value>org.apache.hadoop.mapred.ShuffleHandler<\/value>\n<\/property>\n<\/configuration>/g" -i.bak hadoop/etc/hadoop/yarn-site.xml
sed 's/\/etc\/hadoop/\/usr\/local\/had\/hadoop\/etc\/hadoop/g' hadoop/etc/hadoop/hadoop-env.sh -i
echo 'export HADOOP_OPTS=-Djava.net.preferIPv4Stack=true' >> hadoop/etc/hadoop/hadoop-env.sh
echo 'export JAVA_HOME="/usr/lib/jvm/java-6-openjdk"' >> hadoop/etc/hadoop/hadoop-env.sh
cat /etc/hosts | cut  -f2 -s > hadoop/etc/hadoop/slaves
mv hadoop /usr/local/had

Multinode_Installation.sh

#!/bin/bash

echo "Enter MasterNode ipaddress and hostname::"$cond
read cond
echo "Enter SlaveNode ipaddress and hostname::"$cond1
read cond1
echo "Enter User password ::"$cond2
read cond2

if [  -z "$cond" ]
then
echo Your NameNode host configuration successfully skiped..................
else
echo "$cond" > a
echo Your NameNode host configuration successfully finced..................
fi

if [  -z "$cond1" ]
then
echo Your Slave hostname configuration successfully skiped..................
else
echo "$cond1" > b
cat a > host
cat b >> host
cat pas.txt | sudo -S mv host /etc/hosts
slav=$(echo "$cond1" | cut  -f2 -s)
cat pas.txt | sudo -S scp -r /etc/hosts $slav:/etc
rm a b
echo Your Slave hostname configuration successfully finced..................
fi
if [  -z "$cond2" ]
then
echo Your password file configuration successfully skiped..................
else
echo "$cond2" > pas.txt
echo Your password file configuration successfully finced..................
fi
scp -r pas.txt $slave:~
cat pas.txt | sudo -S  mkdir /usr/local/had
cat pas.txt | sudo -S  mkdir /hadoop
cat pas.txt | sudo -S chown $USER:$GROUP /usr/local/had
cat pas.txt | sudo -S chown $USER:$GROUP /hadoop
master=$(cat /etc/hosts | cut -f2 -s | head -1)
slave=$(cat /etc/hosts | cut -f2 -s | sed -n '2p')
wget http://archive.cloudera.com/cdh4/cdh/4/hadoop-2.0.0-cdh4.4.0.tar.gz
tar xzf hadoop-2.0.0-cdh4.4.0.tar.gz
mv hadoop-2.0.0-cdh4.4.0 hadoop

sed 's/echo\ "This script is Deprecated. Instead use start-dfs.sh and mr-jobhistory-daemon.sh"/#echo\ "This script is Deprecated. Instead use start-dfs.sh and mr-jobhistory-daemon.sh"/g' hadoop/sbin/start-all.sh -i
echo 'if [ -f "${YARN_HOME}"/sbin/mr-jobhistory-daemon.sh ]; then' >> hadoop/sbin/start-all.sh
echo  '"${YARN_HOME}"/sbin/mr-jobhistory-daemon.sh --config $HADOOP_CONF_DIR start historyserver' >> hadoop/sbin/start-all.sh
echo 'fi' >> hadoop/sbin/start-all.sh
sed 's/echo\ "This script is Deprecated. Instead use stop-dfs.sh and stop-yarn.sh"/#echo\ "This script is Deprecated. Instead use stop-dfs.sh and stop-yarn.sh"/g' hadoop/sbin/stop-all.sh -i
echo 'if [ -f "${YARN_HOME}"/sbin/mr-jobhistory-daemon.sh ]; then' >> hadoop/sbin/stop-all.sh
echo  '"${YARN_HOME}"/sbin/mr-jobhistory-daemon.sh --config $HADOOP_CONF_DIR stop historyserver' >> hadoop/sbin/stop-all.sh
echo 'fi' >> hadoop/sbin/stop-all.sh
sed "s/<\/configuration>/<property>\n<name>fs.default.name<\/name>\n<value>hdfs:\/\/$master:8020<\/value>\n<\/property>\n<property>\n<name>hadoop.tmp.dir<\/name>\n<value>\/hadoop\/datastore-hadoop<\/value>\n<\/property>\n<\/configuration>/g" -i.bak hadoop/etc/hadoop/core-site.xml
cp hadoop/etc/hadoop/mapred-site.xml.template hadoop/etc/hadoop/mapred-site.xml
sed "s/<\/configuration>/<property>\n<name>mapreduce.framework.name<\/name>\n<value>yarn<\/value>\n<\/property>\n<\/configuration>/g" -i.bak hadoop/etc/hadoop/mapred-site.xml
sed "s/<\/configuration>/<property>\n<name>dfs.replication<\/name>\n<value>1<\/value>\n<\/property>\n<property>\n<name>dfs.permissions<\/name>\n<value>false<\/value>\n<\/property>\n<!-- Immediately exit safemode as soon as one DataNode checks in. On a multi-node cluster, these configurations must be removed.-->\n<property>\n<name>dfs.safemode.extension<\/name>\n<value>0<\/value>\n<\/property>\n<property>\n<name>dfs.safemode.min.datanodes<\/name>\n<value>1<\/value>\n<\/property>\n<\/configuration>/g" -i.bak hadoop/etc/hadoop/hdfs-site.xml
sed "s/<\/configuration>/<property>\n<name>yarn.resourcemanager.resource-tracker.address<\/name>\n<value>$master:8031<\/value>\n<\/property>\n<property>\n<name>yarn.resourcemanager.address<\/name>\n<value>$master:8032<\/value>\n<\/property>\n<property>\n<name>yarn.resourcemanager.scheduler.address<\/name>\n<value>$master:8030<\/value>\n<\/property>\n<property>\n<name>yarn.resourcemanager.admin.address<\/name>\n<value>$master:8033<\/value>\n<\/property>\n<property>\n<name>yarn.resourcemanager.webapp.address<\/name>\n<value>$master:8088<\/value>\n<\/property>\n<property>\n<name>yarn.nodemanager.aux-services<\/name>\n<value>mapreduce.shuffle<\/value>\n<\/property>\n<property>\n<name>yarn.nodemanager.aux-services.mapreduce_shuffle.class<\/name>\n<value>org.apache.hadoop.mapred.ShuffleHandler<\/value>\n<\/property>\n<\/configuration>/g" -i.bak hadoop/etc/hadoop/yarn-site.xml
sed 's/\/etc\/hadoop/\/usr\/local\/had\/hadoop\/etc\/hadoop/g' hadoop/etc/hadoop/hadoop-env.sh -i
echo 'export HADOOP_OPTS=-Djava.net.preferIPv4Stack=true' >> hadoop/etc/hadoop/hadoop-env.sh
echo 'export JAVA_HOME="/usr/lib/jvm/java-6-openjdk"' >> hadoop/etc/hadoop/hadoop-env.sh
cat /etc/hosts | cut  -f2 -s > hadoop/etc/hadoop/slaves
mv hadoop /usr/local/had
scp -rv  hadoop-2.0.0-cdh4.4.0.tar.gz $slave:~
scp -rv Inst_Slave.sh $slave:~
ssh $slave sh Inst_Slave.sh