Locust Master Slave Installation on Centos

Prequisite :
1. Install python
To check installed version of Python  python -V

sudo yum install python-pip
pip --version
sudo pip install --upgrade pip
sudo pip install locustio
locust --help

Note : Locust installation is same for master and slave , use the below command to make the machine as master and slave .Copy  load testing script in all machines

To start the Master 

locust -f loadtest.py --master --host=master-ip  --port 8089
[2018-08-27 16:19:10,216] /INFO/locust.main: Starting web monitor at *:8089

[2018-08-27 16:19:10,219] /INFO/locust.main: Starting Locust 0.8.1

To start the slave 

 locust -f loadtest.py --slave --master-host=master-ip

Sample script :loadtest,py

***********************************************************
#!/usr/bin/python

import socket
import time
import datetime
import random
from locust import Locust, events, task, TaskSet

class SimpleClient(object):

    def __init__(self):
        self.host="ip address"
        self.port=9003
        self.counter=1
        self.lat = 26.590706
        self.lon= 74.913704
    def execute(self,name):
        start_time = time.time()
        try:

            sock=socket.socket(socket.AF_INET, socket.SOCK_STREAM)
            sock.connect((self.host,self.port))
            dec_lat = random.random()/100
            dec_lon = random.random()/100
            x=datetime.datetime.now()
            data="#Tamil,"+str(x.strftime("%Y%m%d%H%M%S"))+","+ str(self.lat+dec_lat)+","+str(self.lon+dec_lon)+","+str(self.counter)+"^\n";
            sock.sendall(data)
            self.counter +=1
        except Exception as e:
            total_time = int((time.time() - start_time) * 1000)
            events.request_failure.fire(request_type="execute", name=name, response_time=total_time, exception=e)
        else:
            total_time = int((time.time() - start_time) * 1000)
            events.request_success.fire(request_type="execute", name=name, response_time=total_time, response_length=0)


class SimpleTasks(TaskSet):

    @task
    def simple_task(self):
        self.client.execute('IP address:portno')

class SimpleUser(Locust):
    def __init__(self, *args, **kwargs):
        super(Locust, self).__init__(*args, **kwargs)
        self.client=SimpleClient()

    task_set=SimpleTasks
    min_wait=1000

    max_wait=10000

***********************************************************

you can check the UI of locust from browser

In the example I have started locust on 2 slaves you can 2 slaves in the below image

http://masterip:8089

Apache hadoop installation on Ubuntu ( AWS EC2)

Prerequisite
1. Install Java 

sudo apt-get update

sudo apt-get install default-jdk

To choose java version if u have many
sudo update-alternatives --config java

$vi $HOME/.bashrc
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-amd64
export PATH=$PATH:$JAVA_HOME/bin

2.Adding a dedicated Hadoop system user

$ sudo addgroup hadoop
$ sudo adduser --ingroup hadoop hduser


Configuring SSH


user@ubuntu:~$ su - hduser
hduser@ubuntu:~$ ssh-keygen -t rsa -P ""

hduser@ubuntu:~$ cat $HOME/.ssh/id_rsa.pub >> $HOME/.ssh/authorized_keys

hduser@ubuntu:~$ ssh localhost


Note : If ssh loclhost is not working open new terminal and check


Disabling IPv6

$vi /etc/sysctl.conf

net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1

You have to reboot your machine
 Check : $ cat /proc/sys/net/ipv6/conf/all/disable_ipv6
 0 means IPv6 is enabled, a value of 1 means disabled



****************************************
Hadoop Installation

  cd /usr/local

 hduser@ip-privateip:/usr/local$ sudo  wget https://archive.apache.org/dist/hadoop/core/hadoop-1.0.3/hadoop-1.0.3.tar.gz
 [sudo] password for hduser:
--2018-03-08 06:36:50--  https://archive.apache.org/dist/hadoop/core/hadoop-1.0.3/hadoop-1.0.3.tar.gz
Resolving archive.apache.org (archive.apache.org)... 163.172.17.199
Connecting to archive.apache.org (archive.apache.org)|163.172.17.199|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 62428860 (60M) [application/x-gzip]
Saving to: ‘hadoop-1.0.3.tar.gz’

hadoop-1.0.3.tar.gz                       100%[=====================================================================================>]  59.54M  10.7MB/s    in 6.8s

2018-03-08 06:36:57 (8.82 MB/s) - ‘hadoop-1.0.3.tar.gz’ saved [62428860/62428860]

hduser@ip-:/usr/local$ ls
bin  etc  games  hadoop-1.0.3.tar.gz  include  lib  man  sbin  share  src
hduser@ip-:/usr/local$ sudo tar xzf hadoop-1.0.3.tar.gz
hduser@ip-:/usr/local$ sudo mv hadoop-1.0.3 hadoop
hduser@ip-:/usr/local$ sudo chown -R hduser:hadoop hadoop
hduser@ip-:/usr/local$


Update $HOME/.bashrc

***************************************************************
hduser@ip-private ip:/usr/local/hadoop$ cat $HOME/.bashrc
# ~/.bashrc: executed by bash(1) for non-login shells.
# see /usr/share/doc/bash/examples/startup-files (in the package bash-doc)
# for examples

# If not running interactively, don't do anything
case $- in
    *i*) ;;
      *) return;;
esac

# don't put duplicate lines or lines starting with space in the history.
# See bash(1) for more options
HISTCONTROL=ignoreboth

# append to the history file, don't overwrite it
shopt -s histappend

# for setting history length see HISTSIZE and HISTFILESIZE in bash(1)
HISTSIZE=1000
HISTFILESIZE=2000

# check the window size after each command and, if necessary,
# update the values of LINES and COLUMNS.
shopt -s checkwinsize

# If set, the pattern "**" used in a pathname expansion context will
# match all files and zero or more directories and subdirectories.
#shopt -s globstar

# make less more friendly for non-text input files, see lesspipe(1)
[ -x /usr/bin/lesspipe ] && eval "$(SHELL=/bin/sh lesspipe)"

# set variable identifying the chroot you work in (used in the prompt below)
if [ -z "${debian_chroot:-}" ] && [ -r /etc/debian_chroot ]; then
    debian_chroot=$(cat /etc/debian_chroot)
fi

# set a fancy prompt (non-color, unless we know we "want" color)
case "$TERM" in
    xterm-color|*-256color) color_prompt=yes;;
esac

# uncomment for a colored prompt, if the terminal has the capability; turned
# off by default to not distract the user: the focus in a terminal window
# should be on the output of commands, not on the prompt
#force_color_prompt=yes

if [ -n "$force_color_prompt" ]; then
    if [ -x /usr/bin/tput ] && tput setaf 1 >&/dev/null; then
        # We have color support; assume it's compliant with Ecma-48
        # (ISO/IEC-6429). (Lack of such support is extremely rare, and such
        # a case would tend to support setf rather than setaf.)
        color_prompt=yes
    else
        color_prompt=
    fi
fi

if [ "$color_prompt" = yes ]; then
    PS1='${debian_chroot:+($debian_chroot)}\[\033[01;32m\]\u@\h\[\033[00m\]:\[\033[01;34m\]\w\[\033[00m\]\$ '
else
    PS1='${debian_chroot:+($debian_chroot)}\u@\h:\w\$ '
fi
unset color_prompt force_color_prompt

# If this is an xterm set the title to user@host:dir
case "$TERM" in
xterm*|rxvt*)
    PS1="\[\e]0;${debian_chroot:+($debian_chroot)}\u@\h: \w\a\]$PS1"
    ;;
*)
    ;;
esac

# enable color support of ls and also add handy aliases
if [ -x /usr/bin/dircolors ]; then
    test -r ~/.dircolors && eval "$(dircolors -b ~/.dircolors)" || eval "$(dircolors -b)"
    alias ls='ls --color=auto'
    #alias dir='dir --color=auto'
    #alias vdir='vdir --color=auto'

    alias grep='grep --color=auto'
    alias fgrep='fgrep --color=auto'
    alias egrep='egrep --color=auto'
fi

# colored GCC warnings and errors
#export GCC_COLORS='error=01;31:warning=01;35:note=01;36:caret=01;32:locus=01:quote=01'

# some more ls aliases
alias ll='ls -alF'
alias la='ls -A'
alias l='ls -CF'

# Add an "alert" alias for long running commands.  Use like so:
#   sleep 10; alert
alias alert='notify-send --urgency=low -i "$([ $? = 0 ] && echo terminal || echo error)" "$(history|tail -n1|sed -e '\''s/^\s*[0-9]\+\s*//;s/[;&|]\s*alert$//'\'')"'

# Alias definitions.
# You may want to put all your additions into a separate file like
# ~/.bash_aliases, instead of adding them here directly.
# See /usr/share/doc/bash-doc/examples in the bash-doc package.

if [ -f ~/.bash_aliases ]; then
    . ~/.bash_aliases
fi

# enable programmable completion features (you don't need to enable
# this, if it's already enabled in /etc/bash.bashrc and /etc/profile
# sources /etc/bash.bashrc).
if ! shopt -oq posix; then
  if [ -f /usr/share/bash-completion/bash_completion ]; then
    . /usr/share/bash-completion/bash_completion
  elif [ -f /etc/bash_completion ]; then
    . /etc/bash_completion
  fi
fi
# Set Hadoop-related environment variables
export HADOOP_HOME=/usr/local/hadoop
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-amd64
# Some convenient aliases and functions for running Hadoop-related commands
unalias fs &> /dev/null
alias fs="hadoop fs"
unalias hls &> /dev/null
alias hls="fs -ls"

lzohead () {
    hadoop fs -cat $1 | lzop -dc | head -1000 | less
}

export HADOOP_INSTALL=/usr/local/hadoop
export PATH=$PATH:$HADOOP_INSTALL/bin
export PATH=$PATH:$HADOOP_INSTALL/sbin
export HADOOP_MAPRED_HOME=$HADOOP_INSTALL
export HADOOP_COMMON_HOME=$HADOOP_INSTALL
export HADOOP_HDFS_HOME=$HADOOP_INSTALL
export YARN_HOME=$HADOOP_INSTALL
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_INSTALL/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_INSTALL/lib"
# Add Hadoop bin/ directory to PATH
export FLUME_HOME=/home/hduser/apache-flume-1.8.0-bin/
export PATH=$PATH:$HADOOP_HOME/bin:$JAVA_HOME/bin



***************************************************************
Configuration

1.hadoop-env.sh
***************************************************************
hduser@ip:/usr/local/hadoop/conf$ cat hadoop-env.sh
# Set Hadoop-specific environment variables here.

# The only required environment variable is JAVA_HOME.  All others are
# optional.  When running a distributed configuration it is best to
# set JAVA_HOME in this file, so that it is correctly defined on
# remote nodes.

# The java implementation to use.  Required.
# export JAVA_HOME=/usr/lib/j2sdk1.5-sun

# Extra Java CLASSPATH elements.  Optional.
# export HADOOP_CLASSPATH=

# The maximum amount of heap to use, in MB. Default is 1000.
# export HADOOP_HEAPSIZE=2000

# Extra Java runtime options.  Empty by default.
# export HADOOP_OPTS=-server

# Command specific options appended to HADOOP_OPTS when specified
export HADOOP_NAMENODE_OPTS="-Dcom.sun.management.jmxremote $HADOOP_NAMENODE_OPTS"
export HADOOP_SECONDARYNAMENODE_OPTS="-Dcom.sun.management.jmxremote $HADOOP_SECONDARYNAMENODE_OPTS"
export HADOOP_DATANODE_OPTS="-Dcom.sun.management.jmxremote $HADOOP_DATANODE_OPTS"
export HADOOP_BALANCER_OPTS="-Dcom.sun.management.jmxremote $HADOOP_BALANCER_OPTS"
export HADOOP_JOBTRACKER_OPTS="-Dcom.sun.management.jmxremote $HADOOP_JOBTRACKER_OPTS"
# export HADOOP_TASKTRACKER_OPTS=
# The following applies to multiple commands (fs, dfs, fsck, distcp etc)
# export HADOOP_CLIENT_OPTS

# Extra ssh options.  Empty by default.
# export HADOOP_SSH_OPTS="-o ConnectTimeout=1 -o SendEnv=HADOOP_CONF_DIR"

# Where log files are stored.  $HADOOP_HOME/logs by default.
# export HADOOP_LOG_DIR=${HADOOP_HOME}/logs

# File naming remote slave hosts.  $HADOOP_HOME/conf/slaves by default.
# export HADOOP_SLAVES=${HADOOP_HOME}/conf/slaves

# host:path where hadoop code should be rsync'd from.  Unset by default.
# export HADOOP_MASTER=master:/home/$USER/src/hadoop

# Seconds to sleep between slave commands.  Unset by default.  This
# can be useful in large clusters, where, e.g., slave rsyncs can
# otherwise arrive faster than the master can service them.
# export HADOOP_SLAVE_SLEEP=0.1

# The directory where pid files are stored. /tmp by default.
# export HADOOP_PID_DIR=/var/hadoop/pids

# A string representing this instance of hadoop. $USER by default.
# export HADOOP_IDENT_STRING=$USER

# The scheduling priority for daemon processes.  See 'man nice'.
# export HADOOP_NICENESS=10
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-amd64

***************************************************************
2.conf/*-site.xml
create the directory and set the required ownerships and permissions:

$ sudo mkdir -p /app/hadoop/tmp
$ sudo chown hduser:hadoop /app/hadoop/tmp
# ...and if you want to tighten up security, chmod from 755 to 750...
$ sudo chmod 750 /app/hadoop/tmp

conf/core-site.xml:

hduser@ip:/usr/local/hadoop/conf$ cat core-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>
<property>
  <name>hadoop.tmp.dir</name>
  <value>/app/hadoop/tmp</value>
  <description>A base for other temporary directories.</description>
</property>

<property>
  <name>fs.default.name</name>
  <value>hdfs://privateip:9000</value>
  <description>The name of the default file system.  A URI whose
  scheme and authority determine the FileSystem implementation.  The
  uri's scheme determines the config property (fs.SCHEME.impl) naming
  the FileSystem implementation class.  The uri's authority is used to
  determine the host, port, etc. for a filesystem.</description>
</property>
<property>
    <name>dfs.datanode.address</name>
    <value>0.0.0.0:50010</value>
  </property>

  <property>
    <name>dfs.datanode.http.address</name>
    <value>0.0.0.0:50075</value>
  </property>

  <property>
    <name>dfs.http.address</name>
    <value>0.0.0.0:50070</value>
    <description>The name of the default file system.  Either the
       literal string "local" or a host:port for NDFS.
    </description>
    <final>true</final>
  </property>

  <property>
    <name>dfs.datanode.ipc.address</name>
    <value>0.0.0.0:50020</value>
    <description>
      The datanode ipc server address and port.
      If the port is 0 then the server will start on a free port.
    </description>
  </property>
</configuration>
hduser@ip:/usr/local/hadoop/conf$

***************************************************************

3.conf/mapred-site.xml:

hduser@ip-:/usr/local/hadoop/conf$ cat mapred-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>
<property>
  <name>mapred.job.tracker</name>
  <value>privateip:54311</value>
  <description>The host and port that the MapReduce job tracker runs
  at.  If "local", then jobs are run in-process as a single map
  and reduce task.
  </description>
</property>
</configuration>
hduser@ip:/usr/local/hadoop/conf$

***************************************************************

4.conf/hdfs-site.xml:
hduser@ip:/usr/local/hadoop/conf$ cat hdfs-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>
<!--<property>
    <name>dfs.data.dir</name>
    <value>/usr/local/hadoop/tmp/dfs/name/data</value>
    <final>true</final>
    </property>
    <property>
    <name>dfs.name.dir</name>
    <value>/usr/local/hadoop/tmp/dfs/name</value>
    <final>true</final>
</property>-->
<property>
  <name>dfs.replication</name>
  <value>1</value>
  <description>Default block replication.
  The actual number of replications can be specified when the file is created.
  The default is used if replication is not specified in create time.
  </description>
</property>
<property>
   <name>dfs.namenode.name.dir</name>
   <value>file:/usr/local/hadoop_store/hdfs/namenode</value>
 </property>
 <property>
   <name>dfs.datanode.data.dir</name>
   <value>file:/usr/local/hadoop_store/hdfs/datanode</value>
 </property>
 <property>
    <name>dfs.webhdfs.enabled</name>
    <value>true</value>
 </property>
</configuration>
hduser@ip:/usr/local/hadoop/conf$


***************************************************************

5.Formatting the HDFS filesystem via the NameNode
hduser@ubuntu:~$ /usr/local/hadoop/bin/hadoop namenode -format

/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = ip-.ap-south-1.compute.internal/privateip
STARTUP_MSG:   args = [-format]
STARTUP_MSG:   version = 1.0.3
STARTUP_MSG:   build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r 1335192; compiled by 'hortonfo' on Tue May  8 20:31:25 UTC 2012

18/03/08 06:51:13 INFO util.GSet: VM type       = 64-bit
18/03/08 06:51:13 INFO util.GSet: 2% max memory = 19.33375 MB
18/03/08 06:51:13 INFO util.GSet: capacity      = 2^21 = 2097152 entries
18/03/08 06:51:13 INFO util.GSet: recommended=2097152, actual=2097152
18/03/08 06:51:14 INFO namenode.FSNamesystem: fsOwner=hduser
18/03/08 06:51:14 INFO namenode.FSNamesystem: supergroup=supergroup
18/03/08 06:51:14 INFO namenode.FSNamesystem: isPermissionEnabled=true
18/03/08 06:51:14 INFO namenode.FSNamesystem: dfs.block.invalidate.limit=100
18/03/08 06:51:14 INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)
18/03/08 06:51:14 INFO namenode.NameNode: Caching file names occuring more than 10 times
18/03/08 06:51:14 INFO common.Storage: Image file of size 112 saved in 0 seconds.
18/03/08 06:51:14 INFO common.Storage: Storage directory /app/hadoop/tmp/dfs/name has been successfully formatted.
18/03/08 06:51:14 INFO namenode.NameNode: SHUTDOWN_MSG:

SHUTDOWN_MSG: Shutting down NameNode at ip-\.ap-south-1.compute.internal/privateip
***************************************************************


OTHER PROPERTIES file
hduser@ip:/usr/local/hadoop/conf$ cat masters
private ip
hduser@ip:/usr/local/hadoop/conf$ cat slaves
private ip
hduser@ip:/usr/local/hadoop/conf$

hduser@ip-:/usr/local/hadoop/conf$ cat /etc/hosts
#127.0.0.1 localhost
privateip ec2-publicip.ap-south-1.compute.amazonaws.com
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
hduser@ip:/usr/local/hadoop/conf$


***************************************************************
To start single node cluster

hduser@ubuntu:~$ /usr/local/hadoop/bin/start-all.sh

hduser@ubuntu:/usr/local/hadoop$ jps
TaskTracker
JobTracker
DataNode
SecondaryNameNode
Jps
NameNode
***************************************************************
netstat to check  Hadoop onfigured ports.

hduser@ubuntu:~$ sudo netstat -plten | grep java

Stopping single node cluster

hduser@ubuntu:~$ /usr/local/hadoop/bin/stop-all.sh
***************************************************************
***************************************************************

Command line 
To view data in hdfs

hduser@ip/usr/local/hadoop/bin$ hadoop dfs -cat hdfs://ip:9000/filename

To list data in hdfs

 hadoop dfs -ls hdfs://ip:9000/2018-06-1

To delete folder in hdfs

hadoop dfs -rmr hdfs://ip:9000/2018-06-19

To view from external

http:publicip:50070