Testlink - Installation on AWS CentOS

Installation Steps

• Update the system :-

            $  sudo yum install epel-release -y

      $ sudo yum update -y && sudo shutdown -r now

     After install epel , restart the system to apply the updates.

• Install a Web Server -Apache :-

             $  sudo yum install httpd -y

       It is recommended to remove/disable the Apache default welcome page in production environments.

       $ sudo sed -i 's/^/#&/g' /etc/httpd/conf.d/welcome.conf

       Prevent Apache from listing web directory files to visitors:

       $ sudo sed -i "s/Options Indexes FollowSymLinks/Options FollowSymLinks/" /etc/httpd/conf/httpd.conf

       Start the Apache service and enable it to auto-start on boot

       $  sudo systemctl start httpd.service

       $  sudo systemctl enable httpd.service

• Install Mysql Database :-

        refer Mysql Installation

• Install PHP 7.1 and Some extensions :-

      TestLink requires PHP 5.5 or later. In order to get better performance, we will install PHP 7.1 and all the necessary extension for TestLink as follows,

      $ sudo rpm -Uvh https://mirror.webtatic.com/yum/el7/webtatic-release.rpm

      $ sudo yum install mod_php71w php71w-mysqlnd php71w-common php71w-gd php71w-ldap php71w-cli php71w-mcrypt php71w-xml -y

      We will need to modify PHP settings to meet TestLink’s requirements as follows:

      $ sudo cp /etc/php.ini /etc/php.ini.bak

      $ sudo sed -i "s/session.gc_maxlifetime = 1440/session.gc_maxlifetime = 2880/" /etc/php.ini

      $ sudo sed -i "s/max_execution_time = 30/max_execution_time = 120/" /etc/php.ini

• Install Testlink 1.9.16 :-

       Download the testlink and unzip into /var/www/html folder.

       $ sudo cd wget https://github.com/TestLinkOpenSourceTRMS/testlink-code/archive/1.9.16.tar.gz

       $ sudo tar -zxvf 1.9.16.tar.gz -C /var/www/html

       $ sudo chown -R apache:apache /var/www/html/testlink-code-1.9.16

       Create a custom configuration file for Testlink 

       $ sudo cp /var/www/html/testlink-code-1.9.16/custom_config.inc.php.example /var/www/html/testlink-code-1.9.16/custom_config.inc.php

       Modify the custom Testlink configuration file 

       $ sudo vi /var/www/html/testlink-code-1.9.16/custom_config.inc.php

       Find the below lines

       // $tlCfg->log_path = '/var/testlink-ga-testlink-code/logs/'; /* unix example */

       // $g_repositoryPath = '/var/testlink-ga-testlink-code/upload_area/'; /* unix example */

       Replace them with below lines

       $tlCfg->log_path = '/var/www/html/testlink-code-1.9.16/logs/';

       $g_repositoryPath = '/var/www/html/testlink-code-1.9.16/upload_area/';

       Create virtual Host for Testlink       

cat <<EOF | sudo tee -a /etc/httpd/conf.d/testlink.conf <VirtualHost *:80> ServerAdmin admin@example.com DocumentRoot /var/www/html/testlink-code-1.9.16/ ServerName testlink.example.com ServerAlias www.testlink.example.com <Directory /var/www/html/testlink-code-1.9.16/> Options FollowSymLinks AllowOverride All Order allow,deny allow from all </Directory> ErrorLog /var/log/httpd/testlink.example.com-error_log CustomLog /var/log/httpd/testlink.example.com-access_log common </VirtualHost> EOF

• Restart Apache in order to apply all above settings :-

           $ sudo systemctl restart httpd.service

• Finishing the installation :-

      When you visit http://xx.xxx.xxx.xx:80 

            you will get New Installation wizard, there you can do database setup by providing below details             

• Database Type: MySQL/MariaDB (5.6+ / 10.+)
• Database host: localhost
• Database name: testlink
• Table prefix: <LEAVE IT EMPTY>
• Database admin login: root
• Database admin password: <your-MariaDB-root-password>
• TestLink DB login: testlinkuser
• TestLink DB password: yourpassword

           Once these setup done, you can login with username & password.

MongoDB - Installation on CentOS

Installation Steps

Install MongoDB v4.0.10 Community Edition :-

• Configure the package management system (yum) :-

              Create a /etc/yum.repos.d/mongodb-org-4.0.repo file so that you can install MongoDB directly using yum.

       $ Sudo vi /etc/yum.repos.d/mongodb-org-4.0.repo                  

[mongodb-org-4.0]        name=MongoDB Repository        baseurl=https://repo.mongodb.org/yum/redhat/$releasever/mongodb-org/4.0/x86_64/        gpgcheck=1        enabled=1        gpgkey=https://www.mongodb.org/static/pgp/server-4.0.asc

• Install the MongoDB packages:-

       $ sudo yum install -y mongodb-org

• Directory Paths :-

             By default, MongoDB runs using the mongod user account and uses the following default directories:

       /var/lib/mongo (the data directory)

       /var/log/mongodb (the log directory)

• Configuration :-

              Update the below on /etc/mongod.conf file.

# network interfaces               net:               port: 27017               bindIp: 0.0.0.0   #Authentication                security:                authorization: enabled

              

• Start MongoDB :-

              $ sudo service mongod start

• Connect MongoDB :-

              $ mongo

Mysql Installation on AWS CentOS Machine

Installation Steps

• Install wget :- (Optional)

          $ sudo yum install wget

• Download mysql :-

          $ wget https://dev.mysql.com/get/mysql80-community-release-el7-3.noarch.rpm

• Verify Integrity Value :- 

          To check downloaded file is corrupted or not fire the below command

          $ md5sum mysql80-community-release-el7-3.noarch.rpm

As we cross checked both md5 values are same in above images, so now we verified that file was not corrupted. 

• Adding Downloaded package into yum Repository:-

          $ sudo rpm -ivh  mysql80-community-release-el7-3.noarch.rpm

• Installing Mysql:-

            $ sudo yum install mysql-server

• Check root user password :-

          During installation, the root user password is generated and stored in mysqld.log file.

          $ sudo grep 'temporary password' /var/log/mysqld.log

          So the password is 5xgQoryw2F(=    

          The default password policy requires 12 characters, with at least one uppercase letter, one lowercase letter, one number and one special character.

• Configuring MySql :-

          Run the security script to set new password of root user

          $ sudo mysql_secure_installation

          Press Y for all questions – in order to remove anonymous users, disallow remote root login, remove the test database and access to it, and reload the privilege tables.

          

• To Verify our Installation:-

          $ mysqladmin -u root -p version

Note: MySQL is automatically enabled to start at boot when it is installed. You can change that default behavior with

         $ sudo systemctl disable mysqld

• Starting MySql & Checking Status of MySql:-

           $ sudo systemctl start mysqld

           $  sudo systemctl status mysqld

           Installation successfully completed.

• Connecting MySql:-
  $ mysql -u username -p

FAQs

Problem

After Installation we faced issue that couldn't able to access mysql server remotely.

Solution

Note:

1. Add Public Ip on hosts file $ Sudo vi /etc/hosts 
2. Add bind-address=0.0.0.0 on $ Sudo vi /etc/my.cnf
3. Set host as '%' while creating user to access remote from anywhere.

Extracting json data using oracle pl/sql

------------------------------Example 1: 
create table t( doc_id  int,
 doc_details varchar2(1000)
);

--loading json format data:

insert into t values (1 , '{"id":1, "name" : "Jeff"}' );
insert into t values (2 , '{"id":2, "name" : "Jane", "status":"Gold"}' );
insert into t values (3 , '{"id":3, "name" : "Jill", "status":["Important","Gold"]}' );
insert into t values (4 , '{"name" : "John", "status":"Silver"}' );
commit;


--extracting the status key as fields:

select json_value(doc_details, '$.status') from t;


-----------------------Example 2 :

/* sample json data:

{"jobs":[{"name":"Wash Car","notes":[{"title":"Address","text":"1 High Street"},{"title":"Warning","text":"Scaffolding Required"}],"occupants":[{"name":"Mr Smith","gender":"Male"},{"name":"Mrs Smith","gender":"Female"}]},
{"name":"Wash Car","notes":[{"title":"Address","text":"1 Another Street"}],"occupants":[{"name":"Mr Jones","gender":"Male"},{"name":"Mrs Jones","gender":"Female"}]}]}

*/

-- From above sample data, we are going to extract data & store into three different tables.

create table jobs (
  job_id int,
  jb     varchar2(20)
);

create table notes (
  job_id int,
  title  varchar2(20),
  text   varchar2(30)
);

create table occupants (
  job_id int,
  name   varchar2(20),
  gender varchar2(30)
);
       
       
--insert operation performs here..       
insert all
  when rnj = 1 then into jobs values (row_number, job_name)
  when rnnm = 1 then into occupants values (row_number, name, gender)
  when rnnt = 1 then into notes values (row_number, note_title, note_text)
 
----Query starts 
with json as
  (select
'{"jobs":[{"name":"Wash Car","notes":[{"title":"Address","text":"1 High Street"},{"title":"Warning","text":"Scaffolding Required"}],"occupants":[{"name":"Mr Smith","gender":"Male"},{"name":"Mrs Smith","gender":"Female"}]},
{"name":"Wash Car","notes":[{"title":"Address","text":"1 Another Street"}],"occupants":[{"name":"Mr Jones","gender":"Male"},{"name":"Mrs Jones","gender":"Female"}]}]}'
    doc
  from dual
  ), jobs as (
select /*+ no_merge */row_number, job_name, notes, names
from json_table (
  ( select doc from json ) ,
   '$.jobs[*]' null on error
   columns ( row_number for ordinality,
    job_name varchar2 ( 20 ) path '$.name',
    notes varchar2(1000) format json path '$.notes'
    ,
    names varchar2(1000) format json path '$.occupants'
  )
)
)
  select row_number, job_name,
         note_id, note_title, note_text,
         name_id, name, gender,
         row_number() over (partition by row_number order by row_number) rnj,
         row_number() over (partition by row_number, note_id order by row_number, name_id) rnnt,
         row_number() over (partition by row_number, name_id order by row_number, note_id) rnnm
  from   jobs,
         json_table(notes, '$[*]'
          columns
           note_id    for ordinality,
           note_title varchar2(30) path '$.title',
           note_text  varchar2(30) path '$.text'
         ),
         json_table(names, '$[*]'
          columns
           name_id    for ordinality,
           name varchar2(30) path '$.name',
           gender  varchar2(30) path '$.gender'
         );
--Query ends




-- check the results on these tables once fired above query ,

select * from jobs;

JOB_ID  JB       
1       Wash Car 
2       Wash Car

select * from occupants;

JOB_ID  NAME       GENDER 
1       Mr Smith   Male   
1       Mrs Smith  Female 
2       Mr Jones   Male   
2       Mrs Jones  Female 

select * from notes;

JOB_ID  TITLE    TEXT                 
1       Address  1 High Street       
1       Warning  Scaffolding Required 
2       Address  1 Another Street  

MongoDB Installation on Ubuntu

Installation of Mongodb in Ubuntu(16.04):-

Step 1 : Import the public key used by the package management system.

sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 2930ADAE8CAF5059EE73BB4B58712A2291FA4AD5

Step 2 :  Create a list file for MongoDB.

echo "deb [ arch=amd64,arm64 ] https://repo.mongodb.org/apt/ubuntu xenial/mongodb-org/3.6 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-3.6.list

Step 3 : Reload local package database.

sudo apt-get update

Step 4 : Install the MongoDB packages.

sudo apt-get install -y mongodb-org

Step 5 : Install a specific release of MongoDB.

sudo apt-get install -y mongodb-org=3.6.5 mongodb-org-server=3.6.5 mongodb-org-shell=3.6.5 mongodb-org-mongos=3.6.5 mongodb-org-tools=3.6.5

Note:-

Data files:- The MongoDB instance stores its data files in /var/lib/mongodb
Log Files:-  its log files in /var/log/mongodb
To change Data/log file locations:-You can specify alternate log
and data file directories in /etc/mongod.conf. 

Installation completed successfully.

1. To Start mongodb:

sudo service mongod start

2. To Verify that MongoDB has started successfully:

cd /var/log/mongodb/mongod.log

3. To Stop MongoDB:

sudo service mongod stop

4. To Restart MongoDB:

sudo service mongod restart

5. To Begin using MongoDB:

mongo --host 127.0.0.1 --port 27017

Locust Master Slave Installation on Centos

Prequisite :
1. Install python
To check installed version of Python  python -V

sudo yum install python-pip
pip --version
sudo pip install --upgrade pip
sudo pip install locustio
locust --help

Note : Locust installation is same for master and slave , use the below command to make the machine as master and slave .Copy  load testing script in all machines

To start the Master 

locust -f loadtest.py --master --host=master-ip  --port 8089
[2018-08-27 16:19:10,216] /INFO/locust.main: Starting web monitor at *:8089

[2018-08-27 16:19:10,219] /INFO/locust.main: Starting Locust 0.8.1

To start the slave 

 locust -f loadtest.py --slave --master-host=master-ip

Sample script :loadtest,py

***********************************************************
#!/usr/bin/python

import socket
import time
import datetime
import random
from locust import Locust, events, task, TaskSet

class SimpleClient(object):

    def __init__(self):
        self.host="ip address"
        self.port=9003
        self.counter=1
        self.lat = 26.590706
        self.lon= 74.913704
    def execute(self,name):
        start_time = time.time()
        try:

            sock=socket.socket(socket.AF_INET, socket.SOCK_STREAM)
            sock.connect((self.host,self.port))
            dec_lat = random.random()/100
            dec_lon = random.random()/100
            x=datetime.datetime.now()
            data="#Tamil,"+str(x.strftime("%Y%m%d%H%M%S"))+","+ str(self.lat+dec_lat)+","+str(self.lon+dec_lon)+","+str(self.counter)+"^\n";
            sock.sendall(data)
            self.counter +=1
        except Exception as e:
            total_time = int((time.time() - start_time) * 1000)
            events.request_failure.fire(request_type="execute", name=name, response_time=total_time, exception=e)
        else:
            total_time = int((time.time() - start_time) * 1000)
            events.request_success.fire(request_type="execute", name=name, response_time=total_time, response_length=0)


class SimpleTasks(TaskSet):

    @task
    def simple_task(self):
        self.client.execute('IP address:portno')

class SimpleUser(Locust):
    def __init__(self, *args, **kwargs):
        super(Locust, self).__init__(*args, **kwargs)
        self.client=SimpleClient()

    task_set=SimpleTasks
    min_wait=1000

    max_wait=10000

***********************************************************

you can check the UI of locust from browser

In the example I have started locust on 2 slaves you can 2 slaves in the below image

http://masterip:8089

Apache hadoop installation on Ubuntu ( AWS EC2)

Prerequisite
1. Install Java 

sudo apt-get update

sudo apt-get install default-jdk

To choose java version if u have many
sudo update-alternatives --config java

$vi $HOME/.bashrc
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-amd64
export PATH=$PATH:$JAVA_HOME/bin

2.Adding a dedicated Hadoop system user

$ sudo addgroup hadoop
$ sudo adduser --ingroup hadoop hduser


Configuring SSH


user@ubuntu:~$ su - hduser
hduser@ubuntu:~$ ssh-keygen -t rsa -P ""

hduser@ubuntu:~$ cat $HOME/.ssh/id_rsa.pub >> $HOME/.ssh/authorized_keys

hduser@ubuntu:~$ ssh localhost


Note : If ssh loclhost is not working open new terminal and check


Disabling IPv6

$vi /etc/sysctl.conf

net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1

You have to reboot your machine
 Check : $ cat /proc/sys/net/ipv6/conf/all/disable_ipv6
 0 means IPv6 is enabled, a value of 1 means disabled



****************************************
Hadoop Installation

  cd /usr/local

 hduser@ip-privateip:/usr/local$ sudo  wget https://archive.apache.org/dist/hadoop/core/hadoop-1.0.3/hadoop-1.0.3.tar.gz
 [sudo] password for hduser:
--2018-03-08 06:36:50--  https://archive.apache.org/dist/hadoop/core/hadoop-1.0.3/hadoop-1.0.3.tar.gz
Resolving archive.apache.org (archive.apache.org)... 163.172.17.199
Connecting to archive.apache.org (archive.apache.org)|163.172.17.199|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 62428860 (60M) [application/x-gzip]
Saving to: ‘hadoop-1.0.3.tar.gz’

hadoop-1.0.3.tar.gz                       100%[=====================================================================================>]  59.54M  10.7MB/s    in 6.8s

2018-03-08 06:36:57 (8.82 MB/s) - ‘hadoop-1.0.3.tar.gz’ saved [62428860/62428860]

hduser@ip-:/usr/local$ ls
bin  etc  games  hadoop-1.0.3.tar.gz  include  lib  man  sbin  share  src
hduser@ip-:/usr/local$ sudo tar xzf hadoop-1.0.3.tar.gz
hduser@ip-:/usr/local$ sudo mv hadoop-1.0.3 hadoop
hduser@ip-:/usr/local$ sudo chown -R hduser:hadoop hadoop
hduser@ip-:/usr/local$


Update $HOME/.bashrc

***************************************************************
hduser@ip-private ip:/usr/local/hadoop$ cat $HOME/.bashrc
# ~/.bashrc: executed by bash(1) for non-login shells.
# see /usr/share/doc/bash/examples/startup-files (in the package bash-doc)
# for examples

# If not running interactively, don't do anything
case $- in
    *i*) ;;
      *) return;;
esac

# don't put duplicate lines or lines starting with space in the history.
# See bash(1) for more options
HISTCONTROL=ignoreboth

# append to the history file, don't overwrite it
shopt -s histappend

# for setting history length see HISTSIZE and HISTFILESIZE in bash(1)
HISTSIZE=1000
HISTFILESIZE=2000

# check the window size after each command and, if necessary,
# update the values of LINES and COLUMNS.
shopt -s checkwinsize

# If set, the pattern "**" used in a pathname expansion context will
# match all files and zero or more directories and subdirectories.
#shopt -s globstar

# make less more friendly for non-text input files, see lesspipe(1)
[ -x /usr/bin/lesspipe ] && eval "$(SHELL=/bin/sh lesspipe)"

# set variable identifying the chroot you work in (used in the prompt below)
if [ -z "${debian_chroot:-}" ] && [ -r /etc/debian_chroot ]; then
    debian_chroot=$(cat /etc/debian_chroot)
fi

# set a fancy prompt (non-color, unless we know we "want" color)
case "$TERM" in
    xterm-color|*-256color) color_prompt=yes;;
esac

# uncomment for a colored prompt, if the terminal has the capability; turned
# off by default to not distract the user: the focus in a terminal window
# should be on the output of commands, not on the prompt
#force_color_prompt=yes

if [ -n "$force_color_prompt" ]; then
    if [ -x /usr/bin/tput ] && tput setaf 1 >&/dev/null; then
        # We have color support; assume it's compliant with Ecma-48
        # (ISO/IEC-6429). (Lack of such support is extremely rare, and such
        # a case would tend to support setf rather than setaf.)
        color_prompt=yes
    else
        color_prompt=
    fi
fi

if [ "$color_prompt" = yes ]; then
    PS1='${debian_chroot:+($debian_chroot)}\[\033[01;32m\]\u@\h\[\033[00m\]:\[\033[01;34m\]\w\[\033[00m\]\$ '
else
    PS1='${debian_chroot:+($debian_chroot)}\u@\h:\w\$ '
fi
unset color_prompt force_color_prompt

# If this is an xterm set the title to user@host:dir
case "$TERM" in
xterm*|rxvt*)
    PS1="\[\e]0;${debian_chroot:+($debian_chroot)}\u@\h: \w\a\]$PS1"
    ;;
*)
    ;;
esac

# enable color support of ls and also add handy aliases
if [ -x /usr/bin/dircolors ]; then
    test -r ~/.dircolors && eval "$(dircolors -b ~/.dircolors)" || eval "$(dircolors -b)"
    alias ls='ls --color=auto'
    #alias dir='dir --color=auto'
    #alias vdir='vdir --color=auto'

    alias grep='grep --color=auto'
    alias fgrep='fgrep --color=auto'
    alias egrep='egrep --color=auto'
fi

# colored GCC warnings and errors
#export GCC_COLORS='error=01;31:warning=01;35:note=01;36:caret=01;32:locus=01:quote=01'

# some more ls aliases
alias ll='ls -alF'
alias la='ls -A'
alias l='ls -CF'

# Add an "alert" alias for long running commands.  Use like so:
#   sleep 10; alert
alias alert='notify-send --urgency=low -i "$([ $? = 0 ] && echo terminal || echo error)" "$(history|tail -n1|sed -e '\''s/^\s*[0-9]\+\s*//;s/[;&|]\s*alert$//'\'')"'

# Alias definitions.
# You may want to put all your additions into a separate file like
# ~/.bash_aliases, instead of adding them here directly.
# See /usr/share/doc/bash-doc/examples in the bash-doc package.

if [ -f ~/.bash_aliases ]; then
    . ~/.bash_aliases
fi

# enable programmable completion features (you don't need to enable
# this, if it's already enabled in /etc/bash.bashrc and /etc/profile
# sources /etc/bash.bashrc).
if ! shopt -oq posix; then
  if [ -f /usr/share/bash-completion/bash_completion ]; then
    . /usr/share/bash-completion/bash_completion
  elif [ -f /etc/bash_completion ]; then
    . /etc/bash_completion
  fi
fi
# Set Hadoop-related environment variables
export HADOOP_HOME=/usr/local/hadoop
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-amd64
# Some convenient aliases and functions for running Hadoop-related commands
unalias fs &> /dev/null
alias fs="hadoop fs"
unalias hls &> /dev/null
alias hls="fs -ls"

lzohead () {
    hadoop fs -cat $1 | lzop -dc | head -1000 | less
}

export HADOOP_INSTALL=/usr/local/hadoop
export PATH=$PATH:$HADOOP_INSTALL/bin
export PATH=$PATH:$HADOOP_INSTALL/sbin
export HADOOP_MAPRED_HOME=$HADOOP_INSTALL
export HADOOP_COMMON_HOME=$HADOOP_INSTALL
export HADOOP_HDFS_HOME=$HADOOP_INSTALL
export YARN_HOME=$HADOOP_INSTALL
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_INSTALL/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_INSTALL/lib"
# Add Hadoop bin/ directory to PATH
export FLUME_HOME=/home/hduser/apache-flume-1.8.0-bin/
export PATH=$PATH:$HADOOP_HOME/bin:$JAVA_HOME/bin



***************************************************************
Configuration

1.hadoop-env.sh
***************************************************************
hduser@ip:/usr/local/hadoop/conf$ cat hadoop-env.sh
# Set Hadoop-specific environment variables here.

# The only required environment variable is JAVA_HOME.  All others are
# optional.  When running a distributed configuration it is best to
# set JAVA_HOME in this file, so that it is correctly defined on
# remote nodes.

# The java implementation to use.  Required.
# export JAVA_HOME=/usr/lib/j2sdk1.5-sun

# Extra Java CLASSPATH elements.  Optional.
# export HADOOP_CLASSPATH=

# The maximum amount of heap to use, in MB. Default is 1000.
# export HADOOP_HEAPSIZE=2000

# Extra Java runtime options.  Empty by default.
# export HADOOP_OPTS=-server

# Command specific options appended to HADOOP_OPTS when specified
export HADOOP_NAMENODE_OPTS="-Dcom.sun.management.jmxremote $HADOOP_NAMENODE_OPTS"
export HADOOP_SECONDARYNAMENODE_OPTS="-Dcom.sun.management.jmxremote $HADOOP_SECONDARYNAMENODE_OPTS"
export HADOOP_DATANODE_OPTS="-Dcom.sun.management.jmxremote $HADOOP_DATANODE_OPTS"
export HADOOP_BALANCER_OPTS="-Dcom.sun.management.jmxremote $HADOOP_BALANCER_OPTS"
export HADOOP_JOBTRACKER_OPTS="-Dcom.sun.management.jmxremote $HADOOP_JOBTRACKER_OPTS"
# export HADOOP_TASKTRACKER_OPTS=
# The following applies to multiple commands (fs, dfs, fsck, distcp etc)
# export HADOOP_CLIENT_OPTS

# Extra ssh options.  Empty by default.
# export HADOOP_SSH_OPTS="-o ConnectTimeout=1 -o SendEnv=HADOOP_CONF_DIR"

# Where log files are stored.  $HADOOP_HOME/logs by default.
# export HADOOP_LOG_DIR=${HADOOP_HOME}/logs

# File naming remote slave hosts.  $HADOOP_HOME/conf/slaves by default.
# export HADOOP_SLAVES=${HADOOP_HOME}/conf/slaves

# host:path where hadoop code should be rsync'd from.  Unset by default.
# export HADOOP_MASTER=master:/home/$USER/src/hadoop

# Seconds to sleep between slave commands.  Unset by default.  This
# can be useful in large clusters, where, e.g., slave rsyncs can
# otherwise arrive faster than the master can service them.
# export HADOOP_SLAVE_SLEEP=0.1

# The directory where pid files are stored. /tmp by default.
# export HADOOP_PID_DIR=/var/hadoop/pids

# A string representing this instance of hadoop. $USER by default.
# export HADOOP_IDENT_STRING=$USER

# The scheduling priority for daemon processes.  See 'man nice'.
# export HADOOP_NICENESS=10
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-amd64

***************************************************************
2.conf/*-site.xml
create the directory and set the required ownerships and permissions:

$ sudo mkdir -p /app/hadoop/tmp
$ sudo chown hduser:hadoop /app/hadoop/tmp
# ...and if you want to tighten up security, chmod from 755 to 750...
$ sudo chmod 750 /app/hadoop/tmp

conf/core-site.xml:

hduser@ip:/usr/local/hadoop/conf$ cat core-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>
<property>
  <name>hadoop.tmp.dir</name>
  <value>/app/hadoop/tmp</value>
  <description>A base for other temporary directories.</description>
</property>

<property>
  <name>fs.default.name</name>
  <value>hdfs://privateip:9000</value>
  <description>The name of the default file system.  A URI whose
  scheme and authority determine the FileSystem implementation.  The
  uri's scheme determines the config property (fs.SCHEME.impl) naming
  the FileSystem implementation class.  The uri's authority is used to
  determine the host, port, etc. for a filesystem.</description>
</property>
<property>
    <name>dfs.datanode.address</name>
    <value>0.0.0.0:50010</value>
  </property>

  <property>
    <name>dfs.datanode.http.address</name>
    <value>0.0.0.0:50075</value>
  </property>

  <property>
    <name>dfs.http.address</name>
    <value>0.0.0.0:50070</value>
    <description>The name of the default file system.  Either the
       literal string "local" or a host:port for NDFS.
    </description>
    <final>true</final>
  </property>

  <property>
    <name>dfs.datanode.ipc.address</name>
    <value>0.0.0.0:50020</value>
    <description>
      The datanode ipc server address and port.
      If the port is 0 then the server will start on a free port.
    </description>
  </property>
</configuration>
hduser@ip:/usr/local/hadoop/conf$

***************************************************************

3.conf/mapred-site.xml:

hduser@ip-:/usr/local/hadoop/conf$ cat mapred-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>
<property>
  <name>mapred.job.tracker</name>
  <value>privateip:54311</value>
  <description>The host and port that the MapReduce job tracker runs
  at.  If "local", then jobs are run in-process as a single map
  and reduce task.
  </description>
</property>
</configuration>
hduser@ip:/usr/local/hadoop/conf$

***************************************************************

4.conf/hdfs-site.xml:
hduser@ip:/usr/local/hadoop/conf$ cat hdfs-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>
<!--<property>
    <name>dfs.data.dir</name>
    <value>/usr/local/hadoop/tmp/dfs/name/data</value>
    <final>true</final>
    </property>
    <property>
    <name>dfs.name.dir</name>
    <value>/usr/local/hadoop/tmp/dfs/name</value>
    <final>true</final>
</property>-->
<property>
  <name>dfs.replication</name>
  <value>1</value>
  <description>Default block replication.
  The actual number of replications can be specified when the file is created.
  The default is used if replication is not specified in create time.
  </description>
</property>
<property>
   <name>dfs.namenode.name.dir</name>
   <value>file:/usr/local/hadoop_store/hdfs/namenode</value>
 </property>
 <property>
   <name>dfs.datanode.data.dir</name>
   <value>file:/usr/local/hadoop_store/hdfs/datanode</value>
 </property>
 <property>
    <name>dfs.webhdfs.enabled</name>
    <value>true</value>
 </property>
</configuration>
hduser@ip:/usr/local/hadoop/conf$


***************************************************************

5.Formatting the HDFS filesystem via the NameNode
hduser@ubuntu:~$ /usr/local/hadoop/bin/hadoop namenode -format

/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = ip-.ap-south-1.compute.internal/privateip
STARTUP_MSG:   args = [-format]
STARTUP_MSG:   version = 1.0.3
STARTUP_MSG:   build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r 1335192; compiled by 'hortonfo' on Tue May  8 20:31:25 UTC 2012

18/03/08 06:51:13 INFO util.GSet: VM type       = 64-bit
18/03/08 06:51:13 INFO util.GSet: 2% max memory = 19.33375 MB
18/03/08 06:51:13 INFO util.GSet: capacity      = 2^21 = 2097152 entries
18/03/08 06:51:13 INFO util.GSet: recommended=2097152, actual=2097152
18/03/08 06:51:14 INFO namenode.FSNamesystem: fsOwner=hduser
18/03/08 06:51:14 INFO namenode.FSNamesystem: supergroup=supergroup
18/03/08 06:51:14 INFO namenode.FSNamesystem: isPermissionEnabled=true
18/03/08 06:51:14 INFO namenode.FSNamesystem: dfs.block.invalidate.limit=100
18/03/08 06:51:14 INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)
18/03/08 06:51:14 INFO namenode.NameNode: Caching file names occuring more than 10 times
18/03/08 06:51:14 INFO common.Storage: Image file of size 112 saved in 0 seconds.
18/03/08 06:51:14 INFO common.Storage: Storage directory /app/hadoop/tmp/dfs/name has been successfully formatted.
18/03/08 06:51:14 INFO namenode.NameNode: SHUTDOWN_MSG:

SHUTDOWN_MSG: Shutting down NameNode at ip-\.ap-south-1.compute.internal/privateip
***************************************************************


OTHER PROPERTIES file
hduser@ip:/usr/local/hadoop/conf$ cat masters
private ip
hduser@ip:/usr/local/hadoop/conf$ cat slaves
private ip
hduser@ip:/usr/local/hadoop/conf$

hduser@ip-:/usr/local/hadoop/conf$ cat /etc/hosts
#127.0.0.1 localhost
privateip ec2-publicip.ap-south-1.compute.amazonaws.com
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
hduser@ip:/usr/local/hadoop/conf$


***************************************************************
To start single node cluster

hduser@ubuntu:~$ /usr/local/hadoop/bin/start-all.sh

hduser@ubuntu:/usr/local/hadoop$ jps
TaskTracker
JobTracker
DataNode
SecondaryNameNode
Jps
NameNode
***************************************************************
netstat to check  Hadoop onfigured ports.

hduser@ubuntu:~$ sudo netstat -plten | grep java

Stopping single node cluster

hduser@ubuntu:~$ /usr/local/hadoop/bin/stop-all.sh
***************************************************************
***************************************************************

Command line 
To view data in hdfs

hduser@ip/usr/local/hadoop/bin$ hadoop dfs -cat hdfs://ip:9000/filename

To list data in hdfs

 hadoop dfs -ls hdfs://ip:9000/2018-06-1

To delete folder in hdfs

hadoop dfs -rmr hdfs://ip:9000/2018-06-19

To view from external

http:publicip:50070