Extracting json data using oracle pl/sql

------------------------------Example 1: 
create table t( doc_id  int,
 doc_details varchar2(1000)
);

--loading json format data:

insert into t values (1 , '{"id":1, "name" : "Jeff"}' );
insert into t values (2 , '{"id":2, "name" : "Jane", "status":"Gold"}' );
insert into t values (3 , '{"id":3, "name" : "Jill", "status":["Important","Gold"]}' );
insert into t values (4 , '{"name" : "John", "status":"Silver"}' );
commit;


--extracting the status key as fields:

select json_value(doc_details, '$.status') from t;


-----------------------Example 2 :

/* sample json data:

{"jobs":[{"name":"Wash Car","notes":[{"title":"Address","text":"1 High Street"},{"title":"Warning","text":"Scaffolding Required"}],"occupants":[{"name":"Mr Smith","gender":"Male"},{"name":"Mrs Smith","gender":"Female"}]},
{"name":"Wash Car","notes":[{"title":"Address","text":"1 Another Street"}],"occupants":[{"name":"Mr Jones","gender":"Male"},{"name":"Mrs Jones","gender":"Female"}]}]}

*/

-- From above sample data, we are going to extract data & store into three different tables.

create table jobs (
  job_id int,
  jb     varchar2(20)
);

create table notes (
  job_id int,
  title  varchar2(20),
  text   varchar2(30)
);

create table occupants (
  job_id int,
  name   varchar2(20),
  gender varchar2(30)
);
       
       
--insert operation performs here..       
insert all
  when rnj = 1 then into jobs values (row_number, job_name)
  when rnnm = 1 then into occupants values (row_number, name, gender)
  when rnnt = 1 then into notes values (row_number, note_title, note_text)
 
----Query starts 
with json as
  (select
'{"jobs":[{"name":"Wash Car","notes":[{"title":"Address","text":"1 High Street"},{"title":"Warning","text":"Scaffolding Required"}],"occupants":[{"name":"Mr Smith","gender":"Male"},{"name":"Mrs Smith","gender":"Female"}]},
{"name":"Wash Car","notes":[{"title":"Address","text":"1 Another Street"}],"occupants":[{"name":"Mr Jones","gender":"Male"},{"name":"Mrs Jones","gender":"Female"}]}]}'
    doc
  from dual
  ), jobs as (
select /*+ no_merge */row_number, job_name, notes, names
from json_table (
  ( select doc from json ) ,
   '$.jobs[*]' null on error
   columns ( row_number for ordinality,
    job_name varchar2 ( 20 ) path '$.name',
    notes varchar2(1000) format json path '$.notes'
    ,
    names varchar2(1000) format json path '$.occupants'
  )
)
)
  select row_number, job_name,
         note_id, note_title, note_text,
         name_id, name, gender,
         row_number() over (partition by row_number order by row_number) rnj,
         row_number() over (partition by row_number, note_id order by row_number, name_id) rnnt,
         row_number() over (partition by row_number, name_id order by row_number, note_id) rnnm
  from   jobs,
         json_table(notes, '$[*]'
          columns
           note_id    for ordinality,
           note_title varchar2(30) path '$.title',
           note_text  varchar2(30) path '$.text'
         ),
         json_table(names, '$[*]'
          columns
           name_id    for ordinality,
           name varchar2(30) path '$.name',
           gender  varchar2(30) path '$.gender'
         );
--Query ends




-- check the results on these tables once fired above query ,

select * from jobs;

JOB_ID  JB       
1       Wash Car 
2       Wash Car

select * from occupants;

JOB_ID  NAME       GENDER 
1       Mr Smith   Male   
1       Mrs Smith  Female 
2       Mr Jones   Male   
2       Mrs Jones  Female 

select * from notes;

JOB_ID  TITLE    TEXT                 
1       Address  1 High Street       
1       Warning  Scaffolding Required 
2       Address  1 Another Street  

MongoDB Installation on Ubuntu

Installation of Mongodb in Ubuntu(16.04):-

Step 1 : Import the public key used by the package management system.

sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 2930ADAE8CAF5059EE73BB4B58712A2291FA4AD5

Step 2 :  Create a list file for MongoDB.

echo "deb [ arch=amd64,arm64 ] https://repo.mongodb.org/apt/ubuntu xenial/mongodb-org/3.6 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-3.6.list

Step 3 : Reload local package database.

sudo apt-get update

Step 4 : Install the MongoDB packages.

sudo apt-get install -y mongodb-org

Step 5 : Install a specific release of MongoDB.

sudo apt-get install -y mongodb-org=3.6.5 mongodb-org-server=3.6.5 mongodb-org-shell=3.6.5 mongodb-org-mongos=3.6.5 mongodb-org-tools=3.6.5

Note:-

Data files:- The MongoDB instance stores its data files in /var/lib/mongodb
Log Files:-  its log files in /var/log/mongodb
To change Data/log file locations:-You can specify alternate log
and data file directories in /etc/mongod.conf. 

Installation completed successfully.

1. To Start mongodb:

sudo service mongod start

2. To Verify that MongoDB has started successfully:

cd /var/log/mongodb/mongod.log

3. To Stop MongoDB:

sudo service mongod stop

4. To Restart MongoDB:

sudo service mongod restart

5. To Begin using MongoDB:

mongo --host 127.0.0.1 --port 27017

Locust Master Slave Installation on Centos

Prequisite :
1. Install python
To check installed version of Python  python -V

sudo yum install python-pip
pip --version
sudo pip install --upgrade pip
sudo pip install locustio
locust --help

Note : Locust installation is same for master and slave , use the below command to make the machine as master and slave .Copy  load testing script in all machines

To start the Master 

locust -f loadtest.py --master --host=master-ip  --port 8089
[2018-08-27 16:19:10,216] /INFO/locust.main: Starting web monitor at *:8089

[2018-08-27 16:19:10,219] /INFO/locust.main: Starting Locust 0.8.1

To start the slave 

 locust -f loadtest.py --slave --master-host=master-ip

Sample script :loadtest,py

***********************************************************
#!/usr/bin/python

import socket
import time
import datetime
import random
from locust import Locust, events, task, TaskSet

class SimpleClient(object):

    def __init__(self):
        self.host="ip address"
        self.port=9003
        self.counter=1
        self.lat = 26.590706
        self.lon= 74.913704
    def execute(self,name):
        start_time = time.time()
        try:

            sock=socket.socket(socket.AF_INET, socket.SOCK_STREAM)
            sock.connect((self.host,self.port))
            dec_lat = random.random()/100
            dec_lon = random.random()/100
            x=datetime.datetime.now()
            data="#Tamil,"+str(x.strftime("%Y%m%d%H%M%S"))+","+ str(self.lat+dec_lat)+","+str(self.lon+dec_lon)+","+str(self.counter)+"^\n";
            sock.sendall(data)
            self.counter +=1
        except Exception as e:
            total_time = int((time.time() - start_time) * 1000)
            events.request_failure.fire(request_type="execute", name=name, response_time=total_time, exception=e)
        else:
            total_time = int((time.time() - start_time) * 1000)
            events.request_success.fire(request_type="execute", name=name, response_time=total_time, response_length=0)


class SimpleTasks(TaskSet):

    @task
    def simple_task(self):
        self.client.execute('IP address:portno')

class SimpleUser(Locust):
    def __init__(self, *args, **kwargs):
        super(Locust, self).__init__(*args, **kwargs)
        self.client=SimpleClient()

    task_set=SimpleTasks
    min_wait=1000

    max_wait=10000

***********************************************************

you can check the UI of locust from browser

In the example I have started locust on 2 slaves you can 2 slaves in the below image

http://masterip:8089

Apache hadoop installation on Ubuntu ( AWS EC2)

Prerequisite
1. Install Java 

sudo apt-get update

sudo apt-get install default-jdk

To choose java version if u have many
sudo update-alternatives --config java

$vi $HOME/.bashrc
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-amd64
export PATH=$PATH:$JAVA_HOME/bin

2.Adding a dedicated Hadoop system user

$ sudo addgroup hadoop
$ sudo adduser --ingroup hadoop hduser


Configuring SSH


user@ubuntu:~$ su - hduser
hduser@ubuntu:~$ ssh-keygen -t rsa -P ""

hduser@ubuntu:~$ cat $HOME/.ssh/id_rsa.pub >> $HOME/.ssh/authorized_keys

hduser@ubuntu:~$ ssh localhost


Note : If ssh loclhost is not working open new terminal and check


Disabling IPv6

$vi /etc/sysctl.conf

net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1

You have to reboot your machine
 Check : $ cat /proc/sys/net/ipv6/conf/all/disable_ipv6
 0 means IPv6 is enabled, a value of 1 means disabled



****************************************
Hadoop Installation

  cd /usr/local

 hduser@ip-privateip:/usr/local$ sudo  wget https://archive.apache.org/dist/hadoop/core/hadoop-1.0.3/hadoop-1.0.3.tar.gz
 [sudo] password for hduser:
--2018-03-08 06:36:50--  https://archive.apache.org/dist/hadoop/core/hadoop-1.0.3/hadoop-1.0.3.tar.gz
Resolving archive.apache.org (archive.apache.org)... 163.172.17.199
Connecting to archive.apache.org (archive.apache.org)|163.172.17.199|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 62428860 (60M) [application/x-gzip]
Saving to: ‘hadoop-1.0.3.tar.gz’

hadoop-1.0.3.tar.gz                       100%[=====================================================================================>]  59.54M  10.7MB/s    in 6.8s

2018-03-08 06:36:57 (8.82 MB/s) - ‘hadoop-1.0.3.tar.gz’ saved [62428860/62428860]

hduser@ip-:/usr/local$ ls
bin  etc  games  hadoop-1.0.3.tar.gz  include  lib  man  sbin  share  src
hduser@ip-:/usr/local$ sudo tar xzf hadoop-1.0.3.tar.gz
hduser@ip-:/usr/local$ sudo mv hadoop-1.0.3 hadoop
hduser@ip-:/usr/local$ sudo chown -R hduser:hadoop hadoop
hduser@ip-:/usr/local$


Update $HOME/.bashrc

***************************************************************
hduser@ip-private ip:/usr/local/hadoop$ cat $HOME/.bashrc
# ~/.bashrc: executed by bash(1) for non-login shells.
# see /usr/share/doc/bash/examples/startup-files (in the package bash-doc)
# for examples

# If not running interactively, don't do anything
case $- in
    *i*) ;;
      *) return;;
esac

# don't put duplicate lines or lines starting with space in the history.
# See bash(1) for more options
HISTCONTROL=ignoreboth

# append to the history file, don't overwrite it
shopt -s histappend

# for setting history length see HISTSIZE and HISTFILESIZE in bash(1)
HISTSIZE=1000
HISTFILESIZE=2000

# check the window size after each command and, if necessary,
# update the values of LINES and COLUMNS.
shopt -s checkwinsize

# If set, the pattern "**" used in a pathname expansion context will
# match all files and zero or more directories and subdirectories.
#shopt -s globstar

# make less more friendly for non-text input files, see lesspipe(1)
[ -x /usr/bin/lesspipe ] && eval "$(SHELL=/bin/sh lesspipe)"

# set variable identifying the chroot you work in (used in the prompt below)
if [ -z "${debian_chroot:-}" ] && [ -r /etc/debian_chroot ]; then
    debian_chroot=$(cat /etc/debian_chroot)
fi

# set a fancy prompt (non-color, unless we know we "want" color)
case "$TERM" in
    xterm-color|*-256color) color_prompt=yes;;
esac

# uncomment for a colored prompt, if the terminal has the capability; turned
# off by default to not distract the user: the focus in a terminal window
# should be on the output of commands, not on the prompt
#force_color_prompt=yes

if [ -n "$force_color_prompt" ]; then
    if [ -x /usr/bin/tput ] && tput setaf 1 >&/dev/null; then
        # We have color support; assume it's compliant with Ecma-48
        # (ISO/IEC-6429). (Lack of such support is extremely rare, and such
        # a case would tend to support setf rather than setaf.)
        color_prompt=yes
    else
        color_prompt=
    fi
fi

if [ "$color_prompt" = yes ]; then
    PS1='${debian_chroot:+($debian_chroot)}\[\033[01;32m\]\u@\h\[\033[00m\]:\[\033[01;34m\]\w\[\033[00m\]\$ '
else
    PS1='${debian_chroot:+($debian_chroot)}\u@\h:\w\$ '
fi
unset color_prompt force_color_prompt

# If this is an xterm set the title to user@host:dir
case "$TERM" in
xterm*|rxvt*)
    PS1="\[\e]0;${debian_chroot:+($debian_chroot)}\u@\h: \w\a\]$PS1"
    ;;
*)
    ;;
esac

# enable color support of ls and also add handy aliases
if [ -x /usr/bin/dircolors ]; then
    test -r ~/.dircolors && eval "$(dircolors -b ~/.dircolors)" || eval "$(dircolors -b)"
    alias ls='ls --color=auto'
    #alias dir='dir --color=auto'
    #alias vdir='vdir --color=auto'

    alias grep='grep --color=auto'
    alias fgrep='fgrep --color=auto'
    alias egrep='egrep --color=auto'
fi

# colored GCC warnings and errors
#export GCC_COLORS='error=01;31:warning=01;35:note=01;36:caret=01;32:locus=01:quote=01'

# some more ls aliases
alias ll='ls -alF'
alias la='ls -A'
alias l='ls -CF'

# Add an "alert" alias for long running commands.  Use like so:
#   sleep 10; alert
alias alert='notify-send --urgency=low -i "$([ $? = 0 ] && echo terminal || echo error)" "$(history|tail -n1|sed -e '\''s/^\s*[0-9]\+\s*//;s/[;&|]\s*alert$//'\'')"'

# Alias definitions.
# You may want to put all your additions into a separate file like
# ~/.bash_aliases, instead of adding them here directly.
# See /usr/share/doc/bash-doc/examples in the bash-doc package.

if [ -f ~/.bash_aliases ]; then
    . ~/.bash_aliases
fi

# enable programmable completion features (you don't need to enable
# this, if it's already enabled in /etc/bash.bashrc and /etc/profile
# sources /etc/bash.bashrc).
if ! shopt -oq posix; then
  if [ -f /usr/share/bash-completion/bash_completion ]; then
    . /usr/share/bash-completion/bash_completion
  elif [ -f /etc/bash_completion ]; then
    . /etc/bash_completion
  fi
fi
# Set Hadoop-related environment variables
export HADOOP_HOME=/usr/local/hadoop
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-amd64
# Some convenient aliases and functions for running Hadoop-related commands
unalias fs &> /dev/null
alias fs="hadoop fs"
unalias hls &> /dev/null
alias hls="fs -ls"

lzohead () {
    hadoop fs -cat $1 | lzop -dc | head -1000 | less
}

export HADOOP_INSTALL=/usr/local/hadoop
export PATH=$PATH:$HADOOP_INSTALL/bin
export PATH=$PATH:$HADOOP_INSTALL/sbin
export HADOOP_MAPRED_HOME=$HADOOP_INSTALL
export HADOOP_COMMON_HOME=$HADOOP_INSTALL
export HADOOP_HDFS_HOME=$HADOOP_INSTALL
export YARN_HOME=$HADOOP_INSTALL
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_INSTALL/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_INSTALL/lib"
# Add Hadoop bin/ directory to PATH
export FLUME_HOME=/home/hduser/apache-flume-1.8.0-bin/
export PATH=$PATH:$HADOOP_HOME/bin:$JAVA_HOME/bin



***************************************************************
Configuration

1.hadoop-env.sh
***************************************************************
hduser@ip:/usr/local/hadoop/conf$ cat hadoop-env.sh
# Set Hadoop-specific environment variables here.

# The only required environment variable is JAVA_HOME.  All others are
# optional.  When running a distributed configuration it is best to
# set JAVA_HOME in this file, so that it is correctly defined on
# remote nodes.

# The java implementation to use.  Required.
# export JAVA_HOME=/usr/lib/j2sdk1.5-sun

# Extra Java CLASSPATH elements.  Optional.
# export HADOOP_CLASSPATH=

# The maximum amount of heap to use, in MB. Default is 1000.
# export HADOOP_HEAPSIZE=2000

# Extra Java runtime options.  Empty by default.
# export HADOOP_OPTS=-server

# Command specific options appended to HADOOP_OPTS when specified
export HADOOP_NAMENODE_OPTS="-Dcom.sun.management.jmxremote $HADOOP_NAMENODE_OPTS"
export HADOOP_SECONDARYNAMENODE_OPTS="-Dcom.sun.management.jmxremote $HADOOP_SECONDARYNAMENODE_OPTS"
export HADOOP_DATANODE_OPTS="-Dcom.sun.management.jmxremote $HADOOP_DATANODE_OPTS"
export HADOOP_BALANCER_OPTS="-Dcom.sun.management.jmxremote $HADOOP_BALANCER_OPTS"
export HADOOP_JOBTRACKER_OPTS="-Dcom.sun.management.jmxremote $HADOOP_JOBTRACKER_OPTS"
# export HADOOP_TASKTRACKER_OPTS=
# The following applies to multiple commands (fs, dfs, fsck, distcp etc)
# export HADOOP_CLIENT_OPTS

# Extra ssh options.  Empty by default.
# export HADOOP_SSH_OPTS="-o ConnectTimeout=1 -o SendEnv=HADOOP_CONF_DIR"

# Where log files are stored.  $HADOOP_HOME/logs by default.
# export HADOOP_LOG_DIR=${HADOOP_HOME}/logs

# File naming remote slave hosts.  $HADOOP_HOME/conf/slaves by default.
# export HADOOP_SLAVES=${HADOOP_HOME}/conf/slaves

# host:path where hadoop code should be rsync'd from.  Unset by default.
# export HADOOP_MASTER=master:/home/$USER/src/hadoop

# Seconds to sleep between slave commands.  Unset by default.  This
# can be useful in large clusters, where, e.g., slave rsyncs can
# otherwise arrive faster than the master can service them.
# export HADOOP_SLAVE_SLEEP=0.1

# The directory where pid files are stored. /tmp by default.
# export HADOOP_PID_DIR=/var/hadoop/pids

# A string representing this instance of hadoop. $USER by default.
# export HADOOP_IDENT_STRING=$USER

# The scheduling priority for daemon processes.  See 'man nice'.
# export HADOOP_NICENESS=10
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-amd64

***************************************************************
2.conf/*-site.xml
create the directory and set the required ownerships and permissions:

$ sudo mkdir -p /app/hadoop/tmp
$ sudo chown hduser:hadoop /app/hadoop/tmp
# ...and if you want to tighten up security, chmod from 755 to 750...
$ sudo chmod 750 /app/hadoop/tmp

conf/core-site.xml:

hduser@ip:/usr/local/hadoop/conf$ cat core-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>
<property>
  <name>hadoop.tmp.dir</name>
  <value>/app/hadoop/tmp</value>
  <description>A base for other temporary directories.</description>
</property>

<property>
  <name>fs.default.name</name>
  <value>hdfs://privateip:9000</value>
  <description>The name of the default file system.  A URI whose
  scheme and authority determine the FileSystem implementation.  The
  uri's scheme determines the config property (fs.SCHEME.impl) naming
  the FileSystem implementation class.  The uri's authority is used to
  determine the host, port, etc. for a filesystem.</description>
</property>
<property>
    <name>dfs.datanode.address</name>
    <value>0.0.0.0:50010</value>
  </property>

  <property>
    <name>dfs.datanode.http.address</name>
    <value>0.0.0.0:50075</value>
  </property>

  <property>
    <name>dfs.http.address</name>
    <value>0.0.0.0:50070</value>
    <description>The name of the default file system.  Either the
       literal string "local" or a host:port for NDFS.
    </description>
    <final>true</final>
  </property>

  <property>
    <name>dfs.datanode.ipc.address</name>
    <value>0.0.0.0:50020</value>
    <description>
      The datanode ipc server address and port.
      If the port is 0 then the server will start on a free port.
    </description>
  </property>
</configuration>
hduser@ip:/usr/local/hadoop/conf$

***************************************************************

3.conf/mapred-site.xml:

hduser@ip-:/usr/local/hadoop/conf$ cat mapred-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>
<property>
  <name>mapred.job.tracker</name>
  <value>privateip:54311</value>
  <description>The host and port that the MapReduce job tracker runs
  at.  If "local", then jobs are run in-process as a single map
  and reduce task.
  </description>
</property>
</configuration>
hduser@ip:/usr/local/hadoop/conf$

***************************************************************

4.conf/hdfs-site.xml:
hduser@ip:/usr/local/hadoop/conf$ cat hdfs-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>
<!--<property>
    <name>dfs.data.dir</name>
    <value>/usr/local/hadoop/tmp/dfs/name/data</value>
    <final>true</final>
    </property>
    <property>
    <name>dfs.name.dir</name>
    <value>/usr/local/hadoop/tmp/dfs/name</value>
    <final>true</final>
</property>-->
<property>
  <name>dfs.replication</name>
  <value>1</value>
  <description>Default block replication.
  The actual number of replications can be specified when the file is created.
  The default is used if replication is not specified in create time.
  </description>
</property>
<property>
   <name>dfs.namenode.name.dir</name>
   <value>file:/usr/local/hadoop_store/hdfs/namenode</value>
 </property>
 <property>
   <name>dfs.datanode.data.dir</name>
   <value>file:/usr/local/hadoop_store/hdfs/datanode</value>
 </property>
 <property>
    <name>dfs.webhdfs.enabled</name>
    <value>true</value>
 </property>
</configuration>
hduser@ip:/usr/local/hadoop/conf$


***************************************************************

5.Formatting the HDFS filesystem via the NameNode
hduser@ubuntu:~$ /usr/local/hadoop/bin/hadoop namenode -format

/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = ip-.ap-south-1.compute.internal/privateip
STARTUP_MSG:   args = [-format]
STARTUP_MSG:   version = 1.0.3
STARTUP_MSG:   build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r 1335192; compiled by 'hortonfo' on Tue May  8 20:31:25 UTC 2012

18/03/08 06:51:13 INFO util.GSet: VM type       = 64-bit
18/03/08 06:51:13 INFO util.GSet: 2% max memory = 19.33375 MB
18/03/08 06:51:13 INFO util.GSet: capacity      = 2^21 = 2097152 entries
18/03/08 06:51:13 INFO util.GSet: recommended=2097152, actual=2097152
18/03/08 06:51:14 INFO namenode.FSNamesystem: fsOwner=hduser
18/03/08 06:51:14 INFO namenode.FSNamesystem: supergroup=supergroup
18/03/08 06:51:14 INFO namenode.FSNamesystem: isPermissionEnabled=true
18/03/08 06:51:14 INFO namenode.FSNamesystem: dfs.block.invalidate.limit=100
18/03/08 06:51:14 INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)
18/03/08 06:51:14 INFO namenode.NameNode: Caching file names occuring more than 10 times
18/03/08 06:51:14 INFO common.Storage: Image file of size 112 saved in 0 seconds.
18/03/08 06:51:14 INFO common.Storage: Storage directory /app/hadoop/tmp/dfs/name has been successfully formatted.
18/03/08 06:51:14 INFO namenode.NameNode: SHUTDOWN_MSG:

SHUTDOWN_MSG: Shutting down NameNode at ip-\.ap-south-1.compute.internal/privateip
***************************************************************


OTHER PROPERTIES file
hduser@ip:/usr/local/hadoop/conf$ cat masters
private ip
hduser@ip:/usr/local/hadoop/conf$ cat slaves
private ip
hduser@ip:/usr/local/hadoop/conf$

hduser@ip-:/usr/local/hadoop/conf$ cat /etc/hosts
#127.0.0.1 localhost
privateip ec2-publicip.ap-south-1.compute.amazonaws.com
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
hduser@ip:/usr/local/hadoop/conf$


***************************************************************
To start single node cluster

hduser@ubuntu:~$ /usr/local/hadoop/bin/start-all.sh

hduser@ubuntu:/usr/local/hadoop$ jps
TaskTracker
JobTracker
DataNode
SecondaryNameNode
Jps
NameNode
***************************************************************
netstat to check  Hadoop onfigured ports.

hduser@ubuntu:~$ sudo netstat -plten | grep java

Stopping single node cluster

hduser@ubuntu:~$ /usr/local/hadoop/bin/stop-all.sh
***************************************************************
***************************************************************

Command line 
To view data in hdfs

hduser@ip/usr/local/hadoop/bin$ hadoop dfs -cat hdfs://ip:9000/filename

To list data in hdfs

 hadoop dfs -ls hdfs://ip:9000/2018-06-1

To delete folder in hdfs

hadoop dfs -rmr hdfs://ip:9000/2018-06-19

To view from external

http:publicip:50070

Developing First Angular 5 and Above App

Installation:

1.  Install node.js (https://nodejs.org/en/download/)
2. Install Visual Studio Code(https://code.visualstudio.com/download)

Creating first Angular App

• Create a Workspace
• Open Visual Studio Code and Navigate to the Terminal , open your Workspace and run

E:\Workspaces\AngularWorkspace>npm install -g @angular/cli

• This will install the angular libraries in ur node_modules folder

C:\Users\hp\AppData\Roaming\npm\node_modules

• Create your app using the comand 

E:\Workspaces\AngularWorkspace>ng new AngularFirst

• Navigate into your Project folder

E:\Workspaces\AngularWorkspace>cd AngularFirst

• We have created our App and we can run it using the below command

E:\Workspaces\AngularWorkspace\AngularFirst>ng serve --open

• ng serve will deploy our application in the node server. and --open is optional which will open our  Application automatically in the browser.
• You can see our application homepage hosted at http://localhost:4200/

Apache storm installation on AWS Ubuntu

Storm installation on single node

1.sudo apt-get update
2.sudo apt-get install default-jre

$ sudo apt-get install git -y

$ sudo apt-get install libtool -y

$ sudo apt-get install automake -y

$ sudo apt-get install uuid-dev

$ sudo apt-get install g++ -y

$ sudo apt-get install gcc-multilib -y

*****************************************************************
1.ZooKEEPER install

https://archive.apache.org/dist/zookeeper/zookeeper-3.4.6/

$wget https://archive.apache.org/dist/zookeeper/zookeeper-3.4.6/zookeeper-3.4.6.tar.gz


ubuntu@ip:~$ wget https://archive.apache.org/dist/zookeeper/zookeeper-3.4.6/zookeeper-3.4.6.tar.gz
ubuntu@ip:~$ tar -xvf zookeeper-3.4.6.tar.gz
ubuntu@ip:~$ mv zookeeper-3.4.6 zookeeper


 Optionally : a)  Add ZOOKEEPER_HOME under .bashrc b)  Add ZOOKEEPER_HOME/bin to the PATH


========================
ubuntu@ip-:~$ ls /usr/lib/jvm/
default-java  java-1.8.0-openjdk-amd64  java-8-openjdk-amd64
ubuntu@ip-:~$
ubuntu@ip-:~$ ls
zookeeper  zookeeper-3.4.6.tar.gz
ubuntu@ip-:~$ pwd
/home/ubuntu
ubuntu@ip-:~$

=========================

export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-amd64
export ZOOKEEPER_HOME=/home/ubuntu/zookeeper
export PATH=$PATH:$JAVA_HOME/bin:$ZOOKEEPER_HOME/bin

create data directory
mkdir zookeeper-data

create config file

$ZOOKEEPER_HOME/conf/zoo.cfg
tickTme=2000
dataDir=/home/ubuntu/zookeeper-data
clientPort=2181


Restart the session

start zookeeper

$zkServer.sh start


https://zookeeper.apache.org/doc/current/zookeeperStarted.html#sc_InstallingSingleMode

*******************************
2.ZEROMQ

ubuntu@ip-:~$ wget http://download.zeromq.org/zeromq-2.1.7.tar.gz

ubuntu@ip-:~$ tar -xvf zeromq-2.1.7.tar.gz

ubuntu@ip-:~$ cd zeromq-2.1.7/
ubuntu@ip-:~$ ./configure
ubuntu@ip-:~$ sudo apt-get install make
ubuntu@ip-:~$ make
ubuntu@ip-:~$ sudo make install


***********************
3.java bindings for Zeromq (jzmq)
ubuntu@ip-:~$ git clone https://github.com/nathanmarz/jzmq.git
ubuntu@ip-:~$cd jzmq


 Configurating jzmq: 1) $ cd jzmq
2)  $ sed -i ‘s/classdist_noinst.stamp/classnoinst.stamp/g’ src/Makefile.am
3) $ ./autogen.sh

ubuntu@ip:~/jzmq$ ./autogen.sh
autogen.sh: error: could not find pkg-config.  pkg-config is required to run autogen.sh.


https://stackoverflow.com/questions/12115160/compiling-jzmq-on-ubuntu

ubuntu@ip-:~/jzmq/src$ CLASSPATH=.:./.:$CLASSPATH javac -d . org/zeromq/ZMQ.java org/zeromq/ZMQException.java org/zeromq/ZMQQueue.java org/zeromq/ZMQForwarder.java org/zeromq/ZMQStreamer.java
ubuntu@ip-:~/jzmq/src$

ubuntu@ip-:~/jzmq$ sudo apt-get install pkg-config

autogen.sh: error: could not find libtool. libtool is required to run autogen.sh
https://github.com/zeromq/libzmq/issues/1385
sudo ln -s /usr/bin/libtoolize /usr/bin/libtool


ubuntu@ip-:~/jzmq$./configure
ubuntu@ip-:~/jzmq$ make
ubuntu@ip-:~/jzmq$ sudo make install

**********************************************
4.configure storm

ubuntu@ip-:~$ wget http://www.apache.org/dyn/closer.lua/storm/apache-storm-0.9.1-incubating/apache-storm-0.9.1-incubating.tar.gz

wget https://github.com/downloads/nathanmarz/storm/storm-0.8.1.zip
unzip storm-0.8.1.zip


export STORM_HOME=/home/ubuntu/storm
export PATH=$PATH:$STORM_HOME/bin

Restart the session

cd $STORM_HOME
mkdir data

Edit storm.yml which is in $STORM_HOME/conf/storm.yaml

**************************************

storm.zookeeper.servers:
     - "10.0.0.194"
     - "10.0.0.195"
     - "10.0.0.196"

nimbus.host: "10.0.0.182"

storm.local.dir: "/var/storm"

supervisor.slots.ports:
    - 6700
    - 6701
    - 6702


OR


storm.zookeeper.servers:
- "localhost"
storm.zookeeper.port: 2181
nimbus.host: "localhost"
nimbus.thrift.port: 6627
ui.port: 8772
storm.local.dir: "/home/ubuntu/storm/data"
java.library.path: "/usr/lib/jvm/java-1.8.0-openjdk-amd64"
supervisor.slots.ports:
- 6700
- 6701
- 6702



*****************************************

$storm nimbus
$storm supervisor
$storm UI





ubuntu@ip-:~/storm$ storm nimbus
Running: java -server
-Dstorm.options= -Dstorm.home=/home/ubuntu/storm
                 -Djava.library.path=/usr/lib/jvm/java-1.8.0-openjdk-amd64
-cp /home/ubuntu/storm/storm-0.8.1.jar
    :/home/ubuntu/storm/lib/clout-0.4.1.jar
:/home/ubuntu/storm/lib/tools.logging-0.2.3.jar
:/home/ubuntu/storm/lib/ring-jetty-adapter-0.3.11.jar
:/home/ubuntu/storm/lib/math.numeric-tower-0.0.1.jar
:/home/ubuntu/storm/lib/commons-fileupload-1.2.1.jar
:/home/ubuntu/storm/lib/carbonite-1.5.0.jar
:/home/ubuntu/storm/lib/jetty-6.1.26.jar
:/home/ubuntu/storm/lib/json-simple-1.1.jar
:/home/ubuntu/storm/lib/curator-client-1.0.1.jar
:/home/ubuntu/storm/lib/snakeyaml-1.9.jar
:/home/ubuntu/storm/lib/reflectasm-1.07-shaded.jar
:/home/ubuntu/storm/lib/commons-lang-2.5.jar
:/home/ubuntu/storm/lib/httpclient-4.1.1.jar
:/home/ubuntu/storm/lib/jetty-util-6.1.26.jar
:/home/ubuntu/storm/lib/commons-logging-1.1.1.jar
:/home/ubuntu/storm/lib/commons-exec-1.1.jar
:/home/ubuntu/storm/lib/asm-4.0.jar
:/home/ubuntu/storm/lib/slf4j-api-1.5.8.jar
:/home/ubuntu/storm/lib/clj-time-0.4.1.jar
:/home/ubuntu/storm/lib/httpcore-4.1.jar
:/home/ubuntu/storm/lib/joda-time-2.0.jar
:/home/ubuntu/storm/lib/guava-13.0.jar
:/home/ubuntu/storm/lib/junit-3.8.1.jar
:/home/ubuntu/storm/lib/ring-core-0.3.10.jar
:/home/ubuntu/storm/lib/kryo-2.17.jar
:/home/ubuntu/storm/lib/commons-codec-1.4.jar
:/home/ubuntu/storm/lib/tools.macro-0.1.0.jar
:/home/ubuntu/storm/lib/jline-0.9.94.jar
:/home/ubuntu/storm/lib/commons-io-1.4.jar
:/home/ubuntu/storm/lib/slf4j-log4j12-1.5.8.jar
:/home/ubuntu/storm/lib/jzmq-2.1.0.jar
:/home/ubuntu/storm/lib/curator-framework-1.0.1.jar
:/home/ubuntu/storm/lib/jgrapht-0.8.3.jar
:/home/ubuntu/storm/lib/log4j-1.2.16.jar
:/home/ubuntu/storm/lib/servlet-api-2.5-20081211.jar
:/home/ubuntu/storm/lib/minlog-1.2.jar
:/home/ubuntu/storm/lib/disruptor-2.10.1.jar
:/home/ubuntu/storm/lib/libthrift7-0.7.0.jar
:/home/ubuntu/storm/lib/core.incubator-0.1.0.jar
:/home/ubuntu/storm/lib/servlet-api-2.5.jar
:/home/ubuntu/storm/lib/objenesis-1.2.jar
:/home/ubuntu/storm/lib/compojure-0.6.4.jar
:/home/ubuntu/storm/lib/clojure-1.4.0.jar
:/home/ubuntu/storm/lib/hiccup-0.3.6.jar
:/home/ubuntu/storm/lib/ring-servlet-0.3.11.jar
:/home/ubuntu/storm/lib/tools.cli-0.2.2.jar
:/home/ubuntu/storm/lib/zookeeper-3.3.3.jar
:/home/ubuntu/storm/log4j
:/home/ubuntu/storm/conf

-Xmx1024m
-Dlogfile.name=nimbus.log
-Dlog4j.configuration=storm.log.properties backtype.storm.daemon.nimbus




ubuntu@ip-:~/storm/bin$ storm supervisor
Running: java -server
-Dstorm.options= -Dstorm.home=/home/ubuntu/storm
-Djava.library.path=/usr/lib/jvm/java-1.8.0-openjdk-amd64
-cp /home/ubuntu/storm/storm-0.8.1.jar
     :/home/ubuntu/storm/lib/clout-0.4.1.jar
:/home/ubuntu/storm/lib/tools.logging-0.2.3.jar
:/home/ubuntu/storm/lib/ring-jetty-adapter-0.3.11.jar
:/home/ubuntu/storm/lib/math.numeric-tower-0.0.1.jar
:/home/ubuntu/storm/lib/commons-fileupload-1.2.1.jar
:/home/ubuntu/storm/lib/carbonite-1.5.0.jar
:/home/ubuntu/storm/lib/jetty-6.1.26.jar
:/home/ubuntu/storm/lib/json-simple-1.1.jar
:/home/ubuntu/storm/lib/curator-client-1.0.1.jar
:/home/ubuntu/storm/lib/snakeyaml-1.9.jar
:/home/ubuntu/storm/lib/reflectasm-1.07-shaded.jar
:/home/ubuntu/storm/lib/commons-lang-2.5.jar
:/home/ubuntu/storm/lib/httpclient-4.1.1.jar
:/home/ubuntu/storm/lib/jetty-util-6.1.26.jar
:/home/ubuntu/storm/lib/commons-logging-1.1.1.jar
:/home/ubuntu/storm/lib/commons-exec-1.1.jar
:/home/ubuntu/storm/lib/asm-4.0.jar
:/home/ubuntu/storm/lib/slf4j-api-1.5.8.jar
:/home/ubuntu/storm/lib/clj-time-0.4.1.jar
:/home/ubuntu/storm/lib/httpcore-4.1.jar
:/home/ubuntu/storm/lib/joda-time-2.0.jar
:/home/ubuntu/storm/lib/guava-13.0.jar
:/home/ubuntu/storm/lib/junit-3.8.1.jar
:/home/ubuntu/storm/lib/ring-core-0.3.10.jar
:/home/ubuntu/storm/lib/kryo-2.17.jar
:/home/ubuntu/storm/lib/commons-codec-1.4.jar
:/home/ubuntu/storm/lib/tools.macro-0.1.0.jar
:/home/ubuntu/storm/lib/jline-0.9.94.jar
:/home/ubuntu/storm/lib/commons-io-1.4.jar
:/home/ubuntu/storm/lib/slf4j-log4j12-1.5.8.jar
:/home/ubuntu/storm/lib/jzmq-2.1.0.jar
:/home/ubuntu/storm/lib/curator-framework-1.0.1.jar
:/home/ubuntu/storm/lib/jgrapht-0.8.3.jar
:/home/ubuntu/storm/lib/log4j-1.2.16.jar
:/home/ubuntu/storm/lib/servlet-api-2.5-20081211.jar
:/home/ubuntu/storm/lib/minlog-1.2.jar
:/home/ubuntu/storm/lib/disruptor-2.10.1.jar
:/home/ubuntu/storm/lib/libthrift7-0.7.0.jar
:/home/ubuntu/storm/lib/core.incubator-0.1.0.jar
:/home/ubuntu/storm/lib/servlet-api-2.5.jar
:/home/ubuntu/storm/lib/objenesis-1.2.jar
:/home/ubuntu/storm/lib/compojure-0.6.4.jar
:/home/ubuntu/storm/lib/clojure-1.4.0.jar
:/home/ubuntu/storm/lib/hiccup-0.3.6.jar
:/home/ubuntu/storm/lib/ring-servlet-0.3.11.jar
:/home/ubuntu/storm/lib/tools.cli-0.2.2.jar
:/home/ubuntu/storm/lib/zookeeper-3.3.3.jar
:/home/ubuntu/storm/log4j
:/home/ubuntu/storm/conf

-Xmx1024m
-Dlogfile.name=supervisor.log
-Dlog4j.configuration=storm.log.properties backtype.storm.daemon.supervisor



ubuntu@ip-:~/storm/bin$ storm ui
Running: java -server -Dstorm.options= -Dstorm.home=/home/ubuntu/storm -Djava.library.path=/usr/lib/jvm/java-1.8.0-openjdk-amd64 -cp /home/ubuntu/storm/storm-0.8.1.jar:/home/ubuntu/storm/lib/clout-0.4.1.jar:/home/ubuntu/storm/lib/tools.logging-0.2.3.jar:/home/ubuntu/storm/lib/ring-jetty-adapter-0.3.11.jar:/home/ubuntu/storm/lib/math.numeric-tower-0.0.1.jar:/home/ubuntu/storm/lib/commons-fileupload-1.2.1.jar:/home/ubuntu/storm/lib/carbonite-1.5.0.jar:/home/ubuntu/storm/lib/jetty-6.1.26.jar:/home/ubuntu/storm/lib/json-simple-1.1.jar:/home/ubuntu/storm/lib/curator-client-1.0.1.jar:/home/ubuntu/storm/lib/snakeyaml-1.9.jar:/home/ubuntu/storm/lib/reflectasm-1.07-shaded.jar:/home/ubuntu/storm/lib/commons-lang-2.5.jar:/home/ubuntu/storm/lib/httpclient-4.1.1.jar:/home/ubuntu/storm/lib/jetty-util-6.1.26.jar:/home/ubuntu/storm/lib/commons-logging-1.1.1.jar:/home/ubuntu/storm/lib/commons-exec-1.1.jar:/home/ubuntu/storm/lib/asm-4.0.jar:/home/ubuntu/storm/lib/slf4j-api-1.5.8.jar:/home/ubuntu/storm/lib/clj-time-0.4.1.jar:/home/ubuntu/storm/lib/httpcore-4.1.jar:/home/ubuntu/storm/lib/joda-time-2.0.jar:/home/ubuntu/storm/lib/guava-13.0.jar:/home/ubuntu/storm/lib/junit-3.8.1.jar:/home/ubuntu/storm/lib/ring-core-0.3.10.jar:/home/ubuntu/storm/lib/kryo-2.17.jar:/home/ubuntu/storm/lib/commons-codec-1.4.jar:/home/ubuntu/storm/lib/tools.macro-0.1.0.jar:/home/ubuntu/storm/lib/jline-0.9.94.jar:/home/ubuntu/storm/lib/commons-io-1.4.jar:/home/ubuntu/storm/lib/slf4j-log4j12-1.5.8.jar:/home/ubuntu/storm/lib/jzmq-2.1.0.jar:/home/ubuntu/storm/lib/curator-framework-1.0.1.jar:/home/ubuntu/storm/lib/jgrapht-0.8.3.jar:/home/ubuntu/storm/lib/log4j-1.2.16.jar:/home/ubuntu/storm/lib/servlet-api-2.5-20081211.jar:/home/ubuntu/storm/lib/minlog-1.2.jar:/home/ubuntu/storm/lib/disruptor-2.10.1.jar:/home/ubuntu/storm/lib/libthrift7-0.7.0.jar:/home/ubuntu/storm/lib/core.incubator-0.1.0.jar:/home/ubuntu/storm/lib/servlet-api-2.5.jar:/home/ubuntu/storm/lib/objenesis-1.2.jar:/home/ubuntu/storm/lib/compojure-0.6.4.jar:/home/ubuntu/storm/lib/clojure-1.4.0.jar:/home/ubuntu/storm/lib/hiccup-0.3.6.jar:/home/ubuntu/storm/lib/ring-servlet-0.3.11.jar:/home/ubuntu/storm/lib/tools.cli-0.2.2.jar:/home/ubuntu/storm/lib/zookeeper-3.3.3.jar:/home/ubuntu/storm/log4j:/home/ubuntu/storm:/home/ubuntu/storm/conf -Xmx768m -Dlogfile.name=ui.log -Dlog4j.configuration=storm.log.properties backtype.storm.ui.core

Apachce Kafka Installation on AWS Ubuntu

1. add user

useradd kafka -m
passwd kafka
adduser kafka sudo
su - kafka

2. Install java
sudo apt-get update
sudo apt-get install default-jre

3.zookeeper
sudo apt-get install zookeeperd

check 1.sudo netstat -nlpt | grep ':2181'
2.telnet localhost 2181
(type in ruok and press ENTER.ZooKeeper will say imok and end the Telnet session.)

We may start zookeeper in the following way:

$ sudo bin/zkServer.sh start

4.kafka
$ mkdir -p ~/kafka
$ cd kafka
$ wget https://archive.apache.org/dist/kafka/0.10.2.0/kafka_2.10-0.10.2.0.tgz
tar xvzf kafka_2.10-0.10.2.0.tgz --strip 1 (extract the file with "--strip 1" option to extract files directly into the directory:)

To be able to delete topics, take off '#' from ~/kafka/config/server.properties:
# Switch to enable topic deletion or not, default value is false
delete.topic.enable=true

To start server, we ned to run kafka-server-start.sh

$ ~/kafka/bin/kafka-server-start.sh ~/kafka/config/server.properties
Now, we can check listening ports:

ZooKeeper : 2181
Kafka : 9092

kafka@:~/kafka/bin$ /kafka/bin/kafka-server-start.sh /kafka/config/server.properties
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000c0000000, 1073741824, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 1073741824 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /home/kafka/kafka/bin/hs_err_pid15414.log

https://stackoverflow.com/questions/34966739/kafka-failed-to-map-1073741824-bytes-for-committing-reserved-memory
./bin/kafka-server-start.sh

To publish messages, you should create a Kafka producer.
You can easily create one from the command line using the kafka-console-producer.sh script.
It expects the Kafka server's hostname and port, along with a topic name as its arguments.
Publish the string "Hello, World" to a topic called TamilTest by typing in the following:

echo "Hello Tamil java...." | ~/kafka/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic TamilTest

To consume messages, you can create a Kafka consumer using the kafka-console-consumer.sh script.
It expects the ZooKeeper server's hostname and port, along with a topic name as its arguments.

The following command consumes messages from the topic we published to.
Note the use of the --from-beginning flag, which is present because we want to consume a message that was published before the consumer was started.

~/kafka/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic TamilTopic --from-beginning

Stopping Kafka server (broker)
After performing all the operations, we can stop the server using the following command:

$ bin/kafka-server-stop.sh config/server.properties