Apachce Kafka Installation on AWS Ubuntu

1. add user

useradd kafka -m
passwd kafka
adduser kafka sudo
su - kafka

2. Install java
sudo apt-get update
sudo apt-get install default-jre

3.zookeeper
sudo apt-get install zookeeperd

check 1.sudo netstat -nlpt | grep ':2181'
2.telnet localhost 2181
(type in ruok and press ENTER.ZooKeeper will say imok and end the Telnet session.)

We may start zookeeper in the following way:

$ sudo bin/zkServer.sh start

4.kafka
$ mkdir -p ~/kafka
$ cd kafka
$ wget https://archive.apache.org/dist/kafka/0.10.2.0/kafka_2.10-0.10.2.0.tgz
tar xvzf kafka_2.10-0.10.2.0.tgz --strip 1 (extract the file with "--strip 1" option to extract files directly into the directory:)

To be able to delete topics, take off '#' from ~/kafka/config/server.properties:
# Switch to enable topic deletion or not, default value is false
delete.topic.enable=true

To start server, we ned to run kafka-server-start.sh

$ ~/kafka/bin/kafka-server-start.sh ~/kafka/config/server.properties
Now, we can check listening ports:

ZooKeeper : 2181
Kafka : 9092

kafka@:~/kafka/bin$ /kafka/bin/kafka-server-start.sh /kafka/config/server.properties
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000c0000000, 1073741824, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 1073741824 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /home/kafka/kafka/bin/hs_err_pid15414.log

https://stackoverflow.com/questions/34966739/kafka-failed-to-map-1073741824-bytes-for-committing-reserved-memory
./bin/kafka-server-start.sh

To publish messages, you should create a Kafka producer.
You can easily create one from the command line using the kafka-console-producer.sh script.
It expects the Kafka server's hostname and port, along with a topic name as its arguments.
Publish the string "Hello, World" to a topic called TamilTest by typing in the following:

echo "Hello Tamil java...." | ~/kafka/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic TamilTest

To consume messages, you can create a Kafka consumer using the kafka-console-consumer.sh script.
It expects the ZooKeeper server's hostname and port, along with a topic name as its arguments.

The following command consumes messages from the topic we published to.
Note the use of the --from-beginning flag, which is present because we want to consume a message that was published before the consumer was started.

~/kafka/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic TamilTopic --from-beginning

Stopping Kafka server (broker)
After performing all the operations, we can stop the server using the following command:

$ bin/kafka-server-stop.sh config/server.properties