install and execute kafka on my MAC

Contiuedly from last post(

2020/12/18 - [KAFKA] - install and execute kafka from AWS

), let's install and execute it in local, my case is in macOS.

 

 

• curl http://mirror.navercorp.com/apache/kafka/2.5.0/kafka_2.13-2.5.0.tgz --output kafka.tgz

Joses-MacBook-Pro:~ joohyunyoon$ curl http://mirror.navercorp.com/apache/kafka/2.5.0/kafka_2.13-2.5.0.tgz --output kafka.tgz 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 58.6M  100 58.6M    0     0  3553k      0  0:00:16  0:00:16 --:--:-- 4714k

curl : Client URL, a command line tool to transfer data to or from a server, using any of the supported protocols (HTTP, FTP, IMAP, POP3, SCP, SFTP, SMTP, TFTP, TELNET, LDAP or FILE)

 

 

• tar -xvf kafka.tgz

Joses-MacBook-Pro:kafka joohyunyoon$ tar -xvf kafka.tgz
x kafka_2.13-2.5.0/
x kafka_2.13-2.5.0/LICENSE
x kafka_2.13-2.5.0/NOTICE
x kafka_2.13-2.5.0/bin/
x kafka_2.13-2.5.0/bin/kafka-delete-records.sh
...

tar :

2020/12/18 - [KAFKA] - install and execute kafka from AWS

x : Extract

v : Verbose

f : File

 

 

• cd kafka_2.13-2.5.0/bin

bin :  Binaries, it's just a directory where a user of an operating system can expect to find applications

 

 

• ./kafka-topics.sh --create --bootstrap-server {aws ec2 public ip}:9092 --replication-factor 1 --partitions 3 --topic joohyun

Let's create topic.

Joses-MacBook-Pro:bin joohyunyoon$ ./kafka-topics.sh --create --bootstrap-server my AWS Public IP:9092 --replication-factor 1 --partitions 3 --topic joohyun
Created topic joohyun.

Which server I want to connect: My AWS Public IPv4 address my AWS Public IP

How many replication factor? 1, because broker is 1

How many partition want to make? 3

What do you want to name of topic? joohyun

 

 

• ./kafka-console-producer.sh --bootstrap-server my AWS Public IP:9092 --topic joohyun 

Let's contain the data into topic joohyun through the kafka-console from producer.

Joses-MacBook-Pro:bin joohyunyoon$ ./kafka-console-producer.sh --bootstrap-server my AWS Public IP:9092 --topic joohyun
>hello
>kafka
>hello
>world
>

 

 

• ./kafka-console-consumer.sh --bootstrap-server my AWS Public IP:9092 --topic joohyun --from-beginning

Hey, topic joohyun, I just put some data to you, can you show me you have the data what I toss?

Hey, you are out there! through the kafka-console from consumer.

Joses-MacBook-Pro:bin joohyunyoon$ ./kafka-console-consumer.sh --bootstrap-server my AWS Public IP:9092 --topic joohyun --from-beginning
hello
hello
kafka
world

 

 

• ./kafka-console-consumer.sh --bootstrap-server my AWS Public IP:9092 --topic joohyun -group joohyungroup --from-beginning

Then, let's assign group consumer which can assign the offset. I named the group name as joohyungroup.

Strangely, it result hello hello, the reason why consumer receive the data throught the partition randomly. If there is only one partition, then consumer receive by order.

Joses-MacBook-Pro:bin joohyunyoon$ ./kafka-console-consumer.sh --bootstrap-server my AWS Public IP:9092 --topic joohyun -group joohyungroup --from-beginning
hello
hello
kafka
world

 

Then, how this consumer group reveive ravenously, as much as it can?

Once consumer disconnect to recieve the data,

(base) Joses-MacBook-Pro:bin joohyunyoon$ ./kafka-console-consumer.sh --bootstrap-server my AWS Public IP:9092 --topic joohyun -group joohyungroup --from-beginning
hello
hello
kafka
world
^CProcessed a total of 4 messages

 

if producer still produce the data, then consumer group receive the new data since it disconnected. Because previous consumer group has already recorded previous data. Lossless.

Joses-MacBook-Pro:bin joohyunyoon$ ./kafka-console-producer.sh --bootstrap-server my AWS Public IP:9092 --topic joohyun
>hello
>kafka
>hello
>world
>a
>b
>c
>d
>e
>

 

 

• ./kafka-consumer-groups.sh --bootstrap-server my AWS Public IP:9092 --list

Hey, consumer group what named is joohyungroup, are you really out there?

Joses-MacBook-Pro:bin joohyunyoon$ ./kafka-consumer-groups.sh --bootstrap-server my AWS Public IP:9092 --list
joohyungroup

 

 

• ./kafka-consumer-groups.sh --bootstrap-server my AWS Public IP:9092 --group joohyungroup --describe

Hey, then can you describe who you are, how you look like?

Joses-MacBook-Pro:bin joohyunyoon$ ./kafka-consumer-groups.sh --bootstrap-server my AWS Public IP:9092 --group joohyungroup --describe

Consumer group 'joohyungroup' has no active members.

GROUP           TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID     HOST            CLIENT-ID
joohyungroup    joohyun         2          4               4               0               -               -               -
joohyungroup    joohyun         0          2               2               0               -               -               -
joohyungroup    joohyun         1          3               3               0               -               -               -

LAG : Kafka Consumer Lag is the indicator of how much lag there is between Kafka producers and consumers lively. If there are high number, consumer receives slowly rather than speed of producer products. For example, although producer consicutively produce the data while consumer disconnected, it occurs lag.

 

 

• ./kafka-consumer-groups.sh --bootstrap-server my AWS Public IP:9092 --group joohyungroup --topic joohyun --reset-offsets --to-earliest --execute

Can I be reset? I want to go back to the earliest.

Joses-MacBook-Pro:bin joohyunyoon$ ./kafka-consumer-groups.sh --bootstrap-server my AWS Public IP:9092 --group joohyungroup --topic joohyun --reset-offsets --to-earliest --execute

GROUP                          TOPIC                          PARTITION  NEW-OFFSET     
joohyungroup                   joohyun                        0          0              
joohyungroup                   joohyun                        1          0              
joohyungroup                   joohyun                        2          0

 

 

• ./kafka-consumer-groups.sh --bootstrap-server my AWS Public IP:9092 --group joohyungroup --topic joohyun --reset-offsets --to-offset 2 --execute

I am fickle, I want to go back just 2nd partition. How can I be?

Joses-MacBook-Pro:bin joohyunyoon$ ./kafka-consumer-groups.sh --bootstrap-server my AWS Public IP:9092 --group joohyungroup --topic joohyun --reset-offsets --to-offset 2 --execute

GROUP                          TOPIC                          PARTITION  NEW-OFFSET     
joohyungroup                   joohyun                        0          2              
joohyungroup                   joohyun                        1          2              
joohyungroup                   joohyun                        2          2