The last tutorial enabled us to start the flume agent, after which we can send the messages that we want over flume memory channel. In this objective, we have to pass the memory channel capacity to enable it to pass messages from source to sink.
This configuration can be done with the help of following configuration file.
|# example.conf: A single-node Flume configuration|
|# Name the components on this agent|
|a1.sources = r1|
|a1.sinks = k1|
|a1.channels = c1|
|# Describe/configure the source|
|a1.sources.r1.type = netcat|
|a1.sources.r1.bind = localhost|
|a1.sources.r1.port = 44444|
|# Describe the sink|
|a1.sinks.k1.type = logger|
|# Use a channel which buffers events in memory|
|a1.channels.c1.type = memory|
|a1.channels.c1.capacity = 1000|
|a1.channels.c1.transactionCapacity = 100|
|# Bind the source and sink to the channel|
|a1.sources.r1.channels = c1|
|a1.sinks.k1.channel = c1|
As you can see line 16-19 contains the configurations related to the memory channel to be configured. In this objective, the question asks you to configure particular memory channel with respective values for type, capacity and the transaction capacity of that memory channel.
The simplest thing that you have to do is to take those values and set it to respective properties in live 17-19 shown in above file.
For example, I was asked to configure a memory channel with 10000 bytes capacity and 1000 bytes as the transaction capacity. Therefore, my configurations looked something like this.
Once you do these changes in your configuration file and restart the flume agent, it will take these values and start the flume agent.
The next step is to send the messages to this flume agent using the different terminal window using telnet localhost 44444 command. As soon as you send some message from this terminal window to the running flume agent, it acknowledges that messages and prints an “OK” message, confirming the receipt of the message that you sent to flume.
I think following screenshot will help you to understand how it works.
As you can see, the smaller terminal window is the one which I am using to send messages to the flume agent. I have sent the three messages in the following order to the flume agent.
- Hello World
- This is Milind
- How are you?
After every message, the flume agent replies with an “OK” message on the same screen, confirming that it received the message successfully.
Having said that, the terminal window on which we started the flume agent, also prints the messages that it receives from the sources. With this, it is made sure that we do not lose any messages and if we do, those will be printed on this agent screen.
Once, we are done with sending and receiving messages that we want, you can close the already running flume agent by pressing CTRL+C, as shown using OVAL SHAPE in below screenshot. As soon as you press CTRL+C, it will start printing the stop messages, as shown below.
As you can see from the below screenshot, you get “Connection closed by foreign host” message on the terminal window you are using to send messages to flume agent.
The flume agent, before exiting finally, prints the number of messages it successfully received and the number of messages it successfully processed. From the screenshot, we can conclude that we have not lost a single record(s) and have a 100% processing record.
This completes the flume memory channel configuration. I hope the text and screenshots help you to understand how it works and how you can prepare for the HDPCD certification.
Remember, our ultimate goal is to clear the certification with flying colors and therefore I will make sure I do not miss any information to share with you guys, but if I do, I will really appreciate someone correcting/adding that information.
In conclusion, I would like to tell that our data ingestion category is finished with this one and in the next tutorials onward, we will focus on the next objective, which is data manipulation.
Please let me know if you want more detailed explanations and other sorts of help in the comments down below. I’ll really appreciate it.