Post 8 | HDPCD | Configure Flume Memory Channel

In the last tutorial, we saw the process to start the flume agent. This tutorial is an extension to the previous tutorial, so please refer to it before getting started with this tutorial.

The last tutorial enabled us to start the flume agent, after which we can send the messages that we want over flume memory channel. In this objective, we have to pass the memory channel capacity to enable it to pass messages from source to sink.

This configuration can be done with the help of following configuration file.

# example.conf: A single-node Flume configuration
# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1
# Describe/configure the source
a1.sources.r1.type = netcat
a1.sources.r1.bind = localhost
a1.sources.r1.port = 44444
# Describe the sink
a1.sinks.k1.type = logger
# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
# Bind the source and sink to the channel
a1.sources.r1.channels = c1 = c1

As you can see line 16-19 contains the configurations related to the memory channel to be configured. In this objective, the question asks you to configure particular memory channel with respective values for type, capacity and the transaction capacity of that memory channel.

The simplest thing that you have to do is to take those values and set it to respective properties in live 17-19 shown in above file.

For example, I was asked to configure a memory channel with 10000  bytes capacity and 1000 bytes as the transaction capacity. Therefore, my configurations looked something like this.




Once you do these changes in your configuration file and restart the flume agent, it will take these values and start the flume agent.

The next step is to send the messages to this flume agent using the different terminal window using telnet localhost 44444 command. As soon as you send some message from this terminal window to the running flume agent, it acknowledges that messages and prints an “OK” message, confirming the receipt of the message that you sent to flume.

I think following screenshot will help you to understand how it works.

Flume Agent Message Receiption
Flume Agent Message Reception

As you can see, the smaller terminal window is the one which I am using to send messages to the flume agent. I have sent the three messages in the following order to the flume agent.

  1. Hello World
  2. This is Milind
  3. How are you?

After every message, the flume agent replies with an “OK” message on the same screen, confirming that it received the message successfully.

Having said that, the terminal window on which we started the flume agent, also prints the messages that it receives from the sources. With this, it is made sure that we do not lose any messages and if we do, those will be printed on this agent screen.

Once, we are done with sending and receiving messages that we want, you can close the already running flume agent by pressing CTRL+C, as shown using OVAL SHAPE in below screenshot. As soon as you press CTRL+C, it will start printing the stop messages, as shown below.

As you can see from the below screenshot, you get “Connection closed by foreign host” message on the terminal window you are using to send messages to flume agent.

The flume agent, before exiting finally, prints the number of messages it successfully received and the number of messages it successfully processed. From the screenshot, we can conclude that we have not lost a single record(s) and have a 100% processing record.

Exiting Flume Agent
Exiting Flume Agent

This completes the flume memory channel configuration. I hope the text and screenshots help you to understand how it works and how you can prepare for the HDPCD certification.

Remember, our ultimate goal is to clear the certification with flying colors and therefore I will make sure I do not miss any information to share with you guys, but if I do, I will really appreciate someone correcting/adding that information.

In conclusion, I would like to tell that our data ingestion category is finished with this one and in the next tutorials onward, we will focus on the next objective, which is data manipulation.

Please let me know if you want more detailed explanations and other sorts of help in the comments down below. I’ll really appreciate it.



Published by milindjagre

I founded my blog four years ago and am currently working as a Data Scientist Analyst at the Ford Motor Company. I graduated from the University of Connecticut pursuing Master of Science in Business Analytics and Project Management. I am working hard and learning a lot of new things in the field of Data Science. I am a strong believer of constant and directional efforts keeping the teamwork at the highest priority. Please reach out to me at for further information. Cheers!

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: