Fluent Bit – Sending Logs to ELK with Fluent Bit.
In this section, we will install Fluent Bit on a Windows machine to read logs from a file, parse the logs to extract relevant fields, and forward them to an Elasticsearch running on an Ubuntu machine.
Fluent Bit
Fluent Bit is a lightweight and high-performance log processor and forwarder. It is commonly used to collect, process, and ship logs to various destinations, such as Elasticsearch.
Installing and Configuring Fluent Bit on Windows
Download it from the official website, download the appropriate file for your Windows machine.
Extract the contents of the ZIP file to C:\Program Files\fluent-bit
Now, let's take a look at the sample logs file we want to parse
2025-02-05 12:34:56 SRC=192.168.1.10 DST=8.8.8.8 PROTO=UDP SPT=58321 DPT=53 LEN=78 ACTION=ALLOWED
2025-02-05 12:34:57 SRC=10.0.0.5 DST=192.168.1.1 PROTO=TCP SPT=443 DPT=52514 LEN=1500 ACTION=ALLOWED
2025-02-05 12:34:58 SRC=192.168.1.15 DST=203.0.113.5 PROTO=TCP SPT=50234 DPT=80 LEN=576 ACTION=ALLOWED
2025-02-05 12:34:59 SRC=172.16.0.22 DST=192.168.1.255 PROTO=UDP SPT=68 DPT=67 LEN=300 ACTION=ALLOWED
2025-02-05 12:35:00 SRC=192.168.1.12 DST=10.10.10.1 PROTO=ICMP TYPE=8 CODE=0 ID=1001 ACTION=ALLOWED
2025-02-05 12:35:01 SRC=203.0.113.45 DST=192.168.1.20 PROTO=TCP SPT=3389 DPT=445 LEN=1200 ACTION=BLOCKED
2025-02-05 12:35:02 SRC=192.168.1.30 DST=192.168.1.1 PROTO=TCP SPT=56789 DPT=22 LEN=1024 ACTION=ALLOWED
2025-02-05 12:35:03 SRC=198.51.100.2 DST=192.168.1.50 PROTO=TCP SPT=8080 DPT=443 LEN=1400 ACTION=ALLOWED
2025-02-05 12:35:04 SRC=192.168.1.40 DST=8.8.4.4 PROTO=UDP SPT=23456 DPT=53 LEN=70 ACTION=ALLOWED
2025-02-05 12:35:05 SRC=203.0.113.100 DST=192.168.1.100 PROTO=TCP SPT=23 DPT=23 LEN=600 ACTION=BLOCKED We will use Rubular to ensure the appropriate regex pattern
(?<timestamp>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}) SRC=(?<src_ip>\S+) DST=(?<dst_ip>\S+) PROTO=(?<protocol>\S+) SPT=(?<src_port>\d+) DPT=(?<dst_port>\d+) LEN=(?<length>\d+) ACTION=(?<action>\S+)
There is an unmatched line. Looking closely, this happened because that log has some unique attributes: TYPE CODE ID ,We can make another regex pattern to match it.
(?<timestamp>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}) SRC=(?<src_ip>\S+) DST=(?<dst_ip>\S+) PROTO=(?<protocol>\S+) TYPE=(?<icmp_type>\d+) CODE=(?<icmp_code>\d+) ID=(?<icmp_id>\d+) ACTION=(?<action>\S+)
Now we will update the parsers.conf file, which located at C:\Program Files\Fluent-bit\conf , to add the two parsers we made previously
Next we will update the fluent-bit.conf , which is located at the same previous path. But before we start editing, we need to know the 3 main sections we will focus on.
[INPUT]: Specifies where logs are collected from; in our case, it will be collected from the sample logs file we have created.
[OUTPUT]: Specifies where logs should be sent; we want to forward them to Elasticsearch.
[PARSER]: Defines how logs should be structured using regex; we have updated the parser.conf file so there is no need to change it.
The input name tail is an input plugin that allows to monitor one or several text files. It has a similar behavior to the
tail -fshell command.The plugin reads every matched file in the Path pattern and for every new line found (separated by a newline character (\n) ), so we will duplicate some logs in the file to generate new records
We have configured two outputs: es is the Elasticsearch and stdout will prints to our running shell
tls : on , tls.verify : off Those are important if Elasticsearch is configured with https
Now we will run the Fluent-bit
& 'C:\Program Files\Fluent-bit\bin\fluent-bit.exe' -c 'C:\Program Files\Fluent-bit\conf\fluent-bit.conf'
Open Elasticsearch and navigate to index management to ensure the existence of fluent-bit
Start to duplicate some data in the network_sample.txt, then go to discover in Elasticsearch and create a data view to the fluent-bit
We're done! Fluent Bit reads some logs, parses them using regex, and forwards the structured data to Elasticsearch.
Last updated