Fluent Bit – Sending Logs to ELK with Fluent Bit.

In this section, we will install Fluent Bit on a Windows machine to read logs from a file, parse the logs to extract relevant fields, and forward them to an Elasticsearch running on an Ubuntu machine.

Fluent Bit

Fluent Bit is a lightweight and high-performance log processor and forwarder. It is commonly used to collect, process, and ship logs to various destinations, such as Elasticsearch.

Installing and Configuring Fluent Bit on Windows

1. Download it from the official website, download the appropriate file for your Windows machine.

1. Extract the contents of the ZIP file to C:\Program Files\fluent-bit

1. Now, let's take a look at the sample logs file we want to parse

2025-02-05 12:34:56 SRC=192.168.1.10 DST=8.8.8.8 PROTO=UDP SPT=58321 DPT=53 LEN=78 ACTION=ALLOWED  
2025-02-05 12:34:57 SRC=10.0.0.5 DST=192.168.1.1 PROTO=TCP SPT=443 DPT=52514 LEN=1500 ACTION=ALLOWED  
2025-02-05 12:34:58 SRC=192.168.1.15 DST=203.0.113.5 PROTO=TCP SPT=50234 DPT=80 LEN=576 ACTION=ALLOWED  
2025-02-05 12:34:59 SRC=172.16.0.22 DST=192.168.1.255 PROTO=UDP SPT=68 DPT=67 LEN=300 ACTION=ALLOWED  
2025-02-05 12:35:00 SRC=192.168.1.12 DST=10.10.10.1 PROTO=ICMP TYPE=8 CODE=0 ID=1001 ACTION=ALLOWED  
2025-02-05 12:35:01 SRC=203.0.113.45 DST=192.168.1.20 PROTO=TCP SPT=3389 DPT=445 LEN=1200 ACTION=BLOCKED  
2025-02-05 12:35:02 SRC=192.168.1.30 DST=192.168.1.1 PROTO=TCP SPT=56789 DPT=22 LEN=1024 ACTION=ALLOWED  
2025-02-05 12:35:03 SRC=198.51.100.2 DST=192.168.1.50 PROTO=TCP SPT=8080 DPT=443 LEN=1400 ACTION=ALLOWED  
2025-02-05 12:35:04 SRC=192.168.1.40 DST=8.8.4.4 PROTO=UDP SPT=23456 DPT=53 LEN=70 ACTION=ALLOWED  
2025-02-05 12:35:05 SRC=203.0.113.100 DST=192.168.1.100 PROTO=TCP SPT=23 DPT=23 LEN=600 ACTION=BLOCKED  

1. We will use Rubular to ensure the appropriate regex pattern

(?<timestamp>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}) SRC=(?<src_ip>\S+) DST=(?<dst_ip>\S+) PROTO=(?<protocol>\S+) SPT=(?<src_port>\d+) DPT=(?<dst_port>\d+) LEN=(?<length>\d+) ACTION=(?<action>\S+)

• There is an unmatched line. Looking closely, this happened because that log has some unique attributes: TYPE CODE ID ,We can make another regex pattern to match it.

(?<timestamp>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}) SRC=(?<src_ip>\S+) DST=(?<dst_ip>\S+) PROTO=(?<protocol>\S+) TYPE=(?<icmp_type>\d+) CODE=(?<icmp_code>\d+) ID=(?<icmp_id>\d+) ACTION=(?<action>\S+)

1. Now we will update the parsers.conf file, which located at C:\Program Files\Fluent-bit\conf , to add the two parsers we made previously

If you don't have permission to save changes in the file in that path, you can save it in another path, delete the unupdated one, and move the updated file after to replace it.

1. Next we will update the fluent-bit.conf , which is located at the same previous path. But before we start editing, we need to know the 3 main sections we will focus on.

• [INPUT]: Specifies where logs are collected from; in our case, it will be collected from the sample logs file we have created.

• [OUTPUT]: Specifies where logs should be sent; we want to forward them to Elasticsearch.

• [PARSER]: Defines how logs should be structured using regex; we have updated the parser.conf file so there is no need to change it.

• The input name tail is an input plugin that allows to monitor one or several text files. It has a similar behavior to the tail -f shell command.

• The plugin reads every matched file in the Path pattern and for every new line found (separated by a newline character (\n) ), so we will duplicate some logs in the file to generate new records

• We have configured two outputs: es is the Elasticsearch and stdout will prints to our running shell

• tls : on , tls.verify : off Those are important if Elasticsearch is configured with https

1. Now we will run the Fluent-bit

& 'C:\Program Files\Fluent-bit\bin\fluent-bit.exe' -c 'C:\Program Files\Fluent-bit\conf\fluent-bit.conf'

The & operator tells PowerShell explicitly to treat the string as an executable

1. Open Elasticsearch and navigate to index management to ensure the existence of fluent-bit

1. Start to duplicate some data in the network_sample.txt, then go to discover in Elasticsearch and create a data view to the fluent-bit

We're done! Fluent Bit reads some logs, parses them using regex, and forwards the structured data to Elasticsearch.