Cyberithub

How to Install and Configure Filebeat on Ubuntu/Debian 1

How to Install and Configure Filebeat on Ubuntu/Debian

by
Advertisements

In this article, we will see how to install and configure Filebeat on Ubuntu/Debian servers. Filebeat is a lightweight agent installed on your servers that monitors and collect events and then forwards them either to Elasticsearch or Logstash for indexing. It belongs to the family of lightweight shippers where each shipper is known for shipping different kinds of data such as system logs, network data, metric information etc. Filebeat can collect data from various kind of devices such as security devices, cloud hosts, on-premise servers, containers etc. It is also very easy to install and configure. Here we will see the steps to install and configure Filebeat on Ubuntu/Debian based systems.

 

How Filebeat Works

The way filebeat works is that once filebeat is started, it starts one or more inputs that looks in the location specified for log data. It starts a harvester for each log filebeat locates. Then each harvester reads a single log for new content and sends the new log data to libbeat, which aggregates the events and sends the aggregated data to the output that you’ve configured for Filebeat.

 

How to Install and Configure Filebeat on Ubuntu/Debian

How to Install and Configure Filebeat on Ubuntu/Debian

Also Read: [Solved]: "Configure error: no acceptable C compiler found in $PATH"

Step 1: Prerequisites

a) You should have a running Ubuntu/Debian Server.

b) You should have sudo or root access to run privileged commands.

c) You should apt, dpkg and curl utility installed in your Server.

d) You should have Elasticsearch installed on your Server.

e) You should have Kibana installed on your Server.

 

Step 2: Update Your Server

Before installing filebeat, it is recommended to keep your system updated by running sudo apt update && sudo apt upgrade command as shown below. This will download and install all the latest available security and bug fixes along with the package upgrades.

cyberithub@ubuntu:~$ sudo apt update && sudo apt upgrade
Hit:1 https://d3nt0h4h6pmmc4.cloudfront.net/ubuntu focal InRelease
Hit:2 http://ppa.launchpad.net/flatpak/stable/ubuntu focal InRelease
Hit:3 https://dl.google.com/linux/chrome/deb stable InRelease
Hit:4 https://dl.winehq.org/wine-builds/ubuntu focal InRelease
Hit:5 http://in.archive.ubuntu.com/ubuntu focal InRelease
Hit:6 http://security.ubuntu.com/ubuntu focal-security InRelease
Hit:7 http://ppa.launchpad.net/gencfsm/ppa/ubuntu focal InRelease
Hit:8 http://in.archive.ubuntu.com/ubuntu focal-updates InRelease
Hit:9 https://download.sublimetext.com apt/stable/ InRelease
Hit:10 http://in.archive.ubuntu.com/ubuntu focal-backports InRelease
Hit:11 http://ppa.launchpad.net/juju/stable/ubuntu focal InRelease
Hit:12 https://ngrok-agent.s3.amazonaws.com buster InRelease
Hit:13 http://ppa.launchpad.net/libreoffice/ppa/ubuntu focal InRelease
Hit:14 http://ppa.launchpad.net/mojo-maintainers/ppa/ubuntu focal InRelease
Hit:15 http://ppa.launchpad.net/ubuntu-toolchain-r/test/ubuntu focal InRelease
Hit:16 https://ftp.postgresql.org/pub/pgadmin/pgadmin4/apt/focal pgadmin4 InRelease
Reading package lists... Done
...................................................

 

Step 3: Download Filebeat

You have to visit official website and download the latest filebeat deb package available for Ubuntu/Debian based systems using below curl command. This will download the package in current working directory.

cyberithub@ubuntu:~$ curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-8.10.2-amd64.deb
% Total     % Received % Xferd  Average  Speed Time    Time     Time    Current
                                Dload   Upload Total   Spent     Left   Speed
100 47.1M  100 47.1M   0     0  4286k        0 0:00:11 0:00:11 --:--:-- 4683k

 

Step 4: Install Filebeat

Once the package is downloaded, you can install it by running sudo dpkg -i filebeat-8.10.2-amd64.deb command as shown below.

cyberithub@ubuntu:~$ sudo dpkg -i filebeat-8.10.2-amd64.deb
Selecting previously unselected package filebeat.
(Reading database ... 259195 files and directories currently installed.)
Preparing to unpack filebeat-8.10.2-amd64.deb ...
Unpacking filebeat (8.10.2) ...
Setting up filebeat (8.10.2) ...
Processing triggers for systemd (245.4-4ubuntu3.22) ...

 

Step 5: Verify Installation

After successful installation, you can verify the installed files path by using dpkg -L filebeat command as shown below.

cyberithub@ubuntu:~$ dpkg -L filebeat 
/.
/usr
/usr/share
/usr/share/doc
/usr/share/doc/filebeat
/usr/share/doc/filebeat/changelog.gz
/usr/share/filebeat
/usr/share/filebeat/NOTICE.txt
/usr/share/filebeat/bin
/usr/share/filebeat/bin/filebeat-god
/usr/share/filebeat/bin/filebeat
/usr/share/filebeat/kibana
/usr/share/filebeat/kibana/7
/usr/share/filebeat/kibana/7/visualization
/usr/share/filebeat/kibana/7/visualization/a09329d0-73c6-11eb-a3e3-b3cc7c78a70f.json
/usr/share/filebeat/kibana/7/visualization/f0e60404-ddf4-4b46-8e45-e28c4fb6d60d.json
/usr/share/filebeat/kibana/7/visualization/dcccaa80-4791-11e7-be88-2ddb32f3df97-ecs.json
/usr/share/filebeat/kibana/7/visualization/4e5d25c0-76ce-11eb-a3e3-b3cc7c78a70f.json
/usr/share/filebeat/kibana/7/visualization/57eae940-adf0-11e9-8358-1517661d7c84.json
/usr/share/filebeat/kibana/7/visualization/0f4028b2-3dc2-4cb6-80d8-285c847a02a1.json
..........................................................................

 

Step 6: Check Version

You can check the current installed version by running filebeat version command as shown below.

cyberithub@ubuntu:~$ filebeat version
filebeat version 8.10.2 (amd64), libbeat 8.10.2 [480bccf4f0423099bb2c0e672a44c54ecd7a805e built 2023-09-18 18:09:06 +0000 UTC]

 

Step 7: Connect to Elastic Stack

To setup filebeat, you need to have connection to Elasticsearch and Kibana. For that, you have to provide connection information such as cloud.id and cloud.auth in filebeat.yml file.

NOTE:

Please note that if you don't know the path of filebeat.yml file then you can find it by running sudo find / -name filebeat.yml command.
cloud.id: "<cloud.id_of_elasticsearch_service>"
cloud.auth: "filebeat_setup:YOUR_PASSWORD"

If you want to specify the elasticsearch installation path to filebeat then use below configuration in filebeat.yml file.

output.elasticsearch:
  hosts: ["https://myEShost:9200"]
  username: "filebeat_internal"
  password: "YOUR_PASSWORD" 
  ssl:
    enabled: true
    ca_trusted_fingerprint: "<fingerprint>"
  • Hosts: Elasticsearch host URL
  • Username: Filebeat authenticated user
  • Password: Filebeat authenticated user password
  • ca_trusted_fingerprint: It will be available in elasticsearch startup logs or you can refer to connect clients to Elasticsearch documentation for other options on retrieving it.

If Kibana is running on different host than elasticsearch service then you have to configure Kibana endpoint in filebeat.yml as shown below.

 setup.kibana:
   host: "mykibanahost:5601" 
   username: "my_kibana_user" 
   password: "{pwd}"
  • Host: Kibana host
  • Username: Kibana user
  • Password: Kibana user password

 

Step 8: Check Modules List

You can check the list of enabled and disabled modules by running sudo filebeat modules list command as shown below.

cyberithub@ubuntu:~$ sudo filebeat modules list
Enabled:

Disabled:
activemq
apache
auditd
aws
awsfargate
azure
barracuda
bluecoat
cef
checkpoint
cisco
coredns
crowdstrike
cyberarkpas
cylance
elasticsearch
envoyproxy
..............................................

 

Step 9: Enable Module

To enable a specific module, let's say nginx in our case, use sudo filebeat modules enable nginx command as shown below.

cyberithub@ubuntu:~$ sudo filebeat modules enable nginx
Enabled nginx

 

Step 10: Setup Assets

Filebeat comes with predefined assets for parsing, indexing, and visualizing your data. So to load those assets, you need to first make sure the user specified in filebeat.yml is authorized to setup Filebeat. Once verified, run sudo filebeat setup -e command to load the recommended index template for writing to elasticsearch and deploy the sample dashboards for visualizing the data in Kibana.

cyberithub@ubuntu:~$ sudo filebeat setup -e
{"log.level":"info","@timestamp":"2023-09-22T00:21:07.206+0530","log.origin":{"file.name":"instance/beat.go","file.line":783},"message":"Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-09-22T00:21:07.207+0530","log.origin":{"file.name":"instance/beat.go","file.line":791},"message":"Beat ID: 65cab47a-deb9-4948-98d0-b2312ed18748","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-09-22T00:21:07.229+0530","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1303},"message":"Beat info","service.name":"filebeat","system_info":{"beat":{"path":{"config":"/etc/filebeat","data":"/var/lib/filebeat","home":"/usr/share/filebeat","logs":"/var/log/filebeat"},"type":"filebeat","uuid":"65cab47a-deb9-4948-98d0-b2312ed18748"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-09-22T00:21:07.229+0530","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1312},"message":"Build info","service.name":"filebeat","system_info":{"build":{"commit":"480bccf4f0423099bb2c0e672a44c54ecd7a805e","libbeat":"8.10.2","time":"2023-09-18T18:09:06.000Z","version":"8.10.2"},"ecs.version":"1.6.0"}}
..........................................................

 

Step 11: Start Filebeat Service

After setting up the assets, you can start filebeat service by running sudo service filebeat start command and then check status by using sudo service filebeat status command as shown below.

cyberithub@ubuntu:~$ sudo service filebeat start
cyberithub@ubuntu:~$ sudo service filebeat status

 

Step 12: Launch Kibana Dashboard

Since Filebeat comes with pre-built Kibana dashboard, you don't have to install it separately to visualize all the log data. Just go to your favorite browser and open http://<kibana_host_ip>:5601. Once opened, click on Discover in the side navigation and select predefined filebeat-* index pattern to see the filebeat data. You can also open any of the Dashboards by just clicking on Dashboard in side navigation.

Leave a Comment