Sekoia.io Forwarder
Overview
Docker is a tool that can be used to run packaged applications in an isolated environment on a host. Packaged applications are stored in an object called an image, which includes an OS, the dependencies and the configuration. With that, the application will have the same behaviour whatever the OS used on the host as long as it's a x86-64 Linux host.
Sekoia.io offers a preconfigured concentrator based on Docker to forward events on the platform.
This method simplifies as much as possible the configuration needed to set up a concentrator in order to collect logs and send them on each relevant Intakes.
Warning
In this method, each technology MUST send their logs on different ports (of your choice) of the concentrator in order to make it work. The main principle of this method is to discriminate each technology by port to link them with the right Intake key.
Please find our English tutorial video below to see how to configure the forwarder ! French version is also available here.
Prerequisites
- A x86-64 Linux host using one of these templates:
Number of assets | vCPUs | RAM (Go) | Disk size (Go) | Sekoia concentrator settings |
---|---|---|---|---|
1000 | 2 | 4 | 200 | MEMORY_MESSAGES=2000000 / DISK_SPACE=180g |
10 000 | 4 | 8 | 1000 | MEMORY_MESSAGES=5000000 / DISK_SPACE=980g |
50 000 | 6 | 16 | 5000 | MEMORY_MESSAGES=12000000 / DISK_SPACE=4980g |
!!! info These data are recommendations based on standards and observed averages on Sekoia.io, so they may change depending on use cases.
- Last version of Docker Engine. Please follow this section to install it if needed
- INBOUND TCP or UDP flows opened between the systems/applications and the concentrator on the ports of your choice
- OUTBOUND TCP flow opened towards intake.sekoia.io on port 10514
Configure your concentrator
Two files are needed to run the concentrator: docker-compose.yml
and intakes.yaml
.
-
Create a folder where the configuration and data of the concentrator will be stored
mkdir sekoiaio-concentrator && cd sekoiaio-concentrator
-
Create the two files
touch docker-compose.yml && touch intakes.yaml
Configure intakes.yaml file
The intakes.yaml
file is used to tell the concentrator how to bind a port where logs are received to its technology represented by the Intake key.
For each technology, specify:
- a name: it has nothing to do with Sekoia.io, feel free to use the explicite value of your choice
- the protocol:
tcp
orudp
- a port: to process incoming events
- the Intake key: can be retreived from the Intakes page of your community
Here is an example of the file with 3 technologies sending their events to the concentrator on ports 20516
, 20517
and 20518
. Feel free to copy paste it and adapt it to your need:
---
intakes:
- name: Techno1
protocol: tcp
port: 20516
intake_key: INTAKE_KEY_FOR_TECHNO_1
- name: Techno2
protocol: udp
port: 20517
intake_key: INTAKE_KEY_FOR_TECHNO_2
- name: Techno3
protocol: tcp
port: 20518
intake_key: INTAKE_KEY_FOR_TECHNO_3
Tip
You are not limited to 3 entries. Feel free to adapt it to your needs.
Debug
A debug variable is available in order to debug a specific intake, for example
---
intakes:
- name: Techno1
protocol: tcp
port: 20516
intake_key: INTAKE_KEY_FOR_TECHNO_1
- name: Techno2
protocol: tcp
port: 20517
intake_key: INTAKE_KEY_FOR_TECHNO_2
debug: True
- name: Techno3
protocol: tcp
port: 20518
intake_key: INTAKE_KEY_FOR_TECHNO_3
When debug is set to true, the raw event received and the output message will be printed in STDOUT. Each one will be respectively identified using tags: : [Input $INTAKE_KEY] & [Output $INTAKE_KEY]
Configure docker-compose.yml file
Please find below a template of the docker-compose.yml
file.
version: "3.9"
services:
sekoia_concentrator:
image: ghcr.io/sekoia-io/sekoiaio-docker-concentrator:2.0
environment:
- MEMORY_MESSAGES=2000000
- DISK_SPACE=180g
ports:
- "20516-20566:20516-20566"
- "20516-20566:20516-20566/udp"
volumes:
- ./intakes.yaml:/intakes.yaml
- ./conf:/etc/rsyslog.d
- ./disk_queue:/var/spool/rsyslog
restart: always
pull_policy: always
Note
Feel free to copy paste it and adapt it to your needs.
The following sections will describe each element of this file above.
Environment variables
environment:
- MEMORY_MESSAGES=2000000
- DISK_SPACE=180g
Two environment variables are used to customize the container. These variables are used to define a queue for incoming logs in case there is a temporarily issue in transmitting events to Sekoia.io. The queue stores messages in memory up to a certain number of events and then store them on disk. When the issue is fixed, events stored are retrieved from the queue and forward to the plateform.
MEMORY_MESSAGES=2000000
means the queue is allowed to store up to 2,000,000 messages in memory. If we consider a message size is 1.2KB, then you will use 2,4GB of RAM memory (2000000 * 1.2KB = 2.4GB).DISK_SPACE=180g
means the queue is allowed to store on disk up to 180giga of messages.
Here you will find recommendations to set these variables based on the number of assets. You can also define your own values, which should be chosen according to your virtual machine's template.
Ports
ports:
- "20516-20566:20516-20566"
- "20516-20566:20516-20566/udp"
As specified in the Overview section, the concentrator will be run in an isolated environment. That means, by default, no flow is open between the host and the concentrator.
20516-20518:20516-20566
means that every packets coming through the TCP port form 20516
to 20566
to the host will be forwarded to the concentrator container from port 20516
to 20566
.
If you want to open a UDP flow, please add a line with /udp
at the end.
Important
Please adapt these values according to the intakes.yaml file.
Volumes
volumes:
- ./intakes.yaml:/intakes.yaml
- ./conf:/etc/rsyslog.d
- ./rsyslog:/var/spool/rsyslog
Volumes are used to share files and folders between the host and the container:
./intakes.yaml:/intakes.yaml
is used to tell the concentrator what protocol, ports and intake keys to use../conf:/etc/rsyslog.d
is mapped if you want to customize some rsyslog configuration (ADVANCED)./disk_queue:/var/spool/rsyslog
is used when the concentrator queue stores data on disk. The mapping avoids data loss if logs are stored on disk and the container is deleted.
Additional options
restart: always
pull_policy: always
restart: always
: this line indicates to restart the concentrator everytime it stops. That means if it crashes, if you restart Docker or if you restart the host, the concentrator will start automatically.pull_policy: always
: docker compose will always try to pull the image from the registry and check if a new version is available for the tag specified.
Start the concentrator
To start the concentrator, run the following command in the folder where docker-compose.yml
and intakes.yaml
are stored:
sudo docker compose up -d
You should have an answer like this:
[+] Running 1/1
⠿ Container docker-compose-rsyslog-1 Started
Note
The first time you start the container, you will download the image stored on the GitHub repository - ghcr.io/sekoia-io/sekoiaio-docker-concentrator
To check the status of the concentrator, you can run:
sudo docker compose ps
Useful commands
To view all the concentrator logs:
sudo docker compose logs
If you are using the Debug variable, everytime a log is received by the Forwarder, it's visible using docker compose logs
.
To view last logs in real time:
sudo docker compose logs -f
To stop the concentrator:
sudo docker compose stop
To delete the concentrator (the concentrator needs to be stopped):
sudo docker compose rm
Troubleshooting
You can't find the logs in your community? No worries this section will give you an advice to identify what is happening.
Step 1: check if the events are received by the concentrator
To check if the events are received by the concentrator, you can run the following command that will display all the last logs received:
sudo docker compose logs
To view container logs for a specific intake when using the Debug variable:
sudo docker compose logs | grep "YOUR_INTAKE_KEY"
Finally, if you want to check if events are coming in real time:
sudo docker compose logs -f
You don't see your events with these commands?
-
Check the
intakes.yaml
file to see if you have declared the protocols and ports you wanted. -
Verify if this information is taken into account by the concentrator. At start-up, the concentrator always shows the list of Intakes with the protocols and ports.
sudo docker compose logs | more
-
Check that you correctly declared the
ports
section in thedocker-compose.yml
file. They MUST be the same as the ports declared in theintakes.yaml
file. For instance, if you declared 4 technologies on ports25020
,25021
,25022
and25023
, the ports line thedocker-compose.yml
has to be at least"25020-25023:25020-25023"
for TCP and"25020-25023:25020-25023/udp"
if using UDP. -
Verify that traffic is incoming from your log source, meaning no firewall is blocking the events.
sudo tcpdump -i <change_with_interface_name> -c10 -nn src <remote_ip> -vv
To find those values:
change_with_interface_name
use the commandip addr
remote_ip
is the IP from which the logs should be incoming
Step 2: verify everything is correctly configured to forward events to Sekoia.io
-
Check the Intake keys you wrote in
intakes.yaml
are correct. -
Check the network flow between the concentrator host and Sekoia.io is opened to the destination
intake.sekoia.io
on protocolTCP
and port10514
. You can easily check it withtelnet
:sudo apt install telnet && telnet intake.sekoia.io 10514
This command should return the following answer, meaning the network flow is opened:
Connected to intake.sekoia.io. Escape character is '^]'.
Remove
telnet
if don't need it anymoresudo apt remove telnet
-
Finally check the status of the Sekoia.io plateform on https://status.sekoia.io.
Additional information
The image used to run the concentrator is maintained on this github repository. Feel free to contribute and make a pull request to improve the concentrator!
How to update the concentrator
Docker uses the notion of tag to identify the version of an image. The tag is always referenced in line starting with image
in docker-compose.yml
:
image: ghcr.io/sekoia-io/sekoiaio-docker-concentrator:2.0
2.0
means the version used by docker compose
is 2.0. You can find all the versions available on the GitHub repository here
To update the concentrator, just change the tag in docker-compose.yml
, then recreate the concentrator with the command:
sudo docker compose up -d
There is a special tag called latest
which always refers to the latest version. If you use this tag, every time the concentrator will be recreated, the image will be updated to the latest version.
Important
Be very careful if you use the tag latest
in production! If a major change is present in a new version, it could break your current concentrator with the settings in place.
Important
Each image version is also updated every week in order to update the OS contained in the image with the latest security patches. You are responsible for recreating the concentrator periodically to initiate the image update.
How do I add a new Intake later?
To add a new Intake, it's very simple ! Follow these steps:
- Edit the
intakes.yaml
file and add the new intake- name: NewIntake protocol: tcp port: 20519 intake_key: INTAKE_KEY_FOR_NEW_INTAKE
- Edit the
docker-compose.yml
file if needed and modify theports
section according to theintakes.yaml
file. It shouldn't be necessary from port20516
to20566
ports: - "20516-20566:20516-20566" - "20516-20566:20516-20566/udp"
- Recreate the concentrator by running the command:
sudo docker compose up -d
- Check in the concentrator logs that the new Intake is taken into account
sudo docker compose logs | more
Docker Engine installation
This section describes how to install Docker using the apt
repository on one of those Debian 64-bit versions:
- Debian Bullseye 11 (stable)
- Debian Buster 10 (oldstable)
All of the installation steps come from the official Docker Engine installation for Debian. Feel free to consult the official installation page for more information. To install Docker on another Linux OS, please consult the official Docker documentation
Uninstall old versions (if applicable)
Run the following command to uninstall old Docker versions:
sudo apt-get remove docker docker-engine docker.io containerd runc
Set up the repository
-
Update the
apt
package index and install packages to allowapt
to use a repository over HTTPSsudo apt-get update sudo apt-get install \ ca-certificates \ curl \ gnupg \ lsb-release
-
Add Docker's official GPG key
sudo mkdir -m 0755 -p /etc/apt/keyrings curl -fsSL https://download.docker.com/linux/debian/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
-
Use the following command to set up the repository
echo \ "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/debian \ $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
Install Docker Engine
-
Update the
apt
package indexsudo apt-get update
-
Install Docker Engine, containerd, and Docker compose
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
-
Verify that the Docker Engine installation is successful by running the hello-world image
sudo docker run hello-world
5 minutes setup on Debian
Warning
This script will automate all the steps detailed on this page for a Debian host. Please read the content of this page carefully before executing it.
Connect to the remote server where you would like to install the Sekoia.io Forwarder, then follow those steps:
-
Execute a script to setup the docker
wget https://raw.githubusercontent.com/SEKOIA-IO/documentation/main/docs/assets/operation_center/ingestion_methods/sekoiaio_forwarder/sekoiaio_docker_concentrator_autosetup.sh chmod +x sekoiaio_docker_concentrator_autosetup.sh ./sekoiaio_docker_concentrator_autosetup.sh rm sekoiaio_docker_concentrator_autosetup.sh
-
Edit the configuration files
sekoiaio-concentrator/intakes.yaml
by replacing thename
,protocol
,port
andintake_key
for each intake you would like to collectsekoiaio-concentrator/docker-compose.yml
by remplacing the value"20516-20518:20516-20518"
by a relevant content according to thesekoiaio-concentrator/intakes.yaml
previously edited
-
Start the docker
Follow the process you can find on the section Start the concentrator of this page.
sudo docker compose up -d sudo docker compose ps sudo docker compose logs -f
Enjoy your docker!