Skip to content

Sekoia.io Forwarder

Overview

Docker is a tool that can be used to run packaged applications in an isolated environment on a host. Packaged applications are stored in an object called an image, which includes an OS, the dependencies and the configuration. With that, the application will have the same behaviour whatever the OS used on the host as long as it's a x86-64 Linux host.

Sekoia.io offers a preconfigured concentrator based on Docker to forward events on the platform.

This method simplifies as much as possible the configuration needed to set up a concentrator in order to collect logs and send them on each relevant Intakes.

Warning

In this method, each technology MUST send their logs on different ports (of your choice) of the concentrator in order to make it work. The main principle of this method is to discriminate each technology by port to link them with the right Intake key.

Please find our English tutorial video below to see how to configure the forwarder ! French version is also available here. French tutorial

Prerequisites

  • A x86-64 Linux host using one of these templates:
Number of assets vCPUs RAM (Go) Disk size (Go) Sekoia concentrator settings
1000 2 4 200 MEMORY_MESSAGES=2000000 / DISK_SPACE=180g
10 000 4 8 1000 MEMORY_MESSAGES=5000000 / DISK_SPACE=980g
50 000 6 16 5000 MEMORY_MESSAGES=12000000 / DISK_SPACE=4980g

!!! info These data are recommendations based on standards and observed averages on Sekoia.io, so they may change depending on use cases.

  • Last version of Docker Engine. Please follow this section to install it if needed
  • INBOUND TCP or UDP flows opened between the systems/applications and the concentrator on the ports of your choice
  • OUTBOUND TCP flow opened towards:
  • FRA1 intake.sekoia.io on port 10514
  • FRA2 fra2.app.sekoia.io on port 10514
  • MCO1 mco1.app.sekoia.io on port 10514
  • EUA1 app.uae1.sekoia.io on port 10514

!!! note The disk choice (SSD or HDD type) has no impact on the performance of Sekoia.io Forwarder. However, SSD type would be useful when an issue arise for recovery or catchup. Please choose accordingly to your usage and cost.

Configure your concentrator

Two files are needed to run the concentrator: docker-compose.yml and intakes.yaml.

  1. Create a folder where the configuration and data of the concentrator will be stored

    mkdir sekoiaio-concentrator && cd sekoiaio-concentrator
    
  2. Create the two files

    touch docker-compose.yml && touch intakes.yaml
    

Configure intakes.yaml file

The intakes.yaml file is used to tell the concentrator how to bind a port where logs are received to its technology represented by the Intake key. For each technology, specify:

  • a name: it has nothing to do with Sekoia.io, feel free to use the explicite value of your choice
  • the protocol: tcp or udp
  • a port: to process incoming events
  • the Intake key: can be retreived from the Intakes page of your community

Here is an example of the file with 3 technologies sending their events to the concentrator on ports 20516, 20517 and 20518. Feel free to copy paste it and adapt it to your need:

---
intakes:
- name: Techno1
  protocol: tcp
  port: 20516
  intake_key: INTAKE_KEY_FOR_TECHNO_1
- name: Techno2
  protocol: udp
  port: 20517
  intake_key: INTAKE_KEY_FOR_TECHNO_2
- name: Techno3
  protocol: tcp
  port: 20518
  intake_key: INTAKE_KEY_FOR_TECHNO_3

Tip

You are not limited to 3 entries. Feel free to adapt it to your needs.

Debug

A debug variable is available in order to debug a specific intake, for example

---
intakes:
- name: Techno1
  protocol: tcp
  port: 20516
  intake_key: INTAKE_KEY_FOR_TECHNO_1
- name: Techno2
  protocol: tcp
  port: 20517
  intake_key: INTAKE_KEY_FOR_TECHNO_2
  debug: True
- name: Techno3
  protocol: tcp
  port: 20518
  intake_key: INTAKE_KEY_FOR_TECHNO_3

When debug is set to true, the raw event received and the output message will be printed in STDOUT. Each one will be respectively identified using tags: : [Input $INTAKE_KEY] & [Output $INTAKE_KEY]

Configure docker-compose.yml file

Please find below a template of the docker-compose.yml file.

version: "3.9"
services:
  rsyslog:
    image: ghcr.io/sekoia-io/sekoiaio-docker-concentrator:2.5
    environment:
      - MEMORY_MESSAGES=2000000
      - DISK_SPACE=180g
      - REGION=FRA1
    ports:
      - "20516-20566:20516-20566"
      - "20516-20566:20516-20566/udp"
    volumes:
      - ./intakes.yaml:/intakes.yaml
      - ./disk_queue:/var/spool/rsyslog
    restart: always
    pull_policy: always

Note

Feel free to copy paste it and adapt it to your needs.

The following sections will describe each element of this file above.

Environment variables

environment:
    - MEMORY_MESSAGES=2000000
    - DISK_SPACE=180g
    - REGION=FRA1

Two environment variables are used to customize the container. These variables are used to define a queue for incoming logs in case there is a temporarily issue in transmitting events to Sekoia.io. The queue stores messages in memory up to a certain number of events and then store them on disk. When the issue is fixed, events stored are retrieved from the queue and forward to the plateform.

  • MEMORY_MESSAGES=2000000 means the queue is allowed to store up to 2,000,000 messages in memory. If we consider a message size is 1.2KB, then you will use 2,4GB of RAM memory (2000000 * 1.2KB = 2.4GB).
  • DISK_SPACE=180g means the queue is allowed to store on disk up to 180giga of messages.
  • REGION=FRA1 is the region where to send the logs. Currently 4 options are available: FRA1, FRA2, MCO1 and UAE1

Here you will find recommendations to set these variables based on the number of assets. You can also define your own values, which should be chosen according to your virtual machine's template.

Ports

ports:
    - "20516-20566:20516-20566"
    - "20516-20566:20516-20566/udp"

As specified in the Overview section, the concentrator will be run in an isolated environment. That means, by default, no flow is open between the host and the concentrator. 20516-20518:20516-20566 means that every packets coming through the TCP port form 20516 to 20566 to the host will be forwarded to the concentrator container from port 20516 to 20566.

If you want to open a UDP flow, please add a line with /udp at the end.

Important

Please adapt these values according to the intakes.yaml file.

Volumes

volumes:
    - ./intakes.yaml:/intakes.yaml
    - ./rsyslog:/var/spool/rsyslog

Volumes are used to share files and folders between the host and the container:

  • ./intakes.yaml:/intakes.yaml is used to tell the concentrator what protocol, ports and intake keys to use.
  • ./conf:/etc/rsyslog.d is mapped if you want to customize some rsyslog configuration (ADVANCED)
  • ./disk_queue:/var/spool/rsyslog is used when the concentrator queue stores data on disk. The mapping avoids data loss if logs are stored on disk and the container is deleted.

Import a custom rsyslog configuration

You can add your own additional rsyslog configuration. It can be useful to deal with specific use cases which are not supported natively by the Sekoia.io concentrator. To enable it, you simply have to create a new folder called extended_conf and put an additional your rsyslog file into (your file must have the extension *.conf). You do not have to deal with the intake.yaml file. Your custom configuration will be added in addition to the intake definition and will not erase exisiting ones.

You can define your own method for obtaining logs using rsyslog modules, but you still need to forward events to Sekoia.io by providing a syslog-valid message with your intake key as a header, as follows:

input(type="imtcp" port="20521" ruleset="remote20521")
template(name="SEKOIAIO_Template" type="string" string="<%pri%>1 %timegenerated:::date-rfc3339% %hostname% MY-APP-NAME - LOG [SEKOIA@53288 intake_key=\"MY-INTAKE-KEY\"] %msg%\n")
ruleset(name="remote20521"){
action(
    name="action"
    type="omfwd"
    protocol="tcp"
    target="intake.sekoia.io"
    port="10514"
    TCP_Framing="octet-counted"
    StreamDriver="gtls"
    StreamDriverMode="1"
    StreamDriverAuthMode="x509/name"
    StreamDriverPermittedPeers="intake.sekoia.io"
    Template="SEKOIAIO_Template"
    )
}

Once additional configuration has been added, you simply have to mount them in the docker as following:

volumes:
    - ./intakes.yaml:/intakes.yaml
    - ./extended_conf:/extended_conf
    - ./disk_queue:/var/spool/rsyslog

Additional options

restart: always
pull_policy: always
  • restart: always: this line indicates to restart the concentrator everytime it stops. That means if it crashes, if you restart Docker or if you restart the host, the concentrator will start automatically.
  • pull_policy: always: docker compose will always try to pull the image from the registry and check if a new version is available for the tag specified.

Start the concentrator

To start the concentrator, run the following command in the folder where docker-compose.yml and intakes.yaml are stored:

sudo docker compose up -d

You should have an answer like this:

[+] Running 1/1
  Container docker-compose-rsyslog-1  Started

Note

The first time you start the container, you will download the image stored on the GitHub repository - ghcr.io/sekoia-io/sekoiaio-docker-concentrator

To check the status of the concentrator, you can run:

sudo docker compose ps

Useful commands

To view all the concentrator logs:

sudo docker compose logs

If you are using the Debug variable, everytime a log is received by the Forwarder, it's visible using docker compose logs.

To view last logs in real time:

sudo docker compose logs -f

To stop the concentrator:

sudo docker compose stop

To delete the concentrator (the concentrator needs to be stopped):

sudo docker compose rm

Troubleshooting

You can't find the logs in your community? No worries this section will give you an advice to identify what is happening.

Step 1: check if the events are received by the forwarder

To check if the events are received by the forwarder, you can activate the debug mode for a specific Intake. The debug mode will display all logs that are received and sent by the forwarder associated with a specific Intake, in the standard output of the container (docker logs). To activate the debug mode, simply add debug: True in the definition of the Intake you want, in the intakes.yaml file.

Example:

- name: Techno2
  protocol: tcp
  port: 20517
  intake_key: INTAKE_KEY_FOR_TECHNO_2
  debug: True

Then rebuild the container and start it:

docker compose down && docker compose up -d

Now, you can run the following command and display all the last logs received for the Intakes with the mode debug activated:

sudo docker compose logs

To view container logs for a specific intake when using the debug variable:

sudo docker compose logs | grep "YOUR_INTAKE_KEY"

Finally, if you want to check events coming in real time for Intakes with the debug variable:

sudo docker compose logs -f

You don't see your events with these commands?

  1. Check the intakes.yaml file to see if you have declared the protocols and ports you wanted.

  2. Verify if this information is taken into account by the concentrator. At start-up, the concentrator always shows the list of Intakes with the protocols and ports.

    sudo docker compose logs | more
    

  3. Check that you correctly declared the ports section in the docker-compose.yml file. They MUST be the same as the ports declared in the intakes.yaml file. For instance, if you declared 4 technologies on ports 25020, 25021, 25022 and 25023, the ports line the docker-compose.yml has to be at least "25020-25023:25020-25023" for TCP and "25020-25023:25020-25023/udp" if using UDP.

  4. Verify that traffic is incoming from your log source, meaning no firewall is blocking the events.

    sudo tcpdump -i <change_with_interface_name> -c10 -nn src <remote_ip> -vv
    

    To find those values:

    • change_with_interface_nameuse the command ip addr
    • remote_ipis the IP from which the logs should be incoming

Step 2: verify everything is correctly configured to forward events to Sekoia.io

  1. Check the Intake keys you wrote in intakes.yaml are correct.

  2. Check the network flow between the concentrator host and Sekoia.io is opened to the destination intake.sekoia.io on protocol TCP and port 10514. You can easily check it with telnet:

    sudo apt install telnet && telnet intake.sekoia.io 10514
    

    This command should return the following answer, meaning the network flow is opened:

    Connected to intake.sekoia.io.
    Escape character is '^]'.
    

    Remove telnet if don't need it anymore

    sudo apt remove telnet
    
  3. Finally check the status of the Sekoia.io plateform on https://status.sekoia.io.

Additional information

The image used to run the concentrator is maintained on this github repository. Feel free to contribute and make a pull request to improve the concentrator!

How to update the concentrator

Docker uses the notion of tag to identify the version of an image. The tag is always referenced in line starting with image in docker-compose.yml:

image: ghcr.io/sekoia-io/sekoiaio-docker-concentrator:2.0

2.0 means the version used by docker compose is 2.0. You can find all the versions available on the GitHub repository here

To update the concentrator, just change the tag in docker-compose.yml, then recreate the concentrator with the command:

sudo docker compose up -d

There is a special tag called latest which always refers to the latest version. If you use this tag, every time the concentrator will be recreated, the image will be updated to the latest version.

Important

Be very careful if you use the tag latest in production! If a major change is present in a new version, it could break your current concentrator with the settings in place.

Important

Each image version is also updated every week in order to update the OS contained in the image with the latest security patches. You are responsible for recreating the concentrator periodically to initiate the image update.

How do I add a new Intake later?

To add a new Intake, it's very simple ! Follow these steps:

  1. Edit the intakes.yaml file and add the new intake
    - name: NewIntake
      protocol: tcp
      port: 20519
      intake_key: INTAKE_KEY_FOR_NEW_INTAKE
    
  2. Edit the docker-compose.yml file if needed and modify the ports section according to the intakes.yaml file. It shouldn't be necessary from port 20516 to 20566
    ports:
    - "20516-20566:20516-20566"
    - "20516-20566:20516-20566/udp"
    
  3. Recreate the concentrator by running the command:
    sudo docker compose up -d
    
  4. Check in the concentrator logs that the new Intake is taken into account
    sudo docker compose logs | more
    

Docker Engine installation

This section describes how to install Docker using the apt repository on one of those Debian 64-bit versions:

  • Debian Bullseye 11 (stable)
  • Debian Buster 10 (oldstable)

All of the installation steps come from the official Docker Engine installation for Debian. Feel free to consult the official installation page for more information. To install Docker on another Linux OS, please consult the official Docker documentation

Uninstall old versions (if applicable)

Run the following command to uninstall old Docker versions:

sudo apt-get remove docker docker-engine docker.io containerd runc

Set up the repository

  1. Update the apt package index and install packages to allow apt to use a repository over HTTPS

    sudo apt-get update
    sudo apt-get install \
        ca-certificates \
        curl \
        gnupg \
        lsb-release
    
  2. Add Docker's official GPG key

    sudo mkdir -m 0755 -p /etc/apt/keyrings
    curl -fsSL https://download.docker.com/linux/debian/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
    
  3. Use the following command to set up the repository

    echo \
      "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/debian \
      $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
    

Install Docker Engine

  1. Update the apt package index

    sudo apt-get update
    
  2. Install Docker Engine, containerd, and Docker compose

    sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
    
  3. Verify that the Docker Engine installation is successful by running the hello-world image

    sudo docker run hello-world
    

5 minutes setup on Debian

Warning

This script will automate all the steps detailed on this page for a Debian host. Please read the content of this page carefully before executing it.

Connect to the remote server where you would like to install the Sekoia.io Forwarder, then follow those steps:

  1. Execute a script to setup the docker

    wget https://raw.githubusercontent.com/SEKOIA-IO/documentation/main/docs/assets/operation_center/ingestion_methods/sekoiaio_forwarder/sekoiaio_docker_concentrator_autosetup.sh
    chmod +x sekoiaio_docker_concentrator_autosetup.sh
    ./sekoiaio_docker_concentrator_autosetup.sh
    rm sekoiaio_docker_concentrator_autosetup.sh
    
  2. Edit the configuration files

    • sekoiaio-concentrator/intakes.yaml by replacing the name, protocol, port and intake_key for each intake you would like to collect
    • sekoiaio-concentrator/docker-compose.yml by remplacing the value "20516-20518:20516-20518" by a relevant content according to the sekoiaio-concentrator/intakes.yaml previously edited
  3. Start the docker

    Follow the process you can find on the section Start the concentrator of this page.

    sudo docker compose up -d
    sudo docker compose ps
    sudo docker compose logs -f
    

Enjoy your docker!