Skip to content

Prerequisites to retrieve logs from Azure to Sekoia.io

When utilizing an Azure integration with Sekoia.io, the initial step involves centralizing your logs using Event Hub in order to pull events. The following page will guide you through the process of setting up these prerequisites

Azure Event Hubs is a cloud-based event streaming platform and event ingestion service provided by Microsoft Azure. It is designed to handle large amounts of event data generated by various applications, devices, and services in real-time. Event Hubs enables you to ingest, process, and store events, logs, telemetry data, and other streaming data for further analysis, monitoring, and processing. Two ways are suggested in order to set up everything you need to forward your events on Sekoia.io.

If you are not an expert and want an easy way to configure the ressources on Azure, we recommend to use to Automatic way as it is easier to set up.

These two ways will create an Azure Event Hub and a Storage Account.

Use the template to create the ressources

To get started, click on the button below and fill the form on Azure to set up the required environment for Sekoia Deploy to Azure

Some fields must be filled in.

Project details

  • Subscription: select the Azure subscription you want to use
  • Resource Group: select or create a new Resource Group. A Resource Group is a container that holds related resources

Instance details

  • Region: select the appropriated region
  • Project Name: give a name for this project. Here is how the names of the resources will be affected by this project name
Ressource Name
Event Hub Namespace <project_name>ehns
Event Hub <project_name>eh
Shared Access Policy <project_name>ap
Consumer Group <project_name>cp
Storage Account <project_name>sa
  • Event Hub Sku: Select the messaging tier for Event Hub Namespace between Basic , Standard or Premium. We do not recommend Basic model due to its limitation. Please follow this Microsoft web page to get more information avec the different messaging tier.
  • Troughput Unit: A unit gives you up to 1 MB/s or 1,000 events per second (whichever comes first) per Event Hub Namespace. Please adapt it to your need.
  • Enable Auto Inflate: When checked, the Auto-inflate feature of Event Hubs automatically scales up by increasing the number of troughput units, to meet usage needs.
  • Auto Inflate Maximum Throughput Unit: When Enable Auto Inflate is checked, you can specify the maximum throughput units you allow.
  • Partition Count: The number of event hub partitions. Microsoft recommends a maximum throughput of 1 MB/s per partition. Unless you plan to add more Event hubs to the Event Hub Namespace, the Partition Count and Throughput Unit variables should have the same values.
  • Retention Time: How long you will keep events in the Event hub in days.

Use the output variables to create a Sekoia playbook

When the message Your deployment is complete is displayed, click on Outputs.

image

Keep these 5 pieces of information displayed carefully, it will used to configure the Trigger Configuration of the Sekoia playbook.

Overview

This setup guide will show you how to create an Event Hub manually.

Theses changes have to be made from the Azure Web Portal.

Some resources created during this procedure are needed to connect Sekoia.io to the Event Hub, you can complete the following table to save all the useful informations for later use.

Name Value Description Example
hub_name To be completed at step 3 Name of the Event Hub sekoia_event_hub
hub_connection_string To be completed at step 5 Connection string–primary key Endpoint=sb://company-eventhub.servicebus.windows.net/;SharedAccessKeyName=sekoiaio;SharedAccessKey=XXXXXX;EntityPath=sekoia_event_hub
hub_consumer_group To be completed at step 6 Name of the comsumer group consumergroup_sekoiaio
storage_container_name To be completed at step 7 Name of your container sekoiaio
storage_connection_string To be completed at step 8 Storage connection string DefaultEndpointsProtocol=https;AccountName=sekoiaiocheckpoint;AccountKey=XXXXX

As a prerequisite, you need to choose an existing Resource group, or create a new one (e.g. company-resource-group).

Step 1: Find your Subscription ID

  1. Go to: Home > Cost Management + Billing > Subscriptions.
  2. From there, keep the relevant “Subscription ID” that will be used along the process.
    image

Step 2: Create Event Hub Namespace

  1. Navigate to Home > Event Hubs.
  2. Create an Event Hub Namespace.
  3. Select your Subscription and your Resource group. Click on create new if you want your Event Hub Namespace in a new Resource group.
  4. Choose a namespace name.
  5. Select a location based on your events location.
  6. Select the pricing tier plan based on your utilisation: Standard or Premimum (We don't recommend to choose the Basic plan due to its limitations.)
  7. Select the throughput units number based on your events, or enable the auto inflate mode: 1 throughput unit can process up to 1 MB per second or 1000 events per second (whichever comes first).
    image

Step 3: Create Event Hub Instance

When your Event Hub Namespace is created you can create an Event Hub inside:

  1. Navigate to Home > Event Hubs > company-eventhubnamespace.
  2. Create an Event Hub.
  3. Select 4 as default partition count
  4. Select Delete as cleanup policy
  5. Type 168h (7 days) as retention time
  6. Click Create

image

Info

Carefully store the Event Hub name that will be used for sekoia playbook configuration.

Step 4: Create “Shared Access Policies” for the Event Hub

  1. Navigate to Home > Event Hubs > company-eventhubnamespace > eventhubname | Shared access policies.

  2. Create a policy (e.g. sekoiaio) with the claims Listen.

    image

  3. Once created, click on the policy.

    image

Info

Carefully store the connection string–primary key that will be used for sekoia playbook configuration.

Step 5: Create a Consumer group

  1. Navigate to Home > Event Hubs > company-eventhubnamespace > eventhubname | Consumer groups.
  2. Create a Consumer group (e.g. consumergroup_sekoiaio).
    image

Info

Carefully store the Consumer group name that will be used for sekoia playbook configuration.

Step 6: Create a Storage Account with a container

In order to allow Sekoia.io keep track of the consumed events, the next step consists in creating a dedicated Storage account.

  1. Navigate to Home > Storage accounts.
  2. Create a Storage account.
  3. Select your Subscription and your Resource group.
  4. Choose a Storage account name.

  5. Select a Region.

    Info

    You must choose the Region used during the Event Hub Namespace creation.

  6. Select your perfomance and redundancy parameters.

    Info

    We advise at least the following values: - performance: standard - redundancy: Geo-Redundancy storage (GRS)

image

When your storage account is created you can create an container inside. 1. Navigate to Home > Storage accounts > storageaccoutname | containers. 2. Create an container.

Keep the Public Acces Level to Private.

Info

The container name should be the same as the Event Hub name. Carefully store that container name that will be used for sekoia playbook configuration.

Step 7: Retrieve Connection String

You have to retrieve the connection string from Azure Web Portal.

  1. Go to Home > Storage accounts > storageaccoutname | Access Keys.
  2. Click on "Show Keys" on the first Connection String.
    image

Info

Carefully store the Connection String that will be used for sekoia playbook configuration.

Further Readings