Skip to content

S3 for logs

Overview

AWS S3 is a service that enables to store and manage data with scalability, high availability, and low latency with high durability. AWS S3 can hold objects up to five Terabytes in size. Several AWS services offers to store their logs on a S3 bucket. This integration aims to collect line-oriented logs.

Configure

Prerequisites

  • An administator access to the Amazon console with the permission to create SQS queue and S3 notifications.
  • A self managed AWS S3 bucket where events will be published prior to be consumed by Sekoia.io.

Create a SQS queue

This integration relies on S3 Event Notifications (SQS) to discover new S3 objects.

To enable the S3 Event Notification:

  1. Create a queue in the SQS service by following this guide
  2. In the Access Policy step, choose the advanced configuration and adapt this configuration sample with your own SQS Amazon Resource Name (ARN) (the main change is the Service directive allowing S3 bucket access): 
    {
  "Version": "2008-10-17",
  "Id": "__default_policy_ID",
  "Statement": [
    {
      "Sid": "__owner_statement",
      "Effect": "Allow",
      "Principal": {
        "Service": "s3.amazonaws.com"
      },
      "Action": "SQS:*",
      "Resource": "arn:aws:sqs:XXX:XXX"
    }
  ]
}

Note

Keep in mind that you have to create the SQS queue in the same region as the S3 bucket you want to watch.

Create a S3 Event Notification

Use the following guide to create S3 Event Notification and then:

  1. Select the notification for object creation in the Event type section
  2. As the destination, choose the SQS service
  3. Select the queue you created in the previous section

Create the intake

Go to the intake page and create a new intake from the format that will process your logs.

Pull events

Go to the playbook page and create a new playbook with the AWS Fetch new logs on S3 connector.

Set up the module configuration with the AWS Access Key, the secret key and the region name. Set up the trigger configuration with the name of the SQS queue and the intake key, from the intake previously created.

Start the playbook and enjoy your events.

Further Readings