S3 for logs

Overview

AWS S3 is a service that enables to store and manage data with scalability, high availability, and low latency with high durability. AWS S3 can hold objects up to five Terabytes in size. Several AWS services offers to store their logs on a S3 bucket. This integration aims to collect line-oriented logs.

Configure

Create a SQS queue

This integration relies on S3 Event Notifications to discover new S3 objects.

To be enable to set the S3 Event Notification, create a queue in the SQS service according this guide. Please, keep in mind, to create the SQS queue in the same region as the S3 bucket you want to watch.

Create a S3 Event Notification

Use the following guide to create S3 Event Notification. Select the notification for object create in the Event type section. As the destination, choose the SQS service and select the queue you create in the previous section.

Create the intake

Go to the intake page and create a new intake from the format that will process your logs.

Pull events

Go to the playbook page and create a new playbook with the AWS Fetch new logs on S3 connector.

Set up the module configuration with the AWS Access Key, the secret key and the region name. Set up the trigger configuration with the name of the SQS queue and the intake key, from the intake previously created.

Start the playbook and enjoy your events.