VPC Flow Logs

Overview

Amazon VPC Flow Logs is a feature that provides the ability to capture information about IP network traffic as it enters or exits from network interface in your Amazon VPC (Amazon Virtual Private Cloud). VPC Flow Logs can help you with a number of tasks, such as:

Diagnosing overly restrictive security group rules
Monitoring the traffic that is reaching your instance
Determining the direction of the traffic to and from the network interfaces

The following Sekoia.io built-in rules match the intake AWS VPC Flow logs. This documentation is updated automatically and is based solely on the fields used by the intake which are checked against our rules. This means that some rules will be listed but might not be relevant with the intake.

SEKOIA.IO x AWS VPC Flow logs on ATT&CK Navigator

SEKOIA.IO Intelligence Feed

Detect threats based on indicators of compromise (IOCs) collected by SEKOIA's Threat and Detection Research team.

Effort: elementary

Event Categories

The following table lists the data source offered by this integration.

Data Source	Description
`Host network interface`	every packets passing through the AWS account are logged
`Netflow/Enclave netflow`	AWS Flow Logs are Netflow-like
`Network device logs`	packets logged by Flow Logs
`Network protocol analysis`	traffic analysis at levels 2/3/4

In details, the following table denotes the type of events produced by this integration.

Name	Values
Kind	``
Category	`["network"]`
Type	``

Event Samples

Find below few samples of events and how they are normalized by Sekoia.io.

accept.jsonaccept_structured.jsoncustom.jsonipv6traffic.jsonnodata.jsonreject.jsonreject_structured.json

{
    "message": "2 424805057484 eni-0f06a40fc9be596f6 212.83.179.156 10.0.0.96 123 123 17 2 152 1599665193 1599665488 ACCEPT OK",
    "event": {
        "action": "accept",
        "category": [
            "network"
        ],
        "end": "2020-09-09T15:31:28Z",
        "outcome": "ok",
        "start": "2020-09-09T15:26:33Z",
        "type": [
            "allowed"
        ]
    },
    "@timestamp": "2020-09-09T15:26:33Z",
    "action": {
        "name": "accept",
        "outcome": "ok",
        "target": "network-traffic",
        "type": "forward"
    },
    "cloud": {
        "account": {
            "id": "424805057484"
        },
        "provider": "aws",
        "service": {
            "name": "vpc"
        }
    },
    "destination": {
        "address": "10.0.0.96",
        "ip": "10.0.0.96",
        "port": 123
    },
    "network": {
        "iana_number": "17",
        "transport": "udp"
    },
    "observer": {
        "ingress": {
            "interface": {
                "name": "eni-0f06a40fc9be596f6"
            }
        }
    },
    "related": {
        "ip": [
            "10.0.0.96",
            "212.83.179.156"
        ]
    },
    "source": {
        "address": "212.83.179.156",
        "bytes": 152,
        "ip": "212.83.179.156",
        "packets": 2,
        "port": 123
    },
    "user": {
        "id": "424805057484"
    }
}

{
    "message": "{\"version\":2,\"account_id\":\"424805057484\",\"interface_id\":\"eni-0f06a40fc9be596f6\",\"srcaddr\":\"5.6.7.8\",\"dstaddr\":\"1.2.3.4\",\"srcport\":4712,\"dstport\":53205,\"protocol\":6,\"packets\":12,\"bytes\":2610,\"start\":1661950735,\"end\":1661950746,\"action\":\"ACCEPT\",\"log_status\":\"OK\"}\n",
    "event": {
        "action": "accept",
        "category": [
            "network"
        ],
        "end": "2022-08-31T12:59:06Z",
        "outcome": "ok",
        "start": "2022-08-31T12:58:55Z",
        "type": [
            "allowed"
        ]
    },
    "@timestamp": "2022-08-31T12:58:55Z",
    "action": {
        "name": "accept",
        "outcome": "ok",
        "target": "network-traffic",
        "type": "forward"
    },
    "cloud": {
        "account": {
            "id": "424805057484"
        },
        "provider": "aws",
        "service": {
            "name": "vpc"
        }
    },
    "destination": {
        "address": "1.2.3.4",
        "ip": "1.2.3.4",
        "port": 53205
    },
    "network": {
        "iana_number": "6",
        "transport": "tcp"
    },
    "observer": {
        "ingress": {
            "interface": {
                "name": "eni-0f06a40fc9be596f6"
            }
        }
    },
    "related": {
        "ip": [
            "1.2.3.4",
            "5.6.7.8"
        ]
    },
    "source": {
        "address": "5.6.7.8",
        "bytes": 2610,
        "ip": "5.6.7.8",
        "packets": 12,
        "port": 4712
    },
    "user": {
        "id": "424805057484"
    }
}

{
    "message": "5 424805057484 eni-1235b8ca123456789 52.95.128.179 10.0.0.71 46945 53 17 1 73 1658131186 1658131216 ACCEPT OK vpc-abcdefab012345678 subnet-aaaaaaaa012345678 - 0 IPv4 52.95.128.179 10.0.0.71 eu-west-1 euw1-az3 - - - - egress 8",
    "event": {
        "action": "accept",
        "category": [
            "network"
        ],
        "end": "2022-07-18T08:00:16Z",
        "outcome": "ok",
        "start": "2022-07-18T07:59:46Z",
        "type": [
            "allowed"
        ]
    },
    "@timestamp": "2022-07-18T07:59:46Z",
    "action": {
        "name": "accept",
        "outcome": "ok",
        "target": "network-traffic",
        "type": "forward"
    },
    "cloud": {
        "account": {
            "id": "424805057484"
        },
        "provider": "aws",
        "service": {
            "name": "vpc"
        }
    },
    "destination": {
        "address": "10.0.0.71",
        "ip": "10.0.0.71",
        "port": 53
    },
    "network": {
        "iana_number": "17",
        "transport": "udp"
    },
    "observer": {
        "ingress": {
            "interface": {
                "name": "eni-1235b8ca123456789"
            }
        }
    },
    "related": {
        "ip": [
            "10.0.0.71",
            "52.95.128.179"
        ]
    },
    "source": {
        "address": "52.95.128.179",
        "bytes": 73,
        "ip": "52.95.128.179",
        "packets": 1,
        "port": 46945
    },
    "user": {
        "id": "424805057484"
    }
}

{
    "message": "2 123456789010 eni-1235b8ca123456789 2001:db8:1234:a100:8d6e:3477:df66:f105 2001:db8:1234:a102:3304:8879:34cf:4071 34892 22 6 54 8855 1477913708 1477913820 ACCEPT OK",
    "event": {
        "action": "accept",
        "category": [
            "network"
        ],
        "end": "2016-10-31T11:37:00Z",
        "outcome": "ok",
        "start": "2016-10-31T11:35:08Z",
        "type": [
            "allowed"
        ]
    },
    "@timestamp": "2016-10-31T11:35:08Z",
    "action": {
        "name": "accept",
        "outcome": "ok",
        "target": "network-traffic",
        "type": "forward"
    },
    "cloud": {
        "account": {
            "id": "123456789010"
        },
        "provider": "aws",
        "service": {
            "name": "vpc"
        }
    },
    "destination": {
        "address": "2001:db8:1234:a102:3304:8879:34cf:4071",
        "ip": "2001:db8:1234:a102:3304:8879:34cf:4071",
        "port": 22
    },
    "network": {
        "iana_number": "6",
        "transport": "tcp"
    },
    "observer": {
        "ingress": {
            "interface": {
                "name": "eni-1235b8ca123456789"
            }
        }
    },
    "related": {
        "ip": [
            "2001:db8:1234:a100:8d6e:3477:df66:f105",
            "2001:db8:1234:a102:3304:8879:34cf:4071"
        ]
    },
    "source": {
        "address": "2001:db8:1234:a100:8d6e:3477:df66:f105",
        "bytes": 8855,
        "ip": "2001:db8:1234:a100:8d6e:3477:df66:f105",
        "packets": 54,
        "port": 34892
    },
    "user": {
        "id": "123456789010"
    }
}

{
    "message": "2 123456789010 eni-1235b8ca123456789 - - - - - - - 1431280876 1431280934 - NODATA",
    "event": {
        "category": [
            "network"
        ],
        "end": "2015-05-10T18:02:14Z",
        "outcome": "nodata",
        "start": "2015-05-10T18:01:16Z"
    },
    "@timestamp": "2015-05-10T18:01:16Z",
    "action": {
        "outcome": "nodata",
        "type": "forward"
    },
    "cloud": {
        "account": {
            "id": "123456789010"
        },
        "provider": "aws",
        "service": {
            "name": "vpc"
        }
    },
    "observer": {
        "ingress": {
            "interface": {
                "name": "eni-1235b8ca123456789"
            }
        }
    },
    "user": {
        "id": "123456789010"
    }
}

{
    "message": "2 424805057484 eni-0f06a40fc9be596f6 195.14.170.50 10.0.0.96 53996 20248 6 1 40 1599665374 1599665428 REJECT OK",
    "event": {
        "action": "reject",
        "category": [
            "network"
        ],
        "end": "2020-09-09T15:30:28Z",
        "outcome": "ok",
        "start": "2020-09-09T15:29:34Z",
        "type": [
            "denied"
        ]
    },
    "@timestamp": "2020-09-09T15:29:34Z",
    "action": {
        "name": "reject",
        "outcome": "ok",
        "target": "network-traffic",
        "type": "forward"
    },
    "cloud": {
        "account": {
            "id": "424805057484"
        },
        "provider": "aws",
        "service": {
            "name": "vpc"
        }
    },
    "destination": {
        "address": "10.0.0.96",
        "ip": "10.0.0.96",
        "port": 20248
    },
    "network": {
        "iana_number": "6",
        "transport": "tcp"
    },
    "observer": {
        "ingress": {
            "interface": {
                "name": "eni-0f06a40fc9be596f6"
            }
        }
    },
    "related": {
        "ip": [
            "10.0.0.96",
            "195.14.170.50"
        ]
    },
    "source": {
        "address": "195.14.170.50",
        "bytes": 40,
        "ip": "195.14.170.50",
        "packets": 1,
        "port": 53996
    },
    "user": {
        "id": "424805057484"
    }
}

{
    "message": "{\"version\":2,\"account_id\":\"424805057484\",\"interface_id\":\"eni-0f06a40fc9be596f6\",\"srcaddr\":\"1.2.3.4\",\"dstaddr\":\"5.6.7.8\",\"srcport\":53094,\"dstport\":2323,\"protocol\":6,\"packets\":1,\"bytes\":40,\"start\":1661950735,\"end\":1661950746,\"action\":\"REJECT\",\"log_status\":\"OK\"}\n",
    "event": {
        "action": "reject",
        "category": [
            "network"
        ],
        "end": "2022-08-31T12:59:06Z",
        "outcome": "ok",
        "start": "2022-08-31T12:58:55Z",
        "type": [
            "denied"
        ]
    },
    "@timestamp": "2022-08-31T12:58:55Z",
    "action": {
        "name": "reject",
        "outcome": "ok",
        "target": "network-traffic",
        "type": "forward"
    },
    "cloud": {
        "account": {
            "id": "424805057484"
        },
        "provider": "aws",
        "service": {
            "name": "vpc"
        }
    },
    "destination": {
        "address": "5.6.7.8",
        "ip": "5.6.7.8",
        "port": 2323
    },
    "network": {
        "iana_number": "6",
        "transport": "tcp"
    },
    "observer": {
        "ingress": {
            "interface": {
                "name": "eni-0f06a40fc9be596f6"
            }
        }
    },
    "related": {
        "ip": [
            "1.2.3.4",
            "5.6.7.8"
        ]
    },
    "source": {
        "address": "1.2.3.4",
        "bytes": 40,
        "ip": "1.2.3.4",
        "packets": 1,
        "port": 53094
    },
    "user": {
        "id": "424805057484"
    }
}

Extracted Fields

The following table lists the fields that are extracted, normalized under the ECS format, analyzed and indexed by the parser. It should be noted that infered fields are not listed.

Name	Type	Description
`@timestamp`	`date`	Date/time when the event originated.
`action.name`	`keyword`	The name of the action
`action.outcome`	`keyword`	The outcome of the action
`action.target`	`keyword`	The target of the action
`action.type`	`keyword`	The type of the action
`cloud.account.id`	`keyword`	The cloud account or organization id.
`cloud.provider`	`keyword`	Name of the cloud provider.
`cloud.service.name`	`keyword`	The cloud service name.
`destination.ip`	`ip`	IP address of the destination.
`destination.port`	`long`	Port of the destination.
`event.action`	`keyword`	The action captured by the event.
`event.category`	`keyword`	Event category. The second categorization field in the hierarchy.
`event.end`	`date`	event.end contains the date when the event ended or when the activity was last observed.
`event.start`	`date`	event.start contains the date when the event started or when the activity was first observed.
`network.iana_number`	`keyword`	IANA Protocol Number.
`observer.ingress.interface.name`	`keyword`	Interface name
`source.bytes`	`long`	Bytes sent from the source to the destination.
`source.ip`	`ip`	IP address of the source.
`source.packets`	`long`	Packets sent from the source to the destination.
`source.port`	`long`	Port of the source.
`user.id`	`keyword`	Unique identifier of the user.

Configure

Deploying the Data Collection Architecture

This section will guide you through creating all the AWS resources needed to collect AWS logs. If you already have existing resources that you want to use, you may do so, but any potential issues or incompatibilities with this tutorial will be your responsibility.

Prerequisites

In order to set up the AWS architecture, you need an administator access to the Amazon console with the permissions to create and manage S3 buckets, SQS queues, S3 notifications and users.

AutomaticManual

To get started, click on the button below and fill the form on AWS to set up the required environment for Sekoia

You need to fill 4 inputs:

Stack name - Name of the stack in CloudFormation (Name of the template)
BucketName - Name of the S3 Bucket
IAMUserName - Name of the dedicated user to access the S3 and SQS queue
SQSName - Name of the SQS queue

Read the different pages and click on Next, then click on Submit.

You can follow the creation in the Events tab (it can take few minutes).

Once finished, it should be displayed on the left CREATE_COMPLETE. Click on the Outputs tab in order to retrieve the information needed for Sekoia playbook.

Create a S3 Bucket

Please refer to this guide to create a S3 Bucket.

Create a SQS queue

The collect will rely on S3 Event Notifications (SQS) to get new S3 objects.

Create a queue in the SQS service by following this guide

In the Access Policy step, choose the advanced configuration and adapt this configuration sample with your own SQS Amazon Resource Name (ARN) (the main change is the Service directive allowing S3 bucket access):

{
  "Version": "2008-10-17",
  "Id": "__default_policy_ID",
  "Statement": [
    {
      "Sid": "__owner_statement",
      "Effect": "Allow",
      "Principal": {
        "Service": "s3.amazonaws.com"
      },
      "Action": "SQS:SendMessage",
      "Resource": "arn:aws:sqs:XXX:XXX"
    }
  ]
}

Important

Keep in mind that you have to create the SQS queue in the same region as the S3 bucket you want to watch.

Create a S3 Event Notification

Use the following guide to create S3 Event Notification. Once created:

Select the notification for object creation in the Event type section
As the destination, choose the SQS service
Select the queue you created in the previous section

VPC Flow Logs

As a prerequisite, you need an existing VPC, subnet or network interface (Elastic Load Balancing, Amazon RDS, Amazon ElastiCache, Amazon Redshift, Amazon WorkSpaces, NAT gateways, Transit gateways) to create a flow log. If you create a flow log for a subnet or VPC, each network interface in that subnet or VPC is monitored.

In the AWS console, navigate to: Services > VPC. From there, select the resource for which you want to capture information. The flow logs are available on the following resources: VPC, subnet, or network interfaces.

For VPC and subnet:

Select the specific resource to monitor
Go to the tab Flow logs
Click on Create flow log
Set up the flow log: we recommend to capture all traffic (accepted and rejected).

Please follow this guide to configure and set up all the permissions needed.

Create the intake

Go to the intake page and create a new intake from the format AWS Flowlogs.

Pull events

To start to pull events, you have to:

Go to the playbook page and create a new playbook with:
- the AWS Fetch new Flowlogs on S3 connector for plain text files (gzipped included)
- the AWS Fetch new FlowLogs Parquet records on S3 connector for parquet files
Set up the module configuration with the AWS Access Key, the secret key and the region name. Set up the trigger configuration with the name of the SQS queue and the intake key, from the intake previously created.
Start the playbook and enjoy your events.

VPC Flow Logs

Overview

Related Built-in Rules

Event Categories

Event Samples

Extracted Fields

Configure

Deploying the Data Collection Architecture

Prerequisites

VPC Flow Logs

Create the intake

Pull events

Further Readings