Google VPC Flow Logs
Overview
Google Cloud Logging centralizes logs from Google Cloud products.
In this documentation, you will learn how to collect and send Google Cloud logs to SEKOIA.IO.
Configure
Before you begin working with PubSub, verify that you have the right permission.
Follow Google's documentation to configure a dedicated PubSub receiver. At the end of the documentation you should have done the following:
- Setup a project
- Create a topic
- Add a subscription (you should have the role
logging.adminexplicitly set on your account; for more information, see associated documentation) - Try your setup by publishing a message to the topic
Next, create a dedicated service account. At the end of the documentation you should have done the following:
- Create a service account with the role
Pub/Sub Subscriber
Note
To successfully activate the playbook further down this page, ensure the user has been granted the Pub/Sub Subscriber role for both the Topic and Subscription pages. Failure to do so will result in an error with status code 403.
- Create and download JSON keys (service account credentials)
You should now have:
- A credentials file
- A project ID
- A subscription ID
To pull events, you have to:
- Go to the playbooks' page
- Click on
+New playbookto create a new playbook - Select
Use a templatewhen creating a playbook - Search for
Google Cloudthen selectForward Google Pubsub records to Sekoia.io
This playbook consumes records from Google Pubsub and pushes them to Sekoia.io.
You can also create your own on the same basis by using the "Google Pub/Sub" trigger (Connect to the specified)
- Use the JSON keys (service account credentials) information downloaded to complete the fields on the trigger
Fields description
| Field | Meaning |
|---|---|
| name | Configuration name |
| auth_provider_x509_cert_url | The URL of the public x509 certificate, used to verify the signature on JWTs, such as ID tokens, signed by the authentication provider. https://wwww.googleapis.com/oauth2/v1/certs |
| auth_url | Google authentification url https://accounts.google.com/o/oauth2/auth |
| client_email | Client email |
| client_id | Client id |
| client_x509_cert_url | The URL of the public x509 certificate, used to verify JWTs signed by the client |
| private_key | Private key |
| private_key_id | Private key id |
| project_id | Project id |
| token_uri | token server endpoint URI https://oauth2.googleapis.com/token |
| type | Activity type service_account |
To start sending Logs to SEKOIA.IO, please create a Logs Router Sinks with an Inclusion Filter that fits your needs (Read the documentation dedicated to the product you want to monitor).
Last configuration on Google to setup is describe on each Intake.
Google VPC Flow Logs configuration
To start working with VPC Flow Logs, activate the option by editing or creating a subnet (see Google's documentation)
This subnet can be assigned to your node or kubernetes cluster.
The network logs should now be visible in Cloud Logging. A filter can now be created to stream your logs to Sekoia.io. To create this filter, follow this documentation.
Start the playbook and enjoy your events.
Raw Events Samples
In this section, you will find examples of raw logs as generated natively by the source. These examples are provided to help integrators understand the data format before ingestion into Sekoia.io. It is crucial for setting up the correct parsing stages and ensuring that all relevant information is captured.
{
"insertId": "1sxgleif1dyxla",
"jsonPayload": {
"dest_gke_details": {
"cluster": {
"cluster_location": "europe-central2-a",
"cluster_name": "cluster-3"
}
},
"src_location": {
"continent": "Europe",
"country": "pol",
"asn": 15169
},
"dest_vpc": {
"vpc_name": "foo",
"project_id": "hazel-aria-348413",
"subnetwork_name": "foo"
},
"start_time": "2022-06-03T12:09:42.501046130Z",
"end_time": "2022-06-03T12:09:42.768509812Z",
"bytes_sent": "1872",
"reporter": "DEST",
"connection": {
"src_ip": "34.118.64.229",
"dest_port": 45950,
"dest_ip": "10.0.0.4",
"src_port": 443,
"protocol": 6
},
"dest_instance": {
"region": "europe-central2",
"zone": "europe-central2-a",
"vm_name": "gke-cluster-3-default-pool-4e355575-tdhx",
"project_id": "hazel-aria-348413"
},
"packets_sent": "16"
},
"resource": {
"type": "gce_subnetwork",
"labels": {
"subnetwork_id": "7449846049104218257",
"subnetwork_name": "foo",
"project_id": "hazel-aria-348413",
"location": "europe-central2-a"
}
},
"timestamp": "2022-06-03T12:09:43.654174991Z",
"logName": "projects/hazel-aria-348413/logs/compute.googleapis.com%2Fvpc_flows",
"receiveTimestamp": "2022-06-03T12:09:43.654174991Z"
}
{
"insertId": "17aa0kaf4hig5c",
"jsonPayload": {
"end_time": "2022-06-03T12:09:44.424429165Z",
"packets_sent": "32",
"src_location": {
"asn": 15169,
"country": "pol",
"continent": "Europe"
},
"start_time": "2022-06-03T12:09:44.421947861Z",
"dest_vpc": {
"subnetwork_name": "foo",
"vpc_name": "foo",
"project_id": "hazel-aria-348413"
},
"bytes_sent": "33792",
"reporter": "DEST",
"dest_instance": {
"region": "europe-central2",
"project_id": "hazel-aria-348413",
"vm_name": "gke-cluster-3-default-pool-4e355575-k1w8",
"zone": "europe-central2-a"
},
"dest_gke_details": {
"cluster": {
"cluster_location": "europe-central2-a",
"cluster_name": "cluster-3"
}
},
"connection": {
"protocol": 6,
"dest_ip": "10.0.0.3",
"src_ip": "34.118.64.229",
"src_port": 443,
"dest_port": 41834
}
},
"resource": {
"type": "gce_subnetwork",
"labels": {
"project_id": "hazel-aria-348413",
"subnetwork_name": "foo",
"subnetwork_id": "7449846049104218257",
"location": "europe-central2-a"
}
},
"timestamp": "2022-06-03T12:09:52.418604934Z",
"logName": "projects/hazel-aria-348413/logs/compute.googleapis.com%2Fvpc_flows",
"receiveTimestamp": "2022-06-03T12:09:52.418604934Z"
}
Detection section
The following section provides information for those who wish to learn more about the detection capabilities enabled by collecting this intake. It includes details about the built-in rule catalog, event categories, and ECS fields extracted from raw events. This is essential for users aiming to create custom detection rules, perform hunting activities, or pivot in the events page.
Event Categories
The following table lists the data source offered by this integration.
| Data Source | Description |
|---|---|
Process use of network |
VPC Flow Logs records sample for network flows samples from Google Cloud instances |
In details, the following table denotes the type of events produced by this integration.
| Name | Values |
|---|---|
| Kind | `` |
| Category | network |
| Type | info |
Transformed Events Samples after Ingestion
This section demonstrates how the raw logs will be transformed by our parsers. It shows the extracted fields that will be available for use in the built-in detection rules and hunting activities in the events page. Understanding these transformations is essential for analysts to create effective detection mechanisms with custom detection rules and to leverage the full potential of the collected data.
{
"message": "{\n \"insertId\": \"1sxgleif1dyxla\",\n \"jsonPayload\": {\n \"dest_gke_details\": {\n \"cluster\": {\n \"cluster_location\": \"europe-central2-a\",\n \"cluster_name\": \"cluster-3\"\n }\n },\n \"src_location\": {\n \"continent\": \"Europe\",\n \"country\": \"pol\",\n \"asn\": 15169\n },\n \"dest_vpc\": {\n \"vpc_name\": \"foo\",\n \"project_id\": \"hazel-aria-348413\",\n \"subnetwork_name\": \"foo\"\n },\n \"start_time\": \"2022-06-03T12:09:42.501046130Z\",\n \"end_time\": \"2022-06-03T12:09:42.768509812Z\",\n \"bytes_sent\": \"1872\",\n \"reporter\": \"DEST\",\n \"connection\": {\n \"src_ip\": \"34.118.64.229\",\n \"dest_port\": 45950,\n \"dest_ip\": \"10.0.0.4\",\n \"src_port\": 443,\n \"protocol\": 6\n },\n \"dest_instance\": {\n \"region\": \"europe-central2\",\n \"zone\": \"europe-central2-a\",\n \"vm_name\": \"gke-cluster-3-default-pool-4e355575-tdhx\",\n \"project_id\": \"hazel-aria-348413\"\n },\n \"packets_sent\": \"16\"\n },\n \"resource\": {\n \"type\": \"gce_subnetwork\",\n \"labels\": {\n \"subnetwork_id\": \"7449846049104218257\",\n \"subnetwork_name\": \"foo\",\n \"project_id\": \"hazel-aria-348413\",\n \"location\": \"europe-central2-a\"\n }\n },\n \"timestamp\": \"2022-06-03T12:09:43.654174991Z\",\n \"logName\": \"projects/hazel-aria-348413/logs/compute.googleapis.com%2Fvpc_flows\",\n \"receiveTimestamp\": \"2022-06-03T12:09:43.654174991Z\"\n}",
"event": {
"category": [
"network"
],
"end": "2022-06-03T12:09:42.768509Z",
"start": "2022-06-03T12:09:42.501046Z",
"type": [
"info"
]
},
"@timestamp": "2022-06-03T12:09:43.654174Z",
"cloud": {
"availability_zone": "europe-central2-a",
"project": {
"id": "hazel-aria-348413"
},
"region": "europe-central2"
},
"destination": {
"address": "10.0.0.4",
"ip": "10.0.0.4",
"port": 45950
},
"google_vpc_flow_logs": {
"insertId": "1sxgleif1dyxla",
"jsonPayload": {
"connection": {
"protocol": 6
},
"dest_gke_details": {
"cluster": {
"cluster_location": "europe-central2-a"
}
},
"dest_vpc": {
"vpc_name": "foo"
},
"reporter": "DEST"
},
"logName": "projects/hazel-aria-348413/logs/compute.googleapis.com%2Fvpc_flows",
"receiveTimestamp": "2022-06-03T12:09:43.654174991Z",
"resource": {
"labels": {
"subnetwork_id": "7449846049104218257",
"subnetwork_name": "foo"
},
"type": "gce_subnetwork"
}
},
"host": {
"name": "gke-cluster-3-default-pool-4e355575-tdhx"
},
"network": {
"bytes": 1872,
"iana_number": "6",
"name": "foo",
"packets": 16
},
"orchestrator": {
"cluster": {
"name": "cluster-3"
},
"type": "kubernetes"
},
"related": {
"ip": [
"10.0.0.4",
"34.118.64.229"
]
},
"server": {
"geo": {
"name": "europe-central2-a"
}
},
"source": {
"address": "34.118.64.229",
"as": {
"number": 15169
},
"geo": {
"continent_name": "Europe",
"country_iso_code": "POL"
},
"ip": "34.118.64.229",
"port": 443
}
}
{
"message": "{\n \"insertId\": \"17aa0kaf4hig5c\",\n \"jsonPayload\": {\n \"end_time\": \"2022-06-03T12:09:44.424429165Z\",\n \"packets_sent\": \"32\",\n \"src_location\": {\n \"asn\": 15169,\n \"country\": \"pol\",\n \"continent\": \"Europe\"\n },\n \"start_time\": \"2022-06-03T12:09:44.421947861Z\",\n \"dest_vpc\": {\n \"subnetwork_name\": \"foo\",\n \"vpc_name\": \"foo\",\n \"project_id\": \"hazel-aria-348413\"\n },\n \"bytes_sent\": \"33792\",\n \"reporter\": \"DEST\",\n \"dest_instance\": {\n \"region\": \"europe-central2\",\n \"project_id\": \"hazel-aria-348413\",\n \"vm_name\": \"gke-cluster-3-default-pool-4e355575-k1w8\",\n \"zone\": \"europe-central2-a\"\n },\n \"dest_gke_details\": {\n \"cluster\": {\n \"cluster_location\": \"europe-central2-a\",\n \"cluster_name\": \"cluster-3\"\n }\n },\n \"connection\": {\n \"protocol\": 6,\n \"dest_ip\": \"10.0.0.3\",\n \"src_ip\": \"34.118.64.229\",\n \"src_port\": 443,\n \"dest_port\": 41834\n }\n },\n \"resource\": {\n \"type\": \"gce_subnetwork\",\n \"labels\": {\n \"project_id\": \"hazel-aria-348413\",\n \"subnetwork_name\": \"foo\",\n \"subnetwork_id\": \"7449846049104218257\",\n \"location\": \"europe-central2-a\"\n }\n },\n \"timestamp\": \"2022-06-03T12:09:52.418604934Z\",\n \"logName\": \"projects/hazel-aria-348413/logs/compute.googleapis.com%2Fvpc_flows\",\n \"receiveTimestamp\": \"2022-06-03T12:09:52.418604934Z\"\n}",
"event": {
"category": [
"network"
],
"end": "2022-06-03T12:09:44.424429Z",
"start": "2022-06-03T12:09:44.421947Z",
"type": [
"info"
]
},
"@timestamp": "2022-06-03T12:09:52.418604Z",
"cloud": {
"availability_zone": "europe-central2-a",
"project": {
"id": "hazel-aria-348413"
},
"region": "europe-central2"
},
"destination": {
"address": "10.0.0.3",
"ip": "10.0.0.3",
"port": 41834
},
"google_vpc_flow_logs": {
"insertId": "17aa0kaf4hig5c",
"jsonPayload": {
"connection": {
"protocol": 6
},
"dest_gke_details": {
"cluster": {
"cluster_location": "europe-central2-a"
}
},
"dest_vpc": {
"vpc_name": "foo"
},
"reporter": "DEST"
},
"logName": "projects/hazel-aria-348413/logs/compute.googleapis.com%2Fvpc_flows",
"receiveTimestamp": "2022-06-03T12:09:52.418604934Z",
"resource": {
"labels": {
"subnetwork_id": "7449846049104218257",
"subnetwork_name": "foo"
},
"type": "gce_subnetwork"
}
},
"host": {
"name": "gke-cluster-3-default-pool-4e355575-k1w8"
},
"network": {
"bytes": 33792,
"iana_number": "6",
"name": "foo",
"packets": 32
},
"orchestrator": {
"cluster": {
"name": "cluster-3"
},
"type": "kubernetes"
},
"related": {
"ip": [
"10.0.0.3",
"34.118.64.229"
]
},
"server": {
"geo": {
"name": "europe-central2-a"
}
},
"source": {
"address": "34.118.64.229",
"as": {
"number": 15169
},
"geo": {
"continent_name": "Europe",
"country_iso_code": "POL"
},
"ip": "34.118.64.229",
"port": 443
}
}
Extracted Fields
The following table lists the fields that are extracted, normalized under the ECS format, analyzed and indexed by the parser. It should be noted that infered fields are not listed.
| Name | Type | Description |
|---|---|---|
@timestamp |
date |
Date/time when the event originated. |
cloud.availability_zone |
keyword |
Availability zone in which this host, resource, or service is located. |
cloud.project.id |
keyword |
The cloud project id. |
cloud.region |
keyword |
Region in which this host, resource, or service is located. |
destination.ip |
ip |
IP address of the destination. |
destination.port |
long |
Port of the destination. |
event.category |
keyword |
Event category. The second categorization field in the hierarchy. |
event.end |
date |
event.end contains the date when the event ended or when the activity was last observed. |
event.start |
date |
event.start contains the date when the event started or when the activity was first observed. |
event.type |
keyword |
Event type. The third categorization field in the hierarchy. |
google_vpc_flow_logs.insertId |
keyword |
|
google_vpc_flow_logs.jsonPayload.connection.protocol |
number |
|
google_vpc_flow_logs.jsonPayload.dest_gke_details.cluster.cluster_location |
keyword |
|
google_vpc_flow_logs.jsonPayload.dest_vpc.vpc_name |
keyword |
|
google_vpc_flow_logs.jsonPayload.reporter |
keyword |
|
google_vpc_flow_logs.logName |
keyword |
|
google_vpc_flow_logs.receiveTimestamp |
keyword |
|
google_vpc_flow_logs.resource.labels.subnetwork_id |
keyword |
|
google_vpc_flow_logs.resource.labels.subnetwork_name |
keyword |
|
google_vpc_flow_logs.resource.type |
keyword |
|
host.name |
keyword |
Name of the host. |
network.bytes |
long |
Total bytes transferred in both directions. |
network.iana_number |
keyword |
IANA Protocol Number. |
network.name |
keyword |
Name given by operators to sections of their network. |
network.packets |
long |
Total packets transferred in both directions. |
orchestrator.cluster.name |
keyword |
Name of the cluster. |
orchestrator.type |
keyword |
Orchestrator cluster type (e.g. kubernetes, nomad or cloudfoundry). |
server.geo.name |
keyword |
User-defined description of a location. |
source.as.number |
long |
Unique number allocated to the autonomous system. |
source.geo.continent_name |
keyword |
Name of the continent. |
source.geo.country_iso_code |
keyword |
Country ISO code. |
source.ip |
ip |
IP address of the source. |
source.port |
long |
Port of the source. |
For more information on the Intake Format, please find the code of the Parser, Smart Descriptions, and Supported Events here.