Skip to content

Google Kubernetes Engine

Overview

Google Cloud Logging centralizes logs from Google Cloud products.

In this documentation, you will learn how to collect and send Google Cloud logs to SEKOIA.IO.

Configure

Before you begin working with PubSub, verify that you have the right permission.

Follow Google's documentation to configure a dedicated PubSub receiver. At the end of the documentation you should have done the following:

  • Setup a project
  • Create a topic
  • Add a subscription
  • Try your setup by publishing a message to the topic

Next, create a dedicated service account. At the end of the documentation you should have done the following:

  • Create a service account with the role Pub/Sub Subscriber
  • Create and download JSON keys (service account credentials)

You should now have:

  • A credentials file
  • A project ID
  • A subscription ID

To pull events, go to the playbook page and create your playbook with a template: "Create a new playbook" > "Use a template" > Search for Google Cloud.

You can also create your own on the same basis. A typical playbook to retrieve and send Google Cloud logs to SEKOIA.IO will use this kind of chain:

  • A "Google Pub/Sub" trigger
  • An action that sends events to SEKOIA.IO

To start sending Logs to SEKOIA.IO, please create a Logs Router Sinks with an Inclusion Filter that fits your needs (Read the documentation dedicated to the product you want to monitor).

Event Categories

The following table lists the data source offered by this integration.

Data Source Description
Application logs Google Kubernetes Engine logs contains events from applications deployed in Google Cloud containers.

In details, the following table denotes the type of events produced by this integration.

Name Values
Kind event
Category process
Type change

Event Samples

Find below few samples of events and how they are normalized by SEKOIA.IO.

{
    "event": {
        "kind": "event",
        "category": "process",
        "type": [
            "change"
        ],
        "reason": "ImageCreate event &ImageCreate{Name:gke.gcr.io/k8s-dns-sidecar:1.21.4-gke.0,Labels:map[string]string{io.cri-containerd.image: managed,},XXX_unrecognized:[],}",
        "outcome": "success"
    },
    "@timestamp": "2022-06-01T13:50:44.190941Z",
    "orchestrator": {
        "type": "kubernetes",
        "cluster": {
            "name": "cluster-1"
        },
        "resource": {
            "type": "k8s_node"
        }
    },
    "google_kubernetes_engine": {
        "insertId": "rxi2u29itqwvaa18",
        "logName": "projects/hazel-aria-348413/logs/container-runtime",
        "receiveTimestamp": "2022-06-01T13:50:46.223587016Z",
        "jsonPayload": {
            "MESSAGE": "time=\"2022-06-01T13:50:44.190869134Z\" level=info msg=\"ImageCreate event &ImageCreate{Name:gke.gcr.io/k8s-dns-sidecar:1.21.4-gke.0,Labels:map[string]string{io.cri-containerd.image: managed,},XXX_unrecognized:[],}\"",
            "SYSLOG_IDENTIFIER": "containerd",
            "_BOOT_ID": "e61a95dc40fd44f6ba5c6bfcb18b46a2",
            "_CAP_EFFECTIVE": "1ffffffffff",
            "_COMM": "containerd",
            "_GID": 0,
            "_STREAM_ID": "949cd6779ed34897a1b74883881ddfe8",
            "_SYSTEMD_CGROUP": "/system.slice/containerd.service",
            "_SYSTEMD_INVOCATION_ID": "ebd8a874b9bf4797a358a0403ec7e1e7",
            "_SYSTEMD_SLICE": "system.slice",
            "_SYSTEMD_UNIT": "containerd.service",
            "_TRANSPORT": "stdout",
            "_UID": "0"
        }
    },
    "log": {
        "syslog": {
            "priority": 6,
            "facility": {
                "code": 3
            }
        }
    },
    "process": {
        "command_line": "/usr/bin/containerd",
        "executable": "/usr/bin/containerd",
        "pid": 1478
    },
    "host": {
        "name": "gke-cluster-1-default-pool-476246ab-wnl7",
        "id": "3fa273bf9f602a2286f55eac7ffa6d36"
    },
    "server": {
        "geo": {
            "name": "europe-west1-c"
        }
    },
    "cloud": {
        "project": {
            "id": "hazel-aria-348413"
        }
    }
}
{
    "event": {
        "kind": "event",
        "category": "process",
        "type": [
            "change"
        ],
        "reason": "StopContainer for \\\"4c2b21624d4488ea8305bec91bb58135e840ab50b779da3db19ddf87864a760e\\\" with timeout 30 (s)",
        "outcome": "success"
    },
    "@timestamp": "2022-06-01T14:01:35.371492Z",
    "orchestrator": {
        "type": "kubernetes",
        "cluster": {
            "name": "cluster-1"
        },
        "resource": {
            "type": "k8s_node"
        }
    },
    "google_kubernetes_engine": {
        "insertId": "mf28fmdkt05bbyjk",
        "logName": "projects/hazel-aria-348413/logs/container-runtime",
        "receiveTimestamp": "2022-06-01T14:01:36.219094561Z",
        "jsonPayload": {
            "MESSAGE": "time=\"2022-06-01T14:01:35.371006269Z\" level=info msg=\"StopContainer for \\\"4c2b21624d4488ea8305bec91bb58135e840ab50b779da3db19ddf87864a760e\\\" with timeout 30 (s)\"",
            "SYSLOG_IDENTIFIER": "containerd",
            "_BOOT_ID": "e61a95dc40fd44f6ba5c6bfcb18b46a2",
            "_CAP_EFFECTIVE": "1ffffffffff",
            "_COMM": "containerd",
            "_GID": 0,
            "_STREAM_ID": "949cd6779ed34897a1b74883881ddfe8",
            "_SYSTEMD_CGROUP": "/system.slice/containerd.service",
            "_SYSTEMD_INVOCATION_ID": "ebd8a874b9bf4797a358a0403ec7e1e7",
            "_SYSTEMD_SLICE": "system.slice",
            "_SYSTEMD_UNIT": "containerd.service",
            "_TRANSPORT": "stdout",
            "_UID": "0"
        }
    },
    "log": {
        "syslog": {
            "priority": 6,
            "facility": {
                "code": 3
            }
        }
    },
    "process": {
        "command_line": "/usr/bin/containerd",
        "executable": "/usr/bin/containerd",
        "pid": 1478
    },
    "host": {
        "name": "gke-cluster-1-default-pool-476246ab-wnl7",
        "id": "3fa273bf9f602a2286f55eac7ffa6d36"
    },
    "server": {
        "geo": {
            "name": "europe-west1-c"
        }
    },
    "cloud": {
        "project": {
            "id": "hazel-aria-348413"
        }
    }
}
{
    "event": {
        "kind": "event",
        "category": "process",
        "type": [
            "change"
        ],
        "reason": "Pulling image \"gke.gcr.io/prometheus-to-sd:v0.11.3-gke.0\"",
        "outcome": "success"
    },
    "@timestamp": "2022-06-01T14:05:30Z",
    "orchestrator": {
        "type": "kubernetes",
        "api_version": "v1",
        "namespace": "kube-system",
        "cluster": {
            "name": "cluster-1"
        },
        "resource": {
            "name": "kube-dns-56494768b7-544n6",
            "type": "k8s_pod"
        }
    },
    "google_kubernetes_engine": {
        "insertId": "17ahw8eg29q74y6",
        "logName": "projects/hazel-aria-348413/logs/events",
        "receiveTimestamp": "2022-06-01T14:05:39.683992581Z",
        "severity": "INFO",
        "jsonPayload": {
            "apiVersion": "v1",
            "kind": "Event",
            "reason": "Pulling",
            "type": "Normal",
            "involvedObject": {
                "fieldPath": "spec.containers{prometheus-to-sd}",
                "kind": "Pod",
                "name": "kube-dns-56494768b7-544n6",
                "resourceVersion": "6551",
                "uid": "52017f74-5157-4788-a62e-b83c4eac4acf"
            },
            "metadata": {
                "creationTimestamp": "2022-06-01T14:05:30Z",
                "managedFields": [
                    {
                        "manager": "kubelet",
                        "apiVersion": "v1",
                        "fieldsV1": {
                            "f:message": {},
                            "f:involvedObject": {},
                            "f:lastTimestamp": {},
                            "f:source": {
                                "f:host": {},
                                "f:component": {}
                            },
                            "f:type": {},
                            "f:reason": {},
                            "f:count": {},
                            "f:firstTimestamp": {}
                        },
                        "operation": "Update",
                        "fieldsType": "FieldsV1",
                        "time": "2022-06-01T14:05:30Z"
                    }
                ],
                "resourceVersion": "954",
                "uid": "658b3d26-ed26-4d32-a5b4-3bb87bdefa99"
            },
            "source": {
                "component": "kubelet",
                "host": "gke-cluster-1-default-pool-476246ab-wnl7"
            }
        }
    },
    "host": {
        "name": "kube-dns-56494768b7-544n6.16f48435f72a4bd9"
    },
    "server": {
        "geo": {
            "name": "europe-west1-c"
        }
    },
    "cloud": {
        "project": {
            "id": "hazel-aria-348413"
        }
    }
}
{
    "event": {
        "kind": "event",
        "category": "process",
        "type": [
            "change"
        ],
        "reason": "Failed to update endpoint kube-system/kube-dns: Operation cannot be fulfilled on endpoints \"kube-dns\": the object has been modified; please apply your changes to the latest version and try again",
        "outcome": "success"
    },
    "@timestamp": "2022-06-01T14:05:33Z",
    "orchestrator": {
        "type": "kubernetes",
        "api_version": "v1",
        "namespace": "kube-system",
        "cluster": {
            "name": "cluster-1"
        },
        "resource": {
            "type": "k8s_cluster"
        }
    },
    "google_kubernetes_engine": {
        "insertId": "17ahw8eg29q74yc",
        "logName": "projects/hazel-aria-348413/logs/events",
        "receiveTimestamp": "2022-06-01T14:05:39.683992581Z",
        "severity": "WARNING",
        "jsonPayload": {
            "apiVersion": "v1",
            "kind": "Event",
            "reason": "FailedToUpdateEndpoint",
            "type": "Warning",
            "involvedObject": {
                "kind": "Endpoints",
                "name": "kube-dns",
                "resourceVersion": "7416",
                "uid": "75cc3b54-2a5f-42fa-8dd9-1669695113cd"
            },
            "metadata": {
                "creationTimestamp": "2022-06-01T14:05:33Z",
                "managedFields": [
                    {
                        "apiVersion": "v1",
                        "operation": "Update",
                        "fieldsV1": {
                            "f:firstTimestamp": {},
                            "f:involvedObject": {},
                            "f:reason": {},
                            "f:count": {},
                            "f:lastTimestamp": {},
                            "f:type": {},
                            "f:message": {},
                            "f:source": {
                                "f:component": {}
                            }
                        },
                        "manager": "kube-controller-manager",
                        "time": "2022-06-01T14:05:33Z",
                        "fieldsType": "FieldsV1"
                    }
                ],
                "resourceVersion": "960",
                "uid": "828b8cd3-1eec-4093-95fb-907ebeab0efa"
            },
            "source": {
                "component": "endpoint-controller"
            }
        }
    },
    "host": {
        "name": "kube-dns.16f484369d214dae"
    },
    "server": {
        "geo": {
            "name": "europe-west1-c"
        }
    },
    "cloud": {
        "project": {
            "id": "hazel-aria-348413"
        }
    }
}
{
    "event": {
        "kind": "event",
        "category": "process",
        "type": [
            "change"
        ],
        "reason": "Created container prometheus-to-sd",
        "outcome": "success"
    },
    "@timestamp": "2022-06-01T14:05:32Z",
    "orchestrator": {
        "type": "kubernetes",
        "api_version": "v1",
        "namespace": "kube-system",
        "cluster": {
            "name": "cluster-1"
        },
        "resource": {
            "name": "kube-dns-56494768b7-544n6",
            "type": "k8s_pod"
        }
    },
    "google_kubernetes_engine": {
        "insertId": "17ahw8eg29q74yb",
        "logName": "projects/hazel-aria-348413/logs/events",
        "receiveTimestamp": "2022-06-01T14:05:39.683992581Z",
        "severity": "INFO",
        "jsonPayload": {
            "apiVersion": "v1",
            "kind": "Event",
            "reason": "Created",
            "type": "Normal",
            "involvedObject": {
                "fieldPath": "spec.containers{prometheus-to-sd}",
                "kind": "Pod",
                "name": "kube-dns-56494768b7-544n6",
                "resourceVersion": "6551",
                "uid": "52017f74-5157-4788-a62e-b83c4eac4acf"
            },
            "metadata": {
                "creationTimestamp": "2022-06-01T14:05:32Z",
                "managedFields": [
                    {
                        "time": "2022-06-01T14:05:32Z",
                        "manager": "kubelet",
                        "fieldsType": "FieldsV1",
                        "operation": "Update",
                        "apiVersion": "v1",
                        "fieldsV1": {
                            "f:count": {},
                            "f:type": {},
                            "f:involvedObject": {},
                            "f:source": {
                                "f:component": {},
                                "f:host": {}
                            },
                            "f:reason": {},
                            "f:firstTimestamp": {},
                            "f:message": {},
                            "f:lastTimestamp": {}
                        }
                    }
                ],
                "resourceVersion": "959",
                "uid": "2836bb34-8703-4475-a7d8-5cf0ec2232f8"
            },
            "source": {
                "component": "kubelet",
                "host": "gke-cluster-1-default-pool-476246ab-wnl7"
            }
        }
    },
    "host": {
        "name": "kube-dns-56494768b7-544n6.16f48436899e3f4a"
    },
    "server": {
        "geo": {
            "name": "europe-west1-c"
        }
    },
    "cloud": {
        "project": {
            "id": "hazel-aria-348413"
        }
    }
}

Extracted Fields

The following table lists the fields that are extracted, normalized under the ECS format, analyzed and indexed by the parser. It should be noted that infered fields are not listed.

Name Type Description
@timestamp date Date/time when the event originated.
cloud.project.id keyword The cloud project id.
event.category keyword Event category. The second categorization field in the hierarchy.
event.kind keyword The kind of the event. The highest categorization field in the hierarchy.
event.reason keyword Reason why this event happened, according to the source
event.type keyword Event type. The third categorization field in the hierarchy.
google_kubernetes_engine.insertId keyword None
google_kubernetes_engine.jsonPayload.MESSAGE keyword None
google_kubernetes_engine.jsonPayload.SYSLOG_IDENTIFIER keyword None
google_kubernetes_engine.jsonPayload._BOOT_ID keyword None
google_kubernetes_engine.jsonPayload._CAP_EFFECTIVE keyword None
google_kubernetes_engine.jsonPayload._COMM keyword None
google_kubernetes_engine.jsonPayload._GID number None
google_kubernetes_engine.jsonPayload._STREAM_ID keyword None
google_kubernetes_engine.jsonPayload._SYSTEMD_CGROUP keyword None
google_kubernetes_engine.jsonPayload._SYSTEMD_INVOCATION_ID keyword None
google_kubernetes_engine.jsonPayload._SYSTEMD_SLICE keyword None
google_kubernetes_engine.jsonPayload._SYSTEMD_UNIT keyword None
google_kubernetes_engine.jsonPayload._TRANSPORT keyword None
google_kubernetes_engine.jsonPayload._UID keyword None
google_kubernetes_engine.jsonPayload.apiVersion keyword None
google_kubernetes_engine.jsonPayload.eventTime keyword None
google_kubernetes_engine.jsonPayload.involvedObject.fieldPath keyword None
google_kubernetes_engine.jsonPayload.involvedObject.kind keyword None
google_kubernetes_engine.jsonPayload.involvedObject.name keyword None
google_kubernetes_engine.jsonPayload.involvedObject.resourceVersion keyword None
google_kubernetes_engine.jsonPayload.involvedObject.uid keyword None
google_kubernetes_engine.jsonPayload.kind keyword None
google_kubernetes_engine.jsonPayload.metadata.creationTimestamp keyword None
google_kubernetes_engine.jsonPayload.metadata.managedFields object None
google_kubernetes_engine.jsonPayload.metadata.resourceVersion keyword None
google_kubernetes_engine.jsonPayload.metadata.uid keyword None
google_kubernetes_engine.jsonPayload.reason keyword None
google_kubernetes_engine.jsonPayload.reportingComponent keyword None
google_kubernetes_engine.jsonPayload.reportingInstance keyword None
google_kubernetes_engine.jsonPayload.source.component keyword None
google_kubernetes_engine.jsonPayload.source.host keyword None
google_kubernetes_engine.jsonPayload.type keyword None
google_kubernetes_engine.logName keyword None
google_kubernetes_engine.receiveTimestamp keyword None
google_kubernetes_engine.severity keyword None
host.id keyword Unique host id.
host.name keyword Name of the host.
log.syslog.facility.code long Syslog numeric facility of the event.
log.syslog.priority long Syslog priority of the event.
orchestrator.api_version keyword API version being used to carry out the action
orchestrator.cluster.name keyword Name of the cluster.
orchestrator.namespace keyword Namespace in which the action is taking place.
orchestrator.resource.name keyword Name of the resource being acted upon.
orchestrator.resource.type keyword Type of resource being acted upon.
orchestrator.type keyword Orchestrator cluster type (e.g. kubernetes, nomad or cloudfoundry).
process.command_line wildcard Full command line that started the process.
process.executable keyword Absolute path to the process executable.
process.pid long Process id.
server.geo.name keyword User-defined description of a location.

Google Kubernetes Engine (GKE) configuration

There are different types of logs produced by GKE:

Auditd logs: Most important logs from a security point of view. We recommend that you use Auditbeat to collect Auditd logs.

Flow Logs: From Google VPC FLow Logs documentation:

VPC Flow Logs records a sample of network flows sent from and received by VM instances, including instances used as GKE nodes. These logs can be used for network monitoring, forensics, real-time security analysis, and expense optimization.

Please read the dedicated documentation. (Intake type: Google VPC Flow Logs)

Activity logs (Intake type: Google Cloud Audit log):

  • operations done on GCP cluster resources (create a cluster, etc.) resource.type: gke_cluster
  • operations done on Kubernetes objects “inside” clusters (Kubernetes Audit Policy) resource.type: k8s_cluster

Event logs: Kubernetes events are objects that provide insight into what is happening inside a cluster, such as what decisions were made by the scheduler or why some pods were evicted from the node log_id("events") (Intake type: Google Cloud Audit log)

Node logs: Services running on GKE nodes (kubelet, node problem detector, container runtime, etc.) emit their own logs resource.type: k8s_node

Container logs: Container stdout and stderr streams log_id("stdout") log_id("stderr")

Instance logs: Serial console output log_id("serialconsole.googleapis.com/serial_port_1_output")

To learn more, consult the following resources:

Back to top