Google Chronicle Plugin for Log Shipper

This document explains how to configure the Chronicle v3.0.0 plugin with the Log Shipper module of the Netskope Cloud Exchange platform. This plugin is used to deliver alerts (DLP, Malware, Policy, Compromised Credential, Malsite, Quarantine, Remediation, Security Assessment, Watchlist, UBA, CTEP, Anomaly, Legal Hold) and events (Page, Application, Audit, Infrastructure, Network, Incident) data to Google Chronicle platform. The plugin supports sharing of UDM and JSON formatted data. The required API keys are linked to customers and are provided by your Google Chronicle representative.

Prerequisites

To complete this configuration, you need:

Google Chronicle Plugin Support

The Google Chronicle plugin is used to ingest all the Alert and Events in UDM and JSON format. Ingestion of WebTx and Syslog CE Log data is not supported.

{
  "customer_id": "c8c65bfa-5f2c-*********9-64bb7b939f2c",
  "events": [
    {
      "metadata": {
        "event_timestamp": "2019-10-22T12:00:00.000Z",
        "event_type": "USER_LOGIN",
        "product_name": "Acme SSO",
        "vendor_name": "Acme"
      },
      "principal": {
        "ip": [
          "10.1.2.3"
        ]
      },
      "target": {
        "application": "Acme Connect",
        "user": {
          "user_display_name": "Mary Jane",
          "userid": "mary@altostrat.com"
        }
      },
      "extensions": {
        "auth": {
          "type": "MACHINE",
          "mechanism": [
            "NETWORK"
          ]
        }
      }
    },
  ]
}

Status Code: 200 (Success)

{
  "customer_id": "c8c65bfa-5f2c-42d4-9189-64bb7b939f2c",
  "log_type": "BIND_DNS",
  "labels" : [
    {
      "key" : "key_name_one",
      "value" : "value_one"
    },
    {
      "key" : "key_name_two",
      "value" : "value_two"
    }
  ]
  "entries": [
    {
      "log_text": "26-Feb-2019 13:35:02.187 client 10.120.20.32#4238: query: altostrat.com IN A + (203.0.113.102)",
      "ts_epoch_microseconds": 1551188102187000
    },
    {
      "log_text": "26-Feb-2019 13:37:04.523 client 10.50.100.33#1116: query: examplepetstore.com IN A + (203.0.113.102)",
    },
    {
      "log_text": "26-Feb-2019 13:39:01.115 client 10.1.2.3#3333: query: www.example.com IN A + (203.0.113.102)"
    }
  ];
}

Status Code: 200 (Success)

SCOPES = ['https://www.googleapis.com/auth/malachite-ingestion']

credentials = (
                service_account.Credentials.from_service_account_info(
                    json.loads(self.configuration["service_account_key"]),
                    scopes=SCOPES,
                )
            )
self.http_session = request.AuthorizedSession(credentials)

response = self.http_session.request(
                "POST",
                url,
                headers=headers,
                json=payload,)

Performance Matrix

This performance reading is conducted on a Large Stack with these VM specifications. These readings factor that it will ingest around 10K events in 13 seconds to the Google Chronicle platform.

User Agent

netskope-ce-6.0.0-cls-google-chronicle-v3.0.0

Workflow

Watch a Video

Click play to watch a video.

Configure the Google Chronicle Plugin

1. In Cloud Exchange, go to Settings > Plugin Store. Search for and select the Google Chronicle v3.0.0 (CLS) plugin box.

   

2. Add a plugin configuration name and make sure you have the Chronicle Default Mappings file selected. Make sure to keep Format as UDM to ingest the data in the UDM format, and change it to JSON to ingest data in JSON format.

   

3. Click Next and enter the Configuration Parameters:
   • Region: The Chronicle region where the customer account is provisioned.
   • Custom Region URL: Custom region base URL; required only if Custom Region is selected for Region.
   • Service Account Key: Service Account Credentials (provided by your Chronicle team).
   • Customer ID: Unique identifier, corresponding to your Chronicle instance.
   • Log Source Identifier: This will be added as a namespace to all the JSON formatted data. The log source identifier should not contain whitespaces.

   

4. Click Save. Your new plugin configuration will be available on the Log Shipper > Plugins page.

   

Configure a Log Shipper Business Rule for Google Chronicle

1. In Log Shipper, go to the Business Rules. 
2. By default, there is a business rule that filters all alerts and events. If you want to filter out any specific type of alert or event, click Create New Rule and configure a new business rule by adding a rule name and desired filter(s).

   

Configure Log Shipper Log Delivery for Google Chronicle

1. In Log Shipper, go to Log Delivery and click Add Log Delivery Configuration. 
2. Select the Source plugin (Netskope CLS), Destination plugin (Chronicle Demo), a business rule, and click Save.
3. After the Log Delivery is added, the data will start to be pulled from the Netskope tenant, transformed, and ingested into the Google Chronicle platform.

   

Validate the Google Chronicle Plugin

You can validate the plugin in both Cloud Exchange and Google Chronicle.

Validate the Pull

To validate the pulling of Events and Alerts from the Netskope tenant, go to Logging in Cloud Exchange and search for the pulled logs.

Validate the Push

To validate the plugin workflow in Cloud Exchange, go to Logging and search for ingested Events and Alerts with the filter message contains ingested. The ingested logs will be shown.

To validate the push in Google Chronicle:

1. Log in to Google Chronicle.

   

2. Go to Investigate > SIEM Search.

   

3. Set Start Time (UTC) and End Time (UTC) according to your needs. And add a query to search ingested alerts/events. Click Search.

   Note

   Use metadata.log_types = “NETSKOPE_ALERT_V2” for filtering data ingested in JSON format and use metadata.log_types = “UDM” for filtering data ingested in UDM format.

   Sample ingested Events in JSON format:

   

   Sample ingested Events in UDM format:

   

4. Click Event Viewer to see details in the log.

   

   Note

   Make sure you select the correct timeframe while filtering ingested data.

5. You will be able to see the ingested data.

   Sample ingested Events/Alerts in UDM format:

   

   

   

   

   

   

   

   

   

   

   Sample ingested Alerts in JSON format:

   

   

Troubleshooting the Google Chronicle Plugin

Unable to configure the Google Chronicle Plugin

If you are unable to configure the Google Chronicle Plugin, or get an authentication error in logs, it may be due to one of these reasons:

• Provided incorrect Customer ID.
• Provided incorrect Service Account Key.
• Provided incorrect Service Account Key format.

What to do:

1. Make sure to provide the correct Customer ID that is provided from your Chronicle representative, according to the region selected.
2. Make sure to provide the correct Service Account Key according to the region selected.
3. Make sure to provide the correct format of Service Account Key.

Getting an Error in logs related to authentication after Configuring the Google Chronicle Plugin

If you are getting authentication errors in logs, it may be due to one of these reasons:

• Provided incorrect Service Account Key.
• Provided incorrect Service Account Key format.

What to do:

1. Make sure to provide the correct Service Account Key according to the region selected.
2. Make sure to provide the correct format of Service Account Key.

Unable to filter the ingested Alerts/Events in Chronicle

If you are unable to filter the ingested Alerts/Events on Chronicle platform, then check if a wrong filter or wrong time range is being used.

What to do:

The ingested alerts or events may not be immediately visible on the Chronicle UI, as ingestion and processing can take some time. So, wait for some time and then refer to the Validate the Push section to filter the alerts/events. It is necessary to use the correct time range based on ingestion time while filtering the events or alerts.

Unable to ingest data in Google Chronicle

If you are unable to ingest data on the Google Chronicle platform and getting this error, then check the below reason.
If you received an error message like displayed in the below image while ingesting data on the Google Chronicle 2.0.2 version, then go to What to do below.

What to do:
To resolve this error, make sure you have upgraded to the latest version of the Google Chronicle plugin, like 2.2.0.

Format parameter shows CEF instead of UDM after upgrading the plugin to v3.0.0

If you are using Cloud Exchange v6.0.0 and upgrade the plugin to the latest version (v3.0.0), you can check the Format parameter by editing the plugin.

What to do:

Edit the plugin and select the Format as UDM, and then save the plugin.

Known Behaviors

• Alert ingested from Cloud Exchange will be classified as Event on Google Chronicle. You can see the same in the below screenshot:

  

• This plugin only supports ingestion of Alerts/Events listed under Plugin Scope section. 
  Ingestion of these Netskope Alerts and Events are not supported by this plugin:
  Alert Types: Device, Content
  Event Types: Endpoint, Client Status
• The ingested alerts or events may not be immediately visible on the Chronicle UI, as ingestion and processing can take some time. It was observed that if we ingest data in large numbers, then it may take a few hours to reflect on the Chronicle UI.