Google Chronicle Plugin for Log Shipper

This document explains how to configure the Chronicle plugin with the Log Shipper module of the Netskope Cloud Exchange platform. This plugin supports ingestion of alerts and events in UDM format.

Prerequisites

To complete this configuration, you need:

A Netskope tenant (or multiple, for example, production and development/test instances) that is already configured in Cloud Exchange.
A Netskope Cloud Exchange tenant with the Log Shipper module already configured.
A Chronicle account. Obtain your Chronicle Base URL and Service Account Key, and Customer ID from your Chronicle representative before proceeding.
Connectivity to the following hosts (one of these Regional URLs):
- USA: https://malachiteingestion-pa.googleapis.com/
- EU: https://europe-malachiteingestion-pa.googleapis.com/
- ASIA: https://asia-southeast1-malachiteingestion-pa.googleapis.com/
- Other Custom URLs you have.
Get the Chronicle service account Key. Reach out to the Chronicle team to get a service account with the following scopes: https://www.googleapis.com/auth/malachite-ingestion.

Note

Verify your Chronicle instance permissions are secure and not set up for open public access. Only allow access to your cloud storage instance from your Cloud Exchange Host and any other addresses that need access.

CE Version Compatibility

Netskope CE: v4.2.0, and v5.0.0

Google Chronicle Plugin Support

The Google Chronicle plugin is used to ingest all the Alert and Events in UDM format. Ingestion of WebTx data is not supported.

Alerts Support	Yes
Event Support	Yes
WebTx Support	No
CE Logs	No

API Details

List of APIs Used

API Endpoint	Method	Use Case
/v2/udmevents:batchCreate	POST	Ingest UDM events

Ingest UDM Events

API Endpoint: <Base URL>/v2/udmevents:batchCreate
Method: POST
Body:

{
  "customer_id": "c8c65bfa-5f2c-*********9-64bb7b939f2c",
  "events": [
    {
      "metadata": {
        "event_timestamp": "2019-10-22T12:00:00.000Z",
        "event_type": "USER_LOGIN",
        "product_name": "Acme SSO",
        "vendor_name": "Acme"
      },
      "principal": {
        "ip": [
          "10.1.2.3"
        ]
      },
      "target": {
        "application": "Acme Connect",
        "user": {
          "user_display_name": "Mary Jane",
          "userid": "mary@altostrat.com"
        }
      },
      "extensions": {
        "auth": {
          "type": "MACHINE",
          "mechanism": [
            "NETWORK"
          ]
        }
      }
    },
  ]
}

Sample API Response:

Status Code: 200 (Success)

This plugin uses Python libraries to authenticate with the Chronicle Ingestion API.

Library: Google Authentication library for Python (google-auth)
Usage: Google Authentication library for Python (google-auth) to authenticate to Google APIs.

Create a New Session with Credentials:

SCOPES = ['https://www.googleapis.com/auth/malachite-ingestion']

credentials = (
                service_account.Credentials.from_service_account_info(
                    json.loads(self.configuration["service_account_key"]),
                    scopes=SCOPES,
                )
            )
self.http_session = request.AuthorizedSession(credentials)

Chronicle API Request:

response = self.http_session.request(
                "POST",
                url,
                headers=headers,
                json=payload,)

Performance Matrix

This performance reading is conducted on a Large Stack CE with these VM specifications. The below readings are added with the consideration that it will ingest around 10K events in 11 seconds to the Google Chronicle platform.

Stack details	Size: Large RAM: 32 GB CPU: 16 Cores
Alerts/Events ingested to Google Chronicle SIEM	~200K EPM

User Agent

Netskope-ce-5.0.0-cls-google-chronicle-v2.1.0

Workflow

Configure the Chronicle Plugin.
Configure Log Shipper Business Rules for Chronicle.
Configure Log Shipper SIEM Mappings for Chronicle.
Validate the Chronicle plugin.

Click play to watch a video.

Configure the Google Chronicle Plugin

In Cloud Exchange, go to Settings > Plugins.
Search for and select the Chronicle v2.1.0 (CLS) box to open the plugin creation pages.
Enter a Configuration Name.
Select the Chronicle Default Mappings (Default Mappings for all plugins are available).
Make sure that the toggle button is enabled to ingest the data in after transformation in the UDM format. Ingesting data in Raw JSON is not supported. Click Next.
Enter the Configuration Parameters:
- Region: The region of the chronicle where the customer account is provisioned.
- Custom Region URL: Custom region base URL; required only if Custom Region is selected in Region.
- Service Account Key: Service Account Credentials (provided by Chronicle).
- Customer ID: Unique identifier, corresponding to the Chronicle instance.
Click Save. Your plugin will be available at Cloud Log Shipper > Plugins.

Configure a Log Shipper Business Rule for Google Chronicle

Go to Log Shipper > Business Rules.
By default, there is a business rule that filters all alerts and events. If you want to filter out any specific type of alert, or event, click Create New Rule and configure a new business rule by adding the rule name and filter.
Click Save.

Configure Log Shipper SIEM Mappings for Google Chronicle

Go Log Shipper > SIEM Mappings and click Add SIEM Mapping.
Select the Source plugin (Netskope CLS), Destination plugin (Chronicle Demo), and business rule. After the SIEM mapping is added, the data will start to be pulled from the Netskope tenant, transformed, and ingested into the Google Chronicle platform.
Click Save

Validate the Google Chronicle Plugin

You can validate the plugin in both Netskope CE and Google Chronicle.

Validate the Pull

To validate the pulling of Events and Alerts from the Netskope tenant, go to the Logging in Netskope CE. and search for the pulled logs.

Validate the Push

To validate the plugin workflow on Netskope CE, go to Logging and search for ingested Events, & Alerts with the filter “message contains ingested”. The ingested logs will be filtered.

To validate the push on the Google Chronicle platform, follow these steps:

Log in to the Google Chronicle Platform.
Enter a keyword for which you want to search.
Click Search.
Click Raw Log Search.
Set Start Time (UTC) and End Time (UTC) according to your needs. (You can also select both times by reference, like 1 minute, 2 hours, 1 day). Click Search.
Click on the icon to see details of the log.
You will be able to see the ingested data.

Troubleshooting the Google Chronicle Plugin

Unable to configure the Google Chronicle Plugin

If you are unable to configure the Google Chronicle Plugin, or get an authentication error in logs, it might be due to one of the reasons.

Provided incorrect Customer ID.
Provided incorrect Service Account Key.
Provided incorrect Service Account Key format.

What to do:

Make sure to provide the correct Customer ID, that is provided from your chronicle representative, according to the region selected.
Make sure to provide the correct Service Account Key according to the region selected.
Make sure to provide the correct format of Service Account Key

Getting Error in logs related to authentication after Configuring the Google Chronicle Plugin

If you are getting authentication errors in logs, it might be due to one of the reasons.

Provided incorrect Service Account Key.
Provided incorrect Service Account Key format.

What to do:

Make sure to provide the correct Service Account Key according to the region selected.
Make sure to provide the correct format of Service Account Key

Unable to ingest data on the Google Chronicle platform

If you are unable to ingest data on the Google Chronicle platform and getting this error, then check the below reason.
If you received an error message like displayed in the below image while ingesting data on the Google Chronicle 2.0.2 version, then go to What to do below.

What to do:
To resolve this error, make sure you have upgraded to the latest version of the Google Chronicle plugin, like 2.1.0.

Google Chronicle Plugin for Log Shipper