Rapid7 Plugin for Log Shipper - Netskope Technical Documentation

This document explains how to configure your Rapid7 v3.1.1 plugin with the Log Shipper module of the Netskope Cloud Exchange platform. This plugin supports ingestion of Alerts (Anomaly, Legal Hold, Compromised Credential, Policy, Malsite, Malware, DLP, Security Assessment, Quarantine, Remediation, UBA, Watchlist, CTEP), Events (Page, Application, Audit, Infrastructure, Network, Incident, Endpoint), Web Transaction data, and CE logs (Debug, Information, Error, Warning) to Rapid7 in JSON and CEF format.

Prerequisites

To complete this configuration, you need:

A Netskope tenant (or multiple, for example, production and development/test instances).
A Netskope Cloud Exchange tenant with the Tenant plugin and Log Shipper plugin already configured.
A Netskope Cloud Exchange tenant with the AWS Netskope Log Streaming or Azure Netskope Log Streaming plugin already configured (for pulling WebTx from the Netskope Log Streaming plugins)
A Netskope Cloud Exchange tenant with the Syslog for CE plugin already configured (for pulling CE logs).
A Rapid7 server.
Connectivity to the following host: https://insight.rapid7.com/.

Rapid7 Plugin Support

The Rapid7 plugin is used to ingest all the Alert, Events, WebTx (via Netskope LogStreaming), and CE Logs in JSON and CEF format to the specified Rapid7 server.

Data Type	Support
Events	Yes (Audit, Application, Infrastructure, Network, Incident, Page, Endpoint) The ingestion of the Endpoint event type is supported starting with Cloud Exchange version 5.1.0.
Alerts	Yes (Anomaly, Legal Hold, DLP, Malware, Policy, Compromised Credential, Malsite, Quarantine, Remediation, Security Assessment, Watchlist, CTEP, UBA)
WebTx Logs	Yes (via Netskope Log Streaming)
Syslog CE Logs	Yes (Info, Debug, Warning, Error)

Note

CLS WebTX based on Google Pub Sub Lite is deprecated. Please refer to Netskope Product EOL/EOS Announcements – Netskope Knowledge Portal
For ingesting WebTX logs to your Log delivery destinations like SIEM, SOAR, XDR, Data Lake, use the AWS Netskope Log Streaming or Azure Netskope Log Streaming plugin.
The ingestion of Endpoint event type is supported from Cloud Exchange version 5.1.0.

API Details

The plugin uses a logging 3rd-party library to push the data to the Syslog collector.

Library: logging

This module defines functions and classes which implement a flexible event-logging system for applications and libraries.

The key benefit of having the logging API provided by a standard library module is that all Python modules can participate in logging, so your application log can include your own messages integrated with messages from third-party modules.

Refer to the official documentation for more information on the logging library:https://docs.python.org/3/library/logging.html.

List of Methods Used

Method: logging.getLogger(name=None)

Return a logger with the specified name or, if the name is None, return a logger which is the root logger of the hierarchy

All calls to this function with a given name return the same logger instance. This means that logger instances never need to be passed between different parts of an application.

Method: setLevel(level)

Sets the threshold for this logger to level. Logging messages that are less severe than the level will be ignored; logging messages that have a severity level or higher will be emitted by whichever handler or handlers service this logger, unless a handler’s level has been set to a higher severity level than the level.

Method: handlers

The list of handlers is directly attached to this logger instance.

Note that this attribute should be treated as read-only; it is normally changed via the addHandler() and removeHandler() methods, which use locks to ensure thread-safe operation.

Method: addHandler(hdlr): Adds the specified handler hdlr to this logger.
Method: removeHandler(hdlr): Removes the specified handler hdlr from this logger.

Performance Matrix

Description	Specification
Stack details	Size: Large RAM: 32 GB CPU: 16 Cores
Alerts/Events ingested to SIEM	~200K EPM

Workflow

Create a collector and event source on Rapid7.
Configure the Rapid7 plugin.
Configure a Log Shipper Business Rule for Rapid7.
Configure Log Shipper Log Delivery for Rapid7.
Validate the Rapid7 plugin.

Click play to watch a video.

Get your Rapid7 Server and Port Information

Go to your Rapid7 instance at https://insight.rapid7.com.
Log in to Rapid7.
Click Data Collection, Collectors and then click Download Collector. Download the Collector for your OS.
Install the Collector on your machine. (Installation Steps: Collector Installation and Deployment | InsightIDR Documentation)
Click Data Collection, Event Sources, and then click Add Event Source.
Search for and select the Rapid7 Custom Logs box.
Name the Event Source and select the Collector you have activated. Enter a Port Number and select a Protocol.
Click Save.
Click Data Collection Management, and then Event Sources to see the configured Event Source.
Copy the server IP and port number. You will need these to configure the Rapid7 plugin.

Configure the Rapid7 Plugin

In Cloud Exchange, go to Settings > Plugin Store.
Search for and select the Rapid7 v3.1.1 (CLS) plugin.
Enter a Configuration Name and make sure you have the Rapid7 Default Mapping file selected. If you need custom mapping, click Create New Mapping. For JSON format, disable the logs will be transformed toggle.
Click Next.
Select and enter these parameters:
- Rapid7 Server: IP address/FQDN of the Rapid7 server where data will be ingested.Rapid7 Protocol: Protocol to be used while ingesting data.Rapid7 Port: Configured Event Source port on Rapid7.Rapid7 Certificate: The certificate is required only for the TLS protocol.Log Source Identifier: The prefix to be added for the logs.
Click Save. This plugin configuration will be available on the Log Shipper > Plugins page.

Configure Log Shipper Business Rules for Rapid7

Go to Log Shipper > Business Rules.
By default, there is a business rule that filters all alerts and events. If you want to filter out any specific type of alert or event, click Create New Rule and configure a new business rule by adding the rule name and filter.
Enter a Rule Name and select the filters to use. Enter a Folder Name if any.
Click Save.

Configure Log Shipper SIEM Mappings for Rapid7

Go to Log Shipper > Log Delivery and click Add Log Delivery Configuration.
For alerts and events, select the Source plugin (CLS Netskope), Destination plugin (CLS Rapid7), and a business rule.
For WebTx, select the Source plugin (AWS Netskope Log Streaming or Azure Netskope Log Streaming), and Destination plugin (CLS Rapid7).
For Logs sharing, select the Source plugin (CLS Syslog for CE), and Destination plugin (CLS Rapid7).
Click Save.

After the Log Delivery configuation is added, the data will start to be pulled from the Netskope tenant, transformed, and ingested into the Rapid7 platform.

Validate the Rapid7 Plugin

Validate the Pull

To validate the pulling of Events, Alerts, logs, and Webtx from the Netskope tenant, go to Logging and search for the pulled logs.

Validate the Push

To validate on Cloud Exchange, go to Logging and search for ingested Events, Alerts, WebTx, and Logs with the filter message contains ingested. The ingested logs will be filtered. To validate the push on the Rapid7, log in to the Rapid7 platform and click Log Search. Search the logs by filtering from time range. Here are screenshots of data ingested in CEF format. Here are screenshots of data ingested in JSON format.