Syslog Plugin with Splunk for Log Shipper
Syslog Plugin with Splunk for Log Shipper
This document explains how to configure the Syslog plugin with the Cloud Log Shipper module of the Netskope Cloud Exchange platform. This plugin supports ingestion of Alerts (Compromised Credential, Policy, Malsite, Malware, DLP, Security Assessment, Quarantine, Remediation, UBA, Watchlist, CTEP), Events (Page, Application, Audit, Infrastructure, Network, Incident), Web Transaction data, and CE logs (Debug, Information, Error, Warning) to Syslog in CEF and JSON format.
Prerequisites
To complete this configuration, you need:
- A Netskope tenant (or multiple, for example, production and development/test instances) that is already configured in Cloud Exchange.
- A Netskope Cloud Exchange tenant with the Log Shipper module already configured.
- A Netskope Cloud Exchange tenant with the Log Shipper WebTx plugin already configured.
- A Splunk instance.
- Connectivity to a syslog server.
CE Version Compatibility
Netskope CE v4.2.0 and v5.0.1
Sizing Recommendations
Refer to System requirements for different configurations. It is recommended to use the system with the medium specification if your data volume ranges to 100k EPM.
Syslog Plugin Support
Syslog plugin is used to ingest all the Alert, Events, CE Logs, WebTx and CE Logs in CEF and JSON format.
| Event Support | Yes (Audit, Application, Infrastructure, Network, Incident, Page) |
| Alert Support | Yes (DLP, Malware, Policy, Compromised Credential, Malsite, Quarantine, Remediation, Security Assessment, Watchlist, CTEP, UBA) |
| WebTx Support | Yes |
| CE Logs Support | Yes (Info, Error, Warning, Debug) |
All Netskope events, alert logs, and web transaction logs will be shared.
CE Version Compatibility
CE version: v4.2.0, v5.0.1.
API Details
API Details
The plugin uses a ‘logging’ third-party library to push the data to the Syslog collector.
Library: logging
This module defines functions and classes which implement a flexible event-logging system for applications and libraries.
The key benefit of having the logging API provided by a standard library module is that all Python modules can participate in logging, so your application log can include your own messages integrated with messages from third-party modules.
Refer to the official documentation for more information on the logging library: https://docs.python.org/3/library/logging.html.
List of Methods Used
Method: logging.getLogger(name=None)
Return a logger with the specified name or, if the name is None, return a logger which is the root logger of the hierarchy
All calls to this function with a given name return the same logger instance. This means that logger instances never need to be passed between different parts of an application.
Method: setLevel(level)
Sets the threshold for this logger to level. Logging messages that are less severe than the level will be ignored; logging messages that have a severity level or higher will be emitted by whichever handler or handlers service this logger, unless a handler’s level has been set to a higher severity level than the level.
Method: handlers
The list of handlers is directly attached to this logger instance.
Note: This attribute should be treated as read-only; it is normally changed via the addHandler() and removeHandler() methods, which use locks to ensure thread-safe operation.
- Method: addHandler(hdlr)
- Adds the specified handler hdlr to this logger.
- Method: removeHandler(hdlr)
- Removes the specified handler hdlr from this logger.
Performance Matrix
This performance reading is conducted on a Large Stack CE with the below-mentioned VM specifications. The below readings are added with the consideration that it will ingest around 10K events in 2 seconds to the Syslog platform.
| Stack details | Size: Large
RAM: 32 GB CPU: 16 Cores |
| Alerts/Events ingested to SIEM server | ~200K EPM |
| WebTx ingested to SIEM server | ~6 MBps |
Workflow
- Add Data Input on Splunk.
- Configure the Syslog Plugin for the Splunk integration.
- Configure a Log Shipper Business Rule for the Splunk integration.
- Configure Log Shipper SIEM Mappings for the Splunk integration.
- Validate the Splunk integration.
Click play to watch a video.
If you do not have a Splunk instance, follow these steps to install Splunk.
- Log in to your Splunk instance.
- From the dashboard, go to Settings > Data inputs.
- Click Add new for the TCP input.
- Add your port and click Next.
- Select the source type if you already have any, or click New to create a new source type.
- Enter the source type. Select the Source Type Category based on your requirements, or keep it as is.
- Scroll down to Index. If you already have any index that you want to use, select it from the Index dropdown. Otherwise, click Create a new index, add an Index Name, click Save, and then click Review.
- Review the details and click Submit.
- Click Start searching.
- Go to Settings > Plugins. Search for and select the Syslog CLS plugin box.
- Add a Plugin configuration name, select the Syslog Default Mapping file, and then click Next.
- Disable the first toggle button if you want to ingest your alerts and events in the JSON format. Keep it as enabled if you want to ingest your data into CEF format.
- Click Next.
- Enter these parameters:
- Syslog server: IP address/FQDN of Syslog server in which data will be ingested.
- Syslog Format: Data format required to ingest data.
- Syslog Protocol: Protocol to be used while ingesting data.
- Syslog Port: The port used while creating the Data input configuration on Splunk.
- Click Save. Your plugin will be available on the Log Shipper > Plugins page.
Go to Log Shipper > Business Rule. The default business rule filters all alerts and events. If you need to filter out any specific type of alert or event, click Create New Rule and configure a new business rule by adding a rule name and specific filters.
 |
- Go to Log Shipper > SIEM Mappings and click Add SIEM Mapping.
- Select the Source plugin (Netskope CLS plugin), Destination plugin (Syslog plugin), and a business rule, and then click Save.
- For WebTx, click Add SIEM Mapping, select the Source plugin (Netskope WebTx plugin), Destination plugin (Syslog plugin), and a business rule, and then click Save.
- For Logs sharing, click Add SIEM Mapping, select the Source plugin (CLS Syslog for CE), the Destination plugin (CLS Syslog), and a business rule, and click Save.
After the SIEM mapping is added, the data will start to be pulled from the Netskope tenant, transformed, and ingested into the Syslog platform.
Validate the Splunk Integration
Validate the Pull
To validate the pulling of Events, Alerts, logs, and WebTx from the Netskope tenant:
- In Cloud Exchange, go to the Logging and search for the pulled logs.
Validate the Push
To validate the plugin in Cloud Exchange:
- Go to Logging and search for ingested Events, Alerts, WebTx & Logs with the filter “message contains ingested”. The ingested logs will be filtered.
To validate the push in Splunk:
- Log in to the Splunk Platform.
- Click Search & Reporting.
- Enter the source and Protocol along with: and port along with Log Source Identifier (Example: source=”tcp:1111″ netskope.
- This is how logs look from the plugin to Splunk.
- This is how WebTx data looks from the plugin to Splunk.
- This is how data looks when shared in JSON format from the plugin to Splunk (unparsed format).
Troubleshooting
An error occurred while configuring the Syslog Plugin
Despite entering all parameters and clicking the Save, an error may occur, possibly due to the following reasons:
- The server/port configuration may differ from the specified settings (Netskope CE/Splunk)
- The port is not exposed on the Splunk server.
Follow these steps to fix these issues.
- In the Splunk Platform, go to Settings and click Data inputs >TCP (Whichever configuration you have used). Check that both are the same.
- Expose the Port on the Splunk server.
An error occurred while ingesting data from CE to Syslog
If you are unable to push alerts/events/logs/webtx data on the Syslog platform, then it could be due to one of these reasons:
- Port is deleted/disabled on the Syslog platform.
- Splunk server storage is full.
To solve these issues:
- Make sure the port is present and enabled; if not, then create a new port.
- Make sure to clean the event data if not necessary, or increase the storage of the Splunk server.
If ingested data is not reflected on the Syslog Platform
If you are unable to view alerts/events/logs/webtx data on the Syslog platform, it could be due to one of these reasons:
- The filter is not correct on the Splunk platform.
- There might be any error but UDP is selected in the Port while configuring the syslog plugin, Hence logs of ingested are visible.
To solve these issues:
- Make sure Data is searched using the correct filter.
- Make sure to select the TCP port to check if there is any issue.
