Log Shipper Custom Plugin Developers Guide

This document explains how to create a new Cloud Log Shipper plugin that allows to transform and push or ingest data by leveraging the functionality provided within the CLS module. The developer should be able to write a new plugin independently without any technical issues by following this guide.

Prerequisites

Python 3.x programming experience (intermediate level).
Access to the Netskope CE platform.
API or Python SDK access to the product or solution for which you need to write the plugin.
An account with minimum permission for the product.

Log Shipper Module

The Cloud Exchange platform, and its Cloud Log Shipper module, comes with a rich set of features and functionality that allow for a high degree of customization, so we recommend that you familiarize yourself with the different aspects of the platform as listed below.

Note: This module supports sharing of data from Netskope to 3rd-party tools.

Netskope Concepts & Terminology

Core: CE core engine manages the 3rd party plugins and their life cycle methods. It has API endpoints for interacting with the platform to perform various tasks.
Module: The functional code areas that invoke modular-specific plugins to accomplish different workflows. CLS is one of the cloud exchange modules running in Cloud Exchange.
Plugin: Plugins are Python packages that have logic to transform and push or ingest information/logs collected fron the Netskope tenant to 3rd party
Plugin Configurations: Plugin configurations are the plugin class objects which are configured with the required parameters and are scheduled by the CE core engine for transforming and pushing or ingesting information.
Mapping: The format of data you want to send to the platform, which is called mapping

Development Guidelines

Use the Package Directory Structure for all Python code.
Make sure all the 3rd party libraries packaged with the plugin package are checked for known vulnerabilities.
Make sure to follow standard Python code conventions. (https://peps.python.org/pep-0008/)
Run and verify that the flake8 lint check passes with the doc string check enabled.The maximum length of a line should be 80.
Convert the time-stamp values to the human-readable format (from epoch to DateTime object). Make sure the time that is being displayed on the UI should be in the local timezone.
If possible, add a default value while adding a configuration parameter in the plugin.
For Scripts/Integrations written in Python, make sure to create unit tests. Please refer to the below section Unit Testing.
Plugin architecture allows storing states, however, avoids storing huge objects for state management.
If the Plugin description contains a link, it should be a hyperlink redirecting to the documentation page. Refer CTE Trend Micro Vision One plugin for details.
The logger messages and the Toast messages should not contain the API Token and the Password type field values.
Pagination should always be considered while developing any feature in a plugin.
Make sure to add a retry mechanism for the 429 status code.
Use proper validation for the parameters passed to the validate method and provide the proper help text for all the parameters.
Use a notifier object to raise the notification for failures or critical situations (like rate-limiting, or exceeding payload size) to notify the status of the plugin to the user.
Make sure to implement a proper logging mechanism with the logger object passed by the CE platform. Make sure enough logging is done which helps the Ops team in troubleshooting. Make sure any sensitive data is not logged or leaked in the notification.
Provide the proper help text (tooltip) for all the parameters. If feasible, make sure to explain the significance of the parameter in the tooltip.
Make sure to provide a meaningful name and description to plugin configuration parameters.
Make sure to provide an appropriate configuration type (text, number, password, choice, multi-choice) to the parameters.
Make sure to use the proxy configuration dict and the SSL certificate validation flag that is passed by the CE platform while making any outbound request (API/SDK).
Make sure to collect the value of a non-mandatory parameter using the .get() method and provide a default value while using .get() method.
Make sure the plugin directory name (e.g sample_plugin) matches with the manifest.json’s id field
User Agent should be added to the headers while making any API call. Format for the User Agent: netskope-ce-–––. Refer to URE Microsoft Azure AD or URE Crowdstrike Identity Protect plugin for more details. Note: The plugin version should be dynamically fetched and to fetch netskope-ce- string use the method defined by core.
API Tokens and Password fields should not use strip().
The log messages should start with “ Plugin [configuration_name]: “. Example: “CLS Chronicle Plugin [Chronicle Configuration Name]: “. [This is a suggestion, we can avoid configuration name]. (logger.info(“ Plugin: ”))
While logging an error log, if possible, we should add a traceback of the exception. USE: self.logger.error(error, details=traceback.format_exc())
The Toast message should not contain the Plugin: in the message.
Make sure to catch proper exceptions and status codes while and after making the API calls. If feasible, Devs can create a helper method from where the requests would be made and this method can be called with proper parameters wherever required.
The CHANGELOG.md file should be updated with proper tags such as Added, Changed, and Fixed along with a proper user-friendly message. Make sure the file name should exactly match CHANGELOG.md
Check your python code for any vulnerabilities.

Writing a Plugin

This section illustrates the process of writing a plugin from scratch.

Download the sample plugin from the NetskopeOSS public Github repository.

Development setup

Python

Our system utilizes Python3 (v3.7 and above). Make sure to set up python3 in your development environment. Pytest is used to run unit tests.

Included Python libraries

The following Python libraries are included within the Netskope CE platform.

Library Name	Version
aiofiles	22.1.0
amqp	5.1.1
anyio	3.6.2
asgiref	3.6.0
attrs	22.2.0
azure-core	1.26.2
azure-storage-blob	12.14.1
bcrypt	4.0.1
boto3	1.26.51
botocore	1.29.51
billiard	3.6.4.0
celery	5.2.7
cabby	0.1.23
cachetools	5.2.1
celerybeat-mongo	0.2.0
certifi	2022.12.7
cffi	1.15.1
chardet	5.1.0
charset-normalizer	3.0.1
click	8.1.3
click-didyoumean	0.3.0
click-plugins	1.1.1
click-repl	0.2.0
colorama	0.4.6
colorlog	6.7.0
cryptography	39.0.0
cybox	2.1.0.21
defusedxml	0.7.1
dnspython	2.3.0
docker	6.0.1
fastapi	0.89.1
furl	2.1.3
google-api-core	2.11.0
google-auth	2.16.0
google-cloud-core	2.3.2
google-cloud-pubsub	2.13.12
google-cloud-pubsublite	1.6.0
google-cloud-storage	2.7.0
google-crc32c	1.5.0
google-resumable-media	2.4.0
googleapis-common-protos	1.58.0
grpc-google-iam-v1	0.12.6
grpcio	1.51.1
grpcio-status	1.51.1
gunicorn	20.1.0
h11	0.14.0
idna	3.4
importlib-metadata	6.0.0
isodate	0.6.1
jmespath	1.0.1
jsonpath	0.82
jsonschema	4.17.3
kombu	5.2.4
libcst	0.3.21
libtaxii	1.1.119
lxml	4.9.2
mongoengine	0.25.0
more-itertools	9.0.0
MarkupSafe	2.1.2
memory-profiler	0.61.0
mixbox	1.0.5
mongoquery	1.4.2
msrest	0.7.1
multidict	6.0.4
mypy-extensions	0.4.3
netskopesdk	0.0.25
numpy	1.23.5
oauthlib	3.2.2
onelogin	3.1.0
ordered-set	4.1.0
orderedmultidict	1.0.1
overrides	6.5.0
pandas	1.5.0
packaging	23.0
passlib	1.7.4
pycparser	2.21
prompt-toolkit	3.0.36
proto-plus	1.22.2
protobuf	4.21.12
psutil	5.9.4
pydantic	1.10.4
pyasn1	0.4.8
pyasn1-modules	0.2.8
PyJWT	2.6.0
pymongo	4.3.3
pyparsing	3.0.9
python-dateutil	2.8.2
pyrsistent	0.19.3
python-multipart	0.0.5
python3-saml	1.15.0
pytz	2022.7.1
PyYAML	6.0
requests	2.28.2
requests-oauthlib	1.3.1
rsa	4.9
six	1.16.0
starlette	0.22.0
sniffio	1.3.0
s3transfer	0.6.0
stix	1.2.0.11
taxii2-client	2.3.0
typing-inspect	0.8.0
typing-utils	0.1.0
typing_extensions	4.4.0
urllib3	1.26.14
uvicorn	0.20.0
vine	5.0.0
wcwidth	0.2.6
weakrefmethod	1.0.3
websocket-client	1.4.2
Werkzeug	2.2.2
xmlsec	1.3.11
zipp	3.11.0
requests-mock	1.7.0

Including custom plugin libraries

Netskope advises bundling any of the third party python libraries your plugin will need into the plugin package itself. Use the pip installer to achieve this bundling; it provides a switch which takes a directory as an input. If the directory is provided, pip will install the packages into that directory.

For example, the command shown below will install the “cowsay” package into the directory “lib”.

> pip install cowsay --target ./lib

For the official documentation on this, refer https://pip.pypa.io/en/stable/reference/pip_install/#cmdoption-t.

While importing modules from above lib folder, we should be using relative import instead of absolute import. As shown below

from .lib import cowsay

IDE

Recommended IDEs are PyCharm or Visual Studio Code.

Plugin Directory Structure

This section mentions the typical directory structure of a Log Shipper plugin.

/sample_plugin/
  ├── /utils/
	├── syslog_cef_generator.py
	├── syslog_constants.py
	├── syslog_exceptions.py
	├── syslog_helper.py
	├── syslog_ssl.py
	├── syslog_validator.py
  ├── __init__.py
  ├── CHANGELOG.md
  ├── icon.png
  ├── main.py
  ├── mappings.json
  └── manifest.json

__init__.py: Every plugin package is considered a python module by the CE code. Make sure every plugin package contains the empty “__init__.py” file.
icon.png: Plugin icon logo, this will be visible in the plugin chiclet and configuration cards on the UI. The logo should have a transparent background with recommended size of 300*50 pixels or a similar aspect ratio.
CHANGELOG.md: This file contains the details about plugin update and should be updated with proper tags as Added, Changed, Fixed along with a proper user-friendly message.
main.py: This python file contains the Plugin class containing the concrete implementation for the transform, push and validate method.
manifest.json: Manifest file for the plugin package containing information about all the configurable parameters and their data types. This file has more information about the plugin integration as well.
Mappings.json: This file contains the Default mapping file that we need to use for the plugin. If not specified, it will use the default mappings specified in the manifest.json file and Cloud Exchange will throw an error, given that no such mapping file is found.
utils/: This directory is used to write utility functions and it contains different files as per plugin requirements. Here the syslog plugin specific files are shown.
- syslog_cef_generator.py: The generator file contains the logic to convert netskope json data to a 3rd-party supported data type based on the mapping file selected for the plugin.
- syslog_constants.py: This file contains the constants that can be used plugin wide like BASE_URL, Severity maps, etc.
- syslog_exceptions.py: This file contains the custom exceptions that can be raised by the plugin like Mapping validation exception, Data Format specific exceptions
- syslog_helper.py: This file contains the helper functions that can be used to validate mapping schema or to get mapping file based on data_type
- syslog_ssl.py: This file contains the logic to create connection with 3rd party platform.
- syslog_validator.py: This file contains validation for configuration parameters as well some mapping or connection related validation.

The listed files except the “utils” folder and “mappings.json” are mandatory for any plugin integration, but developers can add other files based on specific integration requirements.

Note: Make sure the plugin directory name (e.g sample_plugin) matches with the manifest.json’s id field

CHANGELOG.md

This is a file that contains details about plugin update and should be updated with proper tags as Added, Changed, Fixed along with a proper user-friendly message.

Added: use it, when new features are added
Fixed: use it, when any bug/error fixed.
Changed: use it, when there is any change in existing implementation of plugin

Sample Changelog.md

# 1.0.1
## Fixed
- Fixed pagination when there are more than 10k Logs


# 1.0.0
## Added
- Initial release.

Mappings.json

This is a JSON file that contains the mapping file of the plugin, which is then read by the CLS module while transforming Netskope data to the plugin supported data type.

Common parameters for mappings.json include:
- name: (string) Name of the mapping file. (Required)
- jsonData: (string) field mapping of all fields as Json escaped string. (Required)
- isDefault: (boolean) To specify whether this mapping file is default or not. Users can not delete/edit the default mappings. (Default value: False) (Optional)

You should always create a mapping file for the Log Shipper plugins. The mapping should contain the proper type and subtype that the plugin supports, even if we don’t do transformation, the plugin should define the subtypes. For example:
```
{
     "taxonomy": {
          "json": {
               "events": {
                     "application": [],
                     "page": []
               }
          }
      }
}
```

Sample mappings.json file.

{
    "name": "Default Mappings",
    "jsonData": "{\"delimiter\":\"|\",\"syslog_map_version\":\"2.0.3\",\"cef_version\":\"0\",\"validator\":\"valid_extensions.csv\",\"taxonomy\":{\"logs\":{\"info\":{\"header\":{\"Target_Field\":{\"default_value\":\"Netskope\",\"mapping_field\":\"Netskope_Field\",\"transformation\":\"String\"}},\"extension\":{\"Target_Field\":{\"default_value\":\"default_value\",\"mapping_field\":\"Netskope_Field\",\"transformation\":\"String\"}}},\"warning\":{\"header\":{\"Target_Field\":{\"default_value\":\"Netskope\",\"mapping_field\":\"Netskope_Field\",\"transformation\":\"String\"}},\"extension\":{\"Target_Field\":{\"default_value\":\"default_value\",\"mapping_field\":\"Netskope_Field\",\"transformation\":\"String\"}}},\"error\":{\"header\":{\"Target_Field\":{\"default_value\":\"Netskope\",\"mapping_field\":\"Netskope_Field\",\"transformation\":\"String\"}},\"extension\":{\"Target_Field\":{\"default_value\":\"default_value\",\"mapping_field\":\"Netskope_Field\",\"transformation\":\"String\"}}}},\"webtx\":{\"v2\":{\"header\":{\"Target_Field\":{\"default_value\":\"Netskope\",\"mapping_field\":\"Netskope_Field\",\"transformation\":\"String\"}},\"extension\":{\"Target_Field\":{\"default_value\":\"default_value\",\"mapping_field\":\"Netskope_Field\",\"transformation\":\"String\"}}}},\"alerts\":{\"anomaly\":{\"header\":{\"Target_Field\":{\"default_value\":\"Netskope\",\"mapping_field\":\"Netskope_Field\",\"transformation\":\"String\"}},\"extension\":{\"Target_Field\":{\"default_value\":\"default_value\",\"mapping_field\":\"Netskope_Field\",\"transformation\":\"String\"}}},\"dlp\":{\"header\":{\"Target_Field\":{\"default_value\":\"Netskope\",\"mapping_field\":\"Netskope_Field\",\"transformation\":\"String\"}},\"extension\":{\"Target_Field\":{\"default_value\":\"default_value\",\"mapping_field\":\"Netskope_Field\",\"transformation\":\"String\"}}},\"malware\":{\"header\":{\"Target_Field\":{\"default_value\":\"Netskope\",\"mapping_field\":\"Netskope_Field\",\"transformation\":\"String\"}},\"extension\":{\"Target_Field\":{\"default_value\":\"default_value\",\"mapping_field\":\"Netskope_Field\",\"transformation\":\"String\"}}},\"policy\":{\"header\":{\"Target_Field\":{\"default_value\":\"Netskope\",\"mapping_field\":\"Netskope_Field\",\"transformation\":\"String\"}},\"extension\":{\"Target_Field\":{\"default_value\":\"default_value\",\"mapping_field\":\"Netskope_Field\",\"transformation\":\"String\"}}},\"CompromisedCredential\":{\"header\":{\"Target_Field\":{\"default_value\":\"Netskope\",\"mapping_field\":\"Netskope_Field\",\"transformation\":\"String\"}},\"extension\":{\"Target_Field\":{\"default_value\":\"default_value\",\"mapping_field\":\"Netskope_Field\",\"transformation\":\"String\"}}},\"LegalHold\":{\"header\":{\"Target_Field\":{\"default_value\":\"Netskope\",\"mapping_field\":\"Netskope_Field\",\"transformation\":\"String\"}},\"extension\":{\"Target_Field\":{\"default_value\":\"default_value\",\"mapping_field\":\"Netskope_Field\",\"transformation\":\"String\"}}},\"Malsite\":{\"header\":{\"Target_Field\":{\"default_value\":\"Netskope\",\"mapping_field\":\"Netskope_Field\",\"transformation\":\"String\"}},\"extension\":{\"Target_Field\":{\"default_value\":\"default_value\",\"mapping_field\":\"Netskope_Field\",\"transformation\":\"String\"}}},\"Quarantine\":{\"header\":{\"Target_Field\":{\"default_value\":\"Netskope\",\"mapping_field\":\"Netskope_Field\",\"transformation\":\"String\"}},\"extension\":{\"Target_Field\":{\"default_value\":\"default_value\",\"mapping_field\":\"Netskope_Field\",\"transformation\":\"String\"}}},\"Remediation\":{\"header\":{\"Target_Field\":{\"default_value\":\"Netskope\",\"mapping_field\":\"Netskope_Field\",\"transformation\":\"String\"}},\"extension\":{\"Target_Field\":{\"default_value\":\"default_value\",\"mapping_field\":\"Netskope_Field\",\"transformation\":\"String\"}}},\"SecurityAssessment\":{\"header\":{\"Target_Field\":{\"default_value\":\"Netskope\",\"mapping_field\":\"Netskope_Field\",\"transformation\":\"String\"}},\"extension\":{\"Target_Field\":{\"default_value\":\"default_value\",\"mapping_field\":\"Netskope_Field\",\"transformation\":\"String\"}}},\"Watchlist\":{\"header\":{\"Target_Field\":{\"default_value\":\"Netskope\",\"mapping_field\":\"Netskope_Field\",\"transformation\":\"String\"}},\"extension\":{\"Target_Field\":{\"default_value\":\"default_value\",\"mapping_field\":\"Netskope_Field\",\"transformation\":\"String\"}}},\"uba\":{\"header\":{\"Target_Field\":{\"default_value\":\"Netskope\",\"mapping_field\":\"Netskope_Field\",\"transformation\":\"String\"}},\"extension\":{\"Target_Field\":{\"default_value\":\"default_value\",\"mapping_field\":\"Netskope_Field\",\"transformation\":\"String\"}}}},\"events\":{\"infrastructure\":{\"header\":{\"Target_Field\":{\"default_value\":\"Netskope\",\"mapping_field\":\"Netskope_Field\",\"transformation\":\"String\"}},\"extension\":{\"Target_Field\":{\"default_value\":\"default_value\",\"mapping_field\":\"Netskope_Field\",\"transformation\":\"String\"}}},\"page\":{\"header\":{\"Target_Field\":{\"default_value\":\"Netskope\",\"mapping_field\":\"Netskope_Field\",\"transformation\":\"String\"}},\"extension\":{\"Target_Field\":{\"default_value\":\"default_value\",\"mapping_field\":\"Netskope_Field\",\"transformation\":\"String\"}}},\"application\":{\"header\":{\"Target_Field\":{\"default_value\":\"Netskope\",\"mapping_field\":\"Netskope_Field\",\"transformation\":\"String\"}},\"extension\":{\"Target_Field\":{\"default_value\":\"default_value\",\"mapping_field\":\"Netskope_Field\",\"transformation\":\"String\"}}},\"audit\":{\"header\":{\"Target_Field\":{\"default_value\":\"Netskope\",\"mapping_field\":\"Netskope_Field\",\"transformation\":\"String\"}},\"extension\":{\"Target_Field\":{\"default_value\":\"default_value\",\"mapping_field\":\"Netskope_Field\",\"transformation\":\"String\"}}},\"network\":{\"header\":{\"Target_Field\":{\"default_value\":\"Netskope\",\"mapping_field\":\"Netskope_Field\",\"transformation\":\"String\"}},\"extension\":{\"Target_Field\":{\"default_value\":\"default_value\",\"mapping_field\":\"Netskope_Field\",\"transformation\":\"String\"}}}},\"json\":{\"alerts\":{\"anomaly\":[],\"dlp\":[],\"malware\":[],\"policy\":[],\"CompromisedCredential\":[],\"LegalHold\":[],\"Malsite\":[],\"Quarantine\":[],\"Remediation\":[],\"SecurityAssessment\":[],\"Watchlist\":[],\"uba\":[]},\"events\":{\"application\":[],\"audit\":[],\"infrastructure\":[],\"page\":[],\"network\":[]}}}}"

Manifest.json

This is a JSON file that stores the meta-information related to the plugin, which is then read by the CLS module to render the plugin in the UI as well as enabling the CLS module to know more about the plugin, including required configuration parameters, the plugin-id, the plugin-name, etc.

Every plugin must contain this file with the required information so that CLS can instantiate the Plugin object properly.
Common parameters for manifest.json include:
- name: (string) Name of the plugin. (Required)
- id: (string) Id of the plugin package. Make sure it is unique across all the plugins installed in the Cloud Exchange. The ID has to match the directory name of the plugin package. (Required)
- version: (string) Version of the plugin. Usage of a MAJOR.MINOR.PATCH (ex. 1.0.1) versioning scheme is encouraged although there are no restrictions. (Required)
- types: (array) Types of data type supported by the plugin. Possible data types are “alerts”, “events”, “webtx” and “logs”.
- description: (string) Description of the plugin. Provide a detailed description which mentions the capabilities and instructions to use the plugin. (Ex. – This plugin is used to ingest data to the SIEM platform). The description would appear on the Plugin Configuration card. (Required). If the description contains a link, it should be a hyperlink instead of plaintext.
- Mapping: The type of data format one wants to send to. (Required) (this parameter should be identical with the ‘name’ parameter in the mappings file of the plugin.)
- configuration: (array) Array of JSON objects that contains information about all the parameters required by the plugin – their name, type, id, etc. The common parameters for the nested JSON objects are explained below.
  - label: Name of the parameter. This will be displayed on the plugin configuration page. (Required)
  - key: Unique parameter key, which will be used as a key in the python dict object where the plugin configuration is used. (Required)
  - type: Value type of the parameter. Allowed values are ‘text’, ‘password’, ‘number’, ‘choice’ and ‘multichoice’. (Required) Refer to Plugin Configuration parameter types section below for more details.
  - default: The default value for this parameter. This value will appear in the plugin configuration page on CLS UI. Supported data-types are “text”, “number”, and “list” (for multichoice type). (Required)
  - mandatory: Boolean which indicates whether this parameter is mandatory or not. If a parameter is mandatory CLS UI won’t let you pass an empty value for the parameter. Allowed values are `true` and `false`. (Required)
  - description: Help text level description for the parameter which can give more details about the parameter and expected value. This string will appear in the plugin configuration page as a help-text. (Required)
  - choices: A list of JSON objects containing key and value as JSON keys. This parameter is only supported by 'type': 'choice and multichoice`.

Plugin Configuration Parameter types

Make sure all the required plugin configuration parameters are listed under the configuration section of manifest.json for the plugin.

Password Parameter

Use this parameter for storing any secrets/passwords for authentication with API endpoints. Parameters with type as the password will have a password text box in the Plugin configuration page and will be obfuscated and encrypted by the platform.

Sample JSON

      "configuration": [
           {
               "label": "API Token",
               "key": "api_token",
               "type": "password"
           },
       ]

Plugin configuration view:

Text Parameter

Use this parameter for storing any string information such as base-url, username, etc. This parameter will have a normal text input on the plugin configuration page.

Sample JSON

       "configuration": [
           {
               "label": "Tenant Name",
               "key": "tenant_name",
               "type": "text"
           },
       ]

Plugin configuration view:

Number Parameter

Use this parameter for storing number/float values. This parameter will have a number input box on the plugin configuration page. (Note: This is from Cloud Threat Exchange plugin development guide)

Sample JSON

      "configuration": [
           {
               "label": "Maximum File hash list size in MB.",
               "key": "max_size",
               "type": "number"
           },
       ]

Plugin configuration view:

Choice Parameter

Use this parameter for storing any enumeration parameter values. This parameter will have a dropdown box on the plugin configuration page.

Sample JSON

{
            "label": "Syslog Protocol",
            "key": "syslog_protocol",
            "type": "choice",
            "choices": [
                {
                    "key": "TLS",
                    "value": "TLS"
                },
                {
                    "key": "UDP",
                    "value": "UDP"
                },
                {
                    "key": "TCP",
                    "value": "TCP"
                }
            ],
            "default": "UDP",
            "mandatory": true,
            "description": "Protocol to be used while ingesting data."
        },

Plugin configuration view:

After selecting the input:

Multichoice Parameter

Use this parameter for storing multiple choice values. This parameter will have a dropdown box on the plugin configuration page with ability to select multiple values. (Note: This is from Cloud Threat Exchange plugin development guide.)

While using ‘multichoice’ parameter in the manifest file, the ‘mandatory’ parameter should be kept as False.

Sample JSON

   "configuration": [
        {
            "label": "Severity",
            "key": "severity",
            "type": "multichoice",
            "choices": [
                {
                    "key": "Unknown",
                    "value": "unknown"
                },
                {
                    "key": "Low",
                    "value": "low"
                },
                {
                    "key": "Medium",
                    "value": "medium"
                },
                {
                    "key": "High",
                    "value": "high"
                },
                {
                    "key": "Critical",
                    "value": "critical"
                }
            ],
            "default": [
                "critical",
                "high",
                "medium",
                "low",
                "unknown"
            ],
            "mandatory": false,
            "description": "Only indicators with matching severity will be saved."
        }
    ]

Plugin Configuration View:

Toggle Parameter

This parameter stores a boolean value, toggle enabled is True and toggle disabled is False

Transform the raw logs (‘transformData’):

If enabled, Raw logs will be transformed using selected mapping file, else raw logs will be sent to SIEM. The ingestion may be affected if the SIEM does not accept raw logs format.(Default: True)

Use System Proxy (‘proxy’):

Use system proxy configured in Settings.(Default: False)

Plugin Configuration View:

Note: This parameter is provided by Core, and it is not allowed to add from plugins manifest.json file

main.py

This python file contains the core implementation of the plugin.

Standard Imports

from netskope.integrations.cls.plugin_base import ( 
PluginBase, 
ValidationResult, 
PushResult,
)

PluginBase Variables

PluginBase provides access to variables which can be used during the plugin lifecycle methods. Below is the list of variables.

Variable Name	Usage	Description
Self.name,	self.logger.error(“Message”) self.logger.warn(“Message”) self.logger.info(“Message”)	Logger handles are provided by core. Use this object to log important events. The logs would be visible in CE Audit logs. Go to Logging for more information.
self.configuration	self.configuration. get()	JSON representation of the configuration object of Plugin instance. Use this to access the configuration attributes like authentication credentials, server details, etc. Use the key name of the attribute mentioned in manifest.json.
self.last_run_at	If self.last_run_at:self.last_run_at.timestamp() Use this format to convert the last run time in epoch format.	Datetime value which contains the latest time when this plugin successfully executed.
self.storage		Log Shipper provides the plugin a mechanism to maintain state. Use this object to persist any state that would be required during subsequent calls. The datatype of this object is python dict.
self.notifier	self.notifier.info(“message”) self.notifier.warn(“message”) self.notifier.error(“message”)	This object provides handle of CE core’s notifier. Use this object to push any notification to the platform. The notifications would be visible on CLS UI. Make sure the message contains summarized information for the user to read and take necessary actions. For example: use notifier in Netskope plugin if the push() method exceeds 8MB limit of the product.
self.use_proxy	requests.get(url=url, proxies=self.proxy)	Handle of system’s proxy settings if configured, else {}.
self.ssl_validation	requests.get(url=url, verify=self.ssl_validation)	Boolean value which mentions if ssl validation be enforced for REST API calls.
self.source	helper.get_tenant_cls(self.source)	String value which contains name of source plugin from which data will be pulled.
self.mappings	get_syslog_mappings( self.mappings, data_type)	Json string which contains mapping for the plugin.

Plugin Class

Plugin class has to be inherited from the PluginBase class. PluginBase class is defined in netskope.integrations.cls.plugin_base.
Make sure Plugin class provides implementation for the push and transform. You may implement pull, validate, mapping, or source method, depending on your use case.
Pagination should always be considered while developing any feature in a plugin.
The plugin class will contain all the necessary params to establish connection and authentication with the 3rd party API.
Constants like PLUGIN_NAME, LIMIT etc. should be declared.

"""Sample plugin implementation.


This is a sample implementation of base PluginBase class. Which explains the concrete implementation of the base class.
"""


from netskope.common.utils import AlertsHelper
from netskope.integrations.cls.plugin_base import (
    PluginBase,
    ValidationResult,
    PushResult,
)
PLUGIN_NAME = "<module> <plugin_name> Plugin"


class SamplePlugin(PluginBase):
   """SamplePlugin class having concrete implementation for transforming and pushing logs.


   This class is responsible for implementing transform, push and validate methods with proper return types,
   so that it's lifecycle execution can be scheduled by the CLS core engine.
   """

Def Transform()

This is an abstract method of PluginBase Class.

This method implements the logic to transform the logs to be sent to another platform.
You can create helper and generator functions to transform the netskope events/alerts.
In this method, ‘transformData’ variable for ‘Transform the raw logs’ toggle should be handled properly as mentioned in Toggle parameters section.

   def transform(self, raw_data, data_type, subtype) -> List:
        """To Transform the raw netskope JSON data into target platform supported data formats."""
        try:
            # Get the mapping file content
        except Exception as err:
            self.logger.error(f”{PLUGIN_NAME}: An error occurred while getting mapping. Error: {str(err)}"
      )
            raise


        # Initialize generator/helper objects if required


        transformed_data = []
        for data in raw_data:
		# Map headers and extensions as per mapping file
		# append transformed data in the transformed_data list
        return transformed_data

Def Push()

This is an abstract method of PluginBase Class.

This method implements the logic to push the logs transformed by the CE platform to the product API endpoints or Socket connections.
It receives all the data that the transformed data and then it pushes. You can use any kind of strategy to push data. Be it socket connection or REST API.
This method will be invoked when the CE platform receives transformed data to it.
Make sure to handle the case when the maximum payload size supported by the API endpoint is exceeded. There can be multiple ways to handle this case.
- If the API endpoint supports multiple requests with a fixed payload size, send the data in chunks.
- If the API endpoint does not support multiple requests (i.e. we can push it in one API call only) either plugin can skip the remaining logs and raise a notification to the user to adjust the sharing filters, or it can fail with the error of exceeded payload size.
Handle all the Exceptions with connection and HTTP response code and raise exceptions with error message along with the loggers in case of errors are encountered.
User Agent should be added to the headers while making any API call. Format for the User Agent: netskope-ce-–––.
In this method, ‘proxy’ variable for ‘Use System Proxy’’ toggle should be used while making API call.
Return the PushResult object (Refer PushResult Data Model) with a success flag indicating whether the Push operation was successful or not.

   def push(self, transformed_data, data_type, subtype) -> PushResult:
""Push the logs to the 3rd party systems.
       This method will be invoked while pushing the logs with 3rd party systems.It takes in transformed_data, data_type and subtype as arguments while pushing this data via sockets to the 3rd party platform.
       """
        try:
            # API Calls or Socket write operations to push the data
        except Exception as err:
            self.logger.error(f”{PLUGIN_NAME}: Error occurred during pushing    data. Error: {str(err)}"
            )
            raise

Def Validate()

This is an abstract method of PluginBase Class.

This method validates the plugin configuration and authentication parameters passed while creating a plugin configuration.
This method will be called only when a new configuration is created or updated.
Separate validations should be made for empty field validation and type check validation in the validation method for plugin configuration.
Validates against all the mandatory parameters are passed with the proper data type.
Validate the authentication parameters and the API endpoint to ensure the smooth execution of the plugin lifecycle.
While validating, use strip() for configuration parameters like base URL, email, username etc. except API Tokens and Password fields.
Return the object of ValidationResult(Refer ValidationResult Data Model) with a success flag indicating validation success or failure and the validation message containing validation failure reason.

 	   def validate(self, data):      
          """Validate the Plugin configuration parameters.


       Validation for all the parameters mentioned in the manifest.json for the existence and
       data type. Method returns the netskope.integrations.cls.plugin_base.ValidationResult object with success = True in the case
       of successful validation and success = False and a error message in the case of failure.
       Args:
           data (dict): Dict object having all the Plugin configuration parameters.
       Returns:
           netskope.integrations.cls
.plugin_base.ValidateResult: ValidateResult object with success flag and message.
       """
       self.logger.info(f"{PLUGIN_NAME}: Executing validate method for Sample plugin")
       if (
           "secret_field_id1" not in data
           or not data["secret_field_id1"]
           or type(data["secret_field_id1"]) != str
       ):
           self.logger.error(

Data Models

This section lists down the Data Models and their properties.

PushResult Class

This class contains the result of Push operation of indicators to the product API endpoint.
The success flag indicates the push operation result, and the message flag should have a proper error message in the case of failure. (In a case of success, a simple success message with success flag True should be returned)

Data Model Properties

Name	Type	Description
success	bool	success flag indicates the result of a push operation, whether it succeeded or failed.
message	string	Message field denotes the error in case of failure. In case of success, it can be a simple success message.

from netskope.integrations.cls.plugin_base import PushResult


PushResult(
    success=True, 
    message="Successfully pushed data to 3rd party."
)

ValidationResult Class

This class contains the result of the validation process on the plugin configuration parameters passed to the Plugin object while creating a new configuration for the plugin.
Make sure that all the parameters passed to the validate method are validated against the data-type and value.
Validate method returns the object of this class with a success flag indicating the result of validation operation and message field containing the proper error message in the case of validation failure. (In a case of success, a simple success message with success flag True has to be returned)

Data Model Properties

Name	Type	Description
success	bool	The success flag indicates the result of a validation operation, whether it succeeded or failed.
message	string	Message field denotes the error in case of failure. In case of success, it can be a simple success message.

rom netskope.integrations.cls.plugin_base import ValidationResult


ValidationResult(
           success=True,
           message="Validation Successful for Sample plugin"
       )

Logging

Log Shipper provides a handle of a logger object for logging.

Avoid print statements in the code.
This object logs to the central CE database with the timestamp field. Supported log levels are info, warn and error.
Make sure any API authentication secret or any sensitive information is not exposed in the log messages.
Make sure to implement a proper logging mechanism with the logger object passed by the CE platform.
Make sure enough logging is done, which helps the Ops team in troubleshooting.
Make sure any sensitive data is not logged or leaked in the notification or logs.

 self.logger.error(
               f"{PLUGIN_NAME}: Error log-message goes here."
 )


 self.logger.warn(
               f"{PLUGIN_NAME}: Warning log-message goes here."
 )


 self.logger.info(
               f"{PLUGIN_NAME}: Info log-message goes here."
 )

Notifications

Log Shipper provides a handle of notification object, which can be used to generate notifications on Log Shipper UI.

This object is passed from the CE platform to the Plugin object. Every plugin integration can use this object whenever there is a case of failure which has to be notified to the user immediately. The notification timestamp is managed by the CE platform.
This object will raise the notification in the UI, with a color-coding for the severity of the failure. Supported notification severities are info, warn and error.
Make sure any API authentication secret or any sensitive information is not exposed in the notification messages.
Use a notifier object to raise the notification for failures or critical situations (like rate-limiting, or exceeding payload size) to notify the status of the plugin to the user.

 self.notifier.info(
               f"{PLUGIN_NAME}: Info notification-message goes here."
 )


 self.notifier.error(
               f"{PLUGIN_NAME}: Error notification-message goes here."
 )


 self.notifier.warn(
               f"{PLUGIN_NAME}: Warning notification-message goes here."
 )

Testing

Linting

As part of the build process, we run a few linters to catch common programming errors, stylistic errors, and possible security issues.

Flake8

This is a basic linter. It can be run without having all the dependencies available and will catch common errors. We also use this linter to enforce the standard python pep8 formatting style. On rare occasions, you may encounter a need to disable an error/warning returned from this linter. Do this by adding an inline comment of the sort on the line you want to disable the error:

# noqa:

For example:

example = lambda: 'example' # noqa: E731

When adding an inline comment, always include the error code you are disabling for. That way if there are other errors on the same line they will be reported.

More info: https://flake8.pycqa.org/en/latest/user/violations.html#in-line-ignoring-errors.

PEP8 style docstring check is also enabled with the flake8 linter. So make sure every function/module has a proper docstring added.

Unit Testing

Ensure unit testing to test small units of code in an isolated and deterministic fashion. Make sure that unit tests avoid performing communication with external APIs and use mocking. Make sure unit tests ensure the code coverage is more than 70%.

Environment Setup

In order to work with unit testing the integration or automation script needs to be developed in plugin (directory) structure, We use PIP to install all the required Python module dependencies required to run the setup. Before running the tests, make sure you install all the required dependencies mentioned in the requirements.txt of the CE core repository.

Write Your Unit Tests

Make sure unit tests are written in a separate Python file named: _test.py. Within the unit test file, each unit test function should be named: test_. More information on writing unit tests and their format is available at the PyTest Docs.

Mocking

We use pytest-mock for mocking. pytest-mock is enabled by default and installed in the base environment mentioned above. To use a mocker object simply pass it as a parameter to your test function. The mocker can then be used to mock both the plugin class object and also external APIs.

For example:

def test_push(mocker, common_config):
    """To test push method of CLSPlugin."""
    cls_plugin = CLS_Plugin(
        <name>,
        common_config,
        None,
        None,
        logger,
        source="<source_name>",
        mappings=<mapping_json_string>,
    )
    
    # Initialize required classes
    syslogger = logging.getLogger(
        "SYSLOG_LOGGER_{}".format(threading.get_ident())
    )


    mocker.patch(
   "netskope.plugins.Default.cls.main.CLSPlugin.init_handler",
        return_value=syslogger,
    )


    try:
        cls_plugin.push(data, data_type, subtype)
    except Exception as e:
        assert False, f"Push raised exception: {e}"

Running Your Unit Tests

$ PYTHONPATH=. pytest

Deploy the Plugin on Cloud Exchange

Package the Plugin

CE expects the developed plugin in zip and tar.gz format.

Execute the below command to zip the package:

zip -r sample_plugin.zip sample_plugin

Execute the below command to generate tar.gz package:

tar -zcvf sample_plugin.tar.gz sample_plugin

Upload the Plugin

To deploy this zip or tar.gz in Cloud Exchange:

Log in to Cloud Exchange.
Go to Settings > Plugins.
Click Add New Plugin.
Click Browse.
Select the zip or tar.gz file.
Click Upload.

Add a Repo

To deploy your plugin, add your repo in Cloud Exchange:

Log in to Cloud Exchange.
Go to Settings > Plugin Repository.
Click Configure New Repository.
Provide a Repository Name, Repository URL, Username, and Personal Access Token.
Click Save.
Go to Settings > Plugins.
Select the name of your repository from the Repository dropdown.

Log Shipper Custom Plugin Developers Guide