User Risk Exchange Custom Plugin Developers Guide

This guide explains how to write a new User Risk Exchange plugin and extract maximum value out of your risk ecosystem by leveraging the functionality provided within the User Risk Exchange module. The developer should be able to write a new plugin independently without any technical issues by following this guide.

Prerequisites

Python 3.x programming experience (intermediate level).
Access to the Netskope Cloud Exchange platform.
API or Python SDK access to the product or solution for which you need to write the plugin.
An account with minimum permissions for the product.

User Risk Exchange Module

The Cloud Exchange platform, and its User Risk Exchange module, comes with a rich set of features and functionality that allow for a high degree of customization, so we recommend that you familiarize yourself with the different aspects of the platform as listed below. This module supports sharing of data from Netskope to third-party tools and vice versa.

Netskope Concepts & Terminology

Core: The Cloud Exchange core engine manages the 3rd party plugins and their life cycle methods. It has API endpoints for interacting with the platform to perform various tasks.
Module: The functional code areas that invoke modular-specific plugins to accomplish different workflows. User Risk Exchange is one of the modules running in Cloud Exchange.
Plugin: Plugins are Python packages that have logic to fetch users and risk scores from 3rd party Threat Intel systems, which will then be stored in User Risk Exchange. It can also perform actions.
Plugin Configurations: Plugin configurations are the plugin class objects which are configured with the required parameters and are scheduled by the Cloud Exchange core engine for fetching users and scores.
Users (UBA alerts): Records (of users) are objects having email and their risk scores gathered from various platforms and stored in the Cloud Exchange database.

Development Guidelines

Use the Plugin Directory Structure for all Python code.
Make sure all the 3rd party libraries packaged with the plugin package are checked for known vulnerabilities.
Make sure to follow standard Python code conventions. (https://peps.python.org/pep-0008/)
Run and verify that the flake8 Lint check passes with the docstring check enabled.The maximum length of a line should be 80.
Convert the time-stamp values to the human-readable format (from epoch to DateTime object). Make sure the time that is being displayed on the UI should be in the local timezone.
If possible, add a default value while adding a configuration parameter in the plugin.
For Scripts/Integrations written in Python, make sure to create unit tests. Refer to Unit Testing.
Plugin architecture allows storing states, however, avoid storing huge objects for state management.
Check your python code for any vulnerabilities.
The plugin icon has to be under 10kb. Make sure to use the company logo(and not the product’s) with a transparent background. The recommended size for the logo is 300×50 or a similar aspect ratio.
Use the checkpoint provided by the Cloud Exchange core rather than implementing one on your own.
The logger messages and the Toast messages should not contain the API Token and the Password type field values.
Pagination should always be considered while developing any feature in a plugin.
Make sure to add a retry mechanism for the 429 status code.
Use a notifier object to raise the notification for failures or critical situations (like rate-limiting, or exceeding payload size) to notify the status of the plugin to the user.
Make sure to implement a proper logging mechanism with the logger object passed by the Cloud Exchange platform. Make sure enough logging is done which helps the Ops team in troubleshooting. Make sure any sensitive data is not logged or leaked in the notification.
Provide the proper help text (tooltip) for all the parameters. If feasible, make sure to explain the significance of the parameter in the tooltip.
Make sure to provide a meaningful name and description to plugin configuration parameters.
Make sure to provide an appropriate configuration type (text, number, password, choice, multi-choice) to the parameters.
Make sure to use the proxy configuration dict and the SSL certificate validation flag that is passed by the Cloud Exchange platform while making any outbound request (API/SDK).
Make sure to collect the value of a non-mandatory parameter using the .get() method and provide a default value while using .get() method.
API Tokens and Password fields should not use strip().
The log messages should start with “<module> <app name> Plugin [configuration_name]: “. Example: “URE Crowdstrike Plugin [CrowdStrike Configuration Name]: <log_message>“. [This is a suggestion, we can avoid configuration name]. (logger.info(“<module> <plugin_name> Plugin: <message>”))
While logging an error log, if possible, we should add a traceback of the exception. USE: self.logger.error(error, details=traceback.format_exc())
The Toast message should not contain the <app_name> <module> Plugin: in the message.
Make sure to catch proper exceptions and status codes while and after making the API calls. If feasible, Devs can create a helper method from where the requests would be made and this method can be called with proper parameters wherever required.
The CHANGELOG.md file should be updated with proper tags such as Added, Changed, and Fixed along with a proper user-friendly message. Make sure the file name should exactly matches CHANGELOG.md.
Follow the Plugin Directory Structure guidelines.
If the description contains a link, it should be a hyperlink instead of plaintext.
Make sure to map the various fields received from API calls with the User and Score data model to leverage the full benefit of the system. Fields like Score in Records fields make more sense to the SOC user when analyzing the data. Make sure to map the score field which gives more context to the SOC analyst. The score field should be an integer.
Use proper validation for the parameters passed to the validate method and provide the proper help text for all the parameters.
Use a notifier object to raise the notification for failures or critical situations (like rate-limiting, exceeding payload size) to notify the status of the plugin to the user.
Make sure to implement a proper logging mechanism with the logger object passed by the Cloud Exchange platform. Make sure enough logging is done which helps the Ops team in troubleshooting. Make sure any sensitive data is not logged or leaked in the notification.
Make sure the plugin directory name (sample_plugin) matches with the manifest.json’s ID field.
User Agent should be added to the headers while making any API call. Format for the User Agent: netskope-ce-<ce_version>-<module>–<plugin_name>–<plugin_version>.
The logger statement should follow this format:
- logger.info(“<module> <plugin_name> Plugin: <message>”)

Writing a Plugin

This section illustrates the process of writing a plugin from scratch.

Download the sample plugin from the NetskopeOSS public Github repository or from the Cloud Exchange Knowledge Base found here: https://support.netskope.com/hc/en-us/articles/360052128734-Cloud-Threat-Exchange.

Development setup

Python

Our system utilizes Python3 (v3.7 and above). Make sure to set up python3 in your development environment. Pytest is used to run unit tests.

Included Python Libraries

Following Python libraries are included within the Netskope Cloud Exchange platform.

Library Name	Version
aiofiles	22.1.0
amqp	5.1.1
anyio	3.6.2
asgiref	3.6.0
attrs	22.2.0
azure-core	1.26.2
azure-storage-blob	12.14.1
bcrypt	4.0.1
boto3	1.26.51
botocore	1.29.51
billiard	3.6.4.0
celery	5.2.7
cabby	0.1.23
cachetools	5.2.1
celerybeat-mongo	0.2.0
certifi	2022.12.7
cffi	1.15.1
chardet	5.1.0
charset-normalizer	3.0.1
click	8.1.3
click-didyoumean	0.3.0
click-plugins	1.1.1
click-repl	0.2.0
colorama	0.4.6
colorlog	6.7.0
cryptography	39.0.0
cybox	2.1.0.21
defusedxml	0.7.1
dnspython	2.3.0
docker	6.0.1
fastapi	0.89.1
furl	2.1.3
google-api-core	2.11.0
google-auth	2.16.0
google-cloud-core	2.3.2
google-cloud-pubsub	2.13.12
google-cloud-pubsublite	1.6.0
google-cloud-storage	2.7.0
google-crc32c	1.5.0
google-resumable-media	2.4.0
googleapis-common-protos	1.58.0
grpc-google-iam-v1	0.12.6
grpcio	1.51.1
grpcio-status	1.51.1
gunicorn	20.1.0
h11	0.14.0
idna	3.4
importlib-metadata	6.0.0
isodate	0.6.1
jmespath	1.0.1
jsonpath	0.82
jsonschema	4.17.3
kombu	5.2.4
libcst	0.3.21
libtaxii	1.1.119
lxml	4.9.2
mongoengine	0.25.0
more-itertools	9.0.0
MarkupSafe	2.1.2
memory-profiler	0.61.0
mixbox	1.0.5
mongoquery	1.4.2
msrest	0.7.1
multidict	6.0.4
mypy-extensions	0.4.3
netskopesdk	0.0.25
numpy	1.23.5
oauthlib	3.2.2
onelogin	3.1.0
ordered-set	4.1.0
orderedmultidict	1.0.1
overrides	6.5.0
pandas	1.5.0
packaging	23.0
passlib	1.7.4
pycparser	2.21
prompt-toolkit	3.0.36
proto-plus	1.22.2
protobuf	4.21.12
psutil	5.9.4
pydantic	1.10.4
pyasn1	0.4.8
pyasn1-modules	0.2.8
PyJWT	2.6.0
pymongo	4.3.3
pyparsing	3.0.9
python-dateutil	2.8.2
pyrsistent	0.19.3
python-multipart	0.0.5
python3-saml	1.15.0
pytz	2022.7.1
PyYAML	6.0
requests	2.28.2
requests-oauthlib	1.3.1
rsa	4.9
six	1.16.0
starlette	0.22.0
sniffio	1.3.0
s3transfer	0.6.0
stix	1.2.0.11
taxii2-client	2.3.0
typing-inspect	0.8.0
typing-utils	0.1.0
typing_extensions	4.4.0
urllib3	1.26.14
uvicorn	0.20.0
vine	5.0.0
wcwidth	0.2.6
weakrefmethod	1.0.3
websocket-client	1.4.2
Werkzeug	2.2.2
xmlsec	1.3.11
zipp	3.11.0
requests-mock	1.7.0

Including Custom Plugin Libraries

Netskope advises bundling any of the third party python libraries your plugin will need into the plugin package itself. Use the pip installer to achieve this bundling; it provides a switch that takes a directory as input. If the directory is provided, pip will install the packages into that directory.

For example, the command shown below will install the “cowsay” package into the directory “lib”.

> pip install cowsay --target ./lib

For the official documentation on this, refer https://pip.pypa.io/en/stable/reference/pip_install/#cmdoption-t.

While importing modules from above lib folder, you should be using a relative import instead of an absolute import.

IDE

Recommended IDEs are PyCharm or Visual Studio Code.

Plugin Directory Structure

This section mentions the typical directory structure of a User Risk Exchange Plugin.

/sample_plugin/
  ├── __init__.py
  ├── Changelog.md
  ├── icon.png
  ├── main.py
  └── manifest.json

__init__.py: Every plugin package is considered a python module by the User Risk Exchange code. Make sure every plugin package contains the empty “__init__.py” file.
CHANGELOG.md: This file contains the details about plugin updates and should be updated with proper tags such as Added, Changed, and Fixed along with a proper user-friendly message.
icon.png: Plugin icon logo, this will be visible in the plugin chiclet and configuration cards on the UI. The logo should have a transparent background with a recommended size of 300*50 pixels or a similar aspect ratio.
main.py: This python file contains the Plugin class containing the concrete implementation for the fetch users and scores, execute actions, validate, and validate actions method.
manifest.json: Manifest file for the plugin package containing information about all the configurable parameters and their data types. This file has more information about the plugin integration as well.

The listed files here are mandatory for any plugin integration, but developers can add other files based on specific integration requirements.

Note

Make sure the plugin directory name (sample_plugin) matches with the manifest.json’s ID field.

Changelog.md

This is a file that contains details about plugin updates and should be updated with proper tags such as Added, Changed, and Fixed along with a proper user-friendly message.

Added: use it when new features are added.
Fixed: use it when any bug/error is fixed.
Changed: use it when there is any change in the existing implementation of the plugin.

Sample Changelog.md

# 1.0.1
## Fixed
- Fixed pagination while fetching records.

# 1.0.0
## Added
- Initial release.

Manifest.json

This is a JSON file that stores the meta-information related to the plugin, which is then read by the URE module to render the plugin in the UI as well as enabling the URE module to know more about the plugin, including required configuration parameters, the plugin-id, the plugin-name, etc.

Every plugin must contain this file with the required information so that User Risk Exchange can instantiate the Plugin object properly.
Common parameters for manifest.json include:
- name: (string) Name of the plugin. (Required)
- netskope:(Boolean) it is used to check if the plugin is a netskope plugin or not (example: True if it’s netskope plugin else False)
- description: (string) Description of the plugin. Provide a detailed description that mentions the capabilities and instructions to use the plugin, (ex. Fetches users and user scores and performs actions for members in group/groups) This description would appear on the Plugin Configuration card. (Required)
- ID: (string) ID of the plugin package. Make sure it is unique across all the plugins installed in the Cloud Exchange. The ID has to match the directory name of the plugin package. (Required)
- Type: ( [String] ): The type of users the data fetches. Can be Host or User or Both.
- version: (string) Version of the plugin. Usage of a MAJOR.MINOR.PATCH (ex. 1.0.1) versioning scheme is encouraged, although there are no restrictions. (Required)
- configuration: (array) Array of JSON objects that contains information about all the parameters required by the plugin – their name, type, id, etc. The common parameters for the nested JSON objects are explained below.
  - label: Name of the parameter. This will be displayed on the plugin configuration page. (Required)
  - key: Unique parameter key, which will be used as a key in the python dict object where the plugin configuration is used. (Required)
  - type: Value type of the parameter. Allowed values are ‘text’, ‘password’, ‘number’, ‘choice’, and ‘multichoice’. (Required) Refer to the Plugin Configuration parameter types below for more details.
  - default: The default value for this parameter. This value will appear on the plugin configuration page on URE UI. Supported data-types are “text”, “number”, and “list” (for multichoice type). (Required)
  - mandatory: Boolean which indicates whether this parameter is mandatory or not. If a parameter is mandatory, the User Risk Exchange UI won’t let you pass an empty value for the parameter. Allowed values are `true` and `false`. (Required)
  - description: Help text level description for the parameter which can give more details about the parameter and expected value. This string will appear on the plugin configuration page as a help-text. (Required)
  - choices: A list of JSON objects containing key and value as JSON keys. This parameter is only supported by ‘type’: ‘choice and multichoice`.

Plugin Configuration Parameter types

Make sure all the required plugin configuration parameters are listed under the configuration section of manifest.json for the plugin.

Password Parameter

Use this parameter for storing any secrets/passwords for authentication with API endpoints. Parameters with type as the password will have a password text box in the Plugin configuration page and will be obfuscated and encrypted by the platform.
Sample JSON

      "configuration": [
           {
               "label": "API Token",
               "key": "api_token",
               "type": "password"
           },
       ]

Plugin Configuration View

Text Parameter

Use this parameter for storing any string information, such as base-url, username, etc. This parameter will have a normal text input on the plugin configuration page.
Sample JSON

       "configuration": [
           {
               "label": "Tenant Name",
               "key": "tenant_name",
               "type": "text"
           },
       ]

Plugin Configuration View

Number Parameter

Use this parameter for storing number/float values. This parameter will have a number input box on the plugin configuration page. (The hash list is in the Threat Exchange module.)

       "configuration": [
           {
               "label": "Maximum File hash list size in MB.",
               "key": "max_size",
               "type": "number"
           },
       ]

Plugin Configuration View

Choice Parameter

Use this parameter for storing any enumeration parameter values. This parameter will have a dropdown box on the plugin configuration page.
Sample JSON

	"configuration": [
        {
            "label": "Base URL",
            "key": "base_url",
            "type": "choice",
            "choices": [
                {
                    "key": "Commercial cloud (api.crowdstrike.com)",
                    "value": "https://api.crowdstrike.com"
                },
                {
                    "key": "US 2 (api.us-2.crowdstrike.com)",
                    "value": "https://api.us-2.crowdstrike.com"
                },
                {
                    "key": "Falcon on GovCloud (api.laggar.gcw.crowdstrike.com)",
                    "value": "https://api.laggar.gcw.crowdstrike.com"
                },
                {
                    "key": "EU cloud (api.eu-1.crowdstrike.com)",
                    "value": "https://api.eu-1.crowdstrike.com"
                }
            ],
            "default": "https://api.crowdstrike.com",
            "mandatory": true,
            "description": "API Base URL."
        },

Plugin Configuration View

After selecting the input:

Multichoice Parameter

Use this parameter for storing multiple choice values. This parameter will have a dropdown box on the plugin configuration page with the ability to select multiple values. (Severity is in the Threat Exchange module.)

Sample JSON

    "configuration": [
        {
            "label": "Severity",
            "key": "severity",
            "type": "multichoice",
            "choices": [
                {
                    "key": "Unknown",
                    "value": "unknown"
                },
                {
                    "key": "Low",
                    "value": "low"
                },
                {
                    "key": "Medium",
                    "value": "medium"
                },
                {
                    "key": "High",
                    "value": "high"
                },
                {
                    "key": "Critical",
                    "value": "critical"
                }
            ],
            "default": [
                "critical",
                "high",
                "medium",
                "low",
                "unknown"
            ],
            "mandatory": false,
            "description": "Only indicators with matching severity will be saved."
        }
    ]

Plugin Configuration View

Toggle Parameter

This parameter stores a boolean value, toggle enabled is True and toggle disabled is False.
Use System Proxy (‘proxy’): Use system proxy configured in Settings.(Default: False)
Plugin Configuration View

Note

This parameter is provided by Core, and it is not allowed to add from the plugin’s manifest.json file.

Main.py

This python file contains the core implementation of the plugin.
Standard imports from netskope.integrations.cre.plugin_base import PluginBase, ValidationResult, PushResult
from netskope.integrations.cre.models.business_rule import Action, ActionWithoutParams, ActionWithoutParams, Action.
PluginBase variables
PluginBase provides access to variables that can be used during the plugin lifecycle Methods. Here is the list of variables.

Variable Name	Usage	Description
self.logger	self.logger.error(“Message”) self.logger.warn(“Message”) self.logger.info(“Message”)	Logger handle provided by core. Use this object to log important events. The logs would be visible in the Cloud Exchange Audit logs. Refer to the Logger Object documentation.
self.configuration	self.configuration.get(<attribute-key-name>)	JSON representation of the configuration object of the Plugin instance. Use this to access the configuration attributes like authentication credentials, server details, etc. Use the key name of the attribute mentioned in manifest.json.
self.last_run_at	If self.last_run_at: self.last_run_at.timestamp() Use this format to convert the last run time in epoch format.	Provides the timestamp of the last successful run time of the Plugin’s pull method. The Cloud Exchange core maintains the checkpoint time after each successful pull() execution. For the first execution, the value would be None. The datatype of the object is datetime.
self.storage		Cloud Exchange provides the plugin a mechanism to maintain the state. Use this object to persist in any state that would be required during subsequent calls. The datatype of this object is python dict.
self.notifier	self.notifier.info(“message”) self.notifier.warn(“message”) self.notifier.error(“message”)	This object provides a handle for User Risk Exchange core’s notifier. Use this object to push any notification to the platform. The notifications would be visible in the User Risk Exchange UI. Make sure the message contains summarized information for the user to read and take necessary actions. Used notifier in Netskope plugin if the push() method exceeds 8MB limit of the product.
self.proxy	requests.get(url=url, proxies=self.proxy)	Handle of system’s proxy settings if configured, else {}.
self.ssl_validation	requests.get(url=url, verify=self.ssl_validation)	Boolean value which mentions if ssl validation is enforced for REST API calls.