Local Export Plugin for Log Shipper

This document explains how to configure the Local Export with the Log Shipper module of the Netskope Cloud Exchange platform. This plugin is used to deliver web transactions data to a designated location in your local storage.

Prerequisites

• A Netskope tenant (or multiple, for example, production and development/test instances) that is already configured in Cloud Exchange.
• A Netskope Cloud Exchange tenant with the Log Shipper module already configured.
• A Netskope Cloud Exchange tenant with the WebTx plugin already configured.

CE Version Compatibility

This plugin is compatible with Netskope CE: v4.2.0 and v5.0.0.

Plugin Scope

This plugin is used to deliver web transactions data to the Local Storage where your Netskope CE is installed.

Local Export Plugin Support

Event Types	No
Alert Types	No
WebTx Support	Yes

Permissions

• Permission to create a folder in a container with Read, Write, and Execute rights.
• Permission to mount the folder in Docker File (required only if you want the data outside of the containers).

Performance Matrix

This performance reading is for a Large Stack CE with 6MBps WebTx data tested with these VM specifications.

Stack details	Size: Large RAM: 32 GB CPU: 16 Cores
WebTx ingested to Local Storage	~6 MB per second

Workflow

1. Mount a directory outside of the core container.
2. Configure the CLS Local Export plugin.
3. Configure SIEM Mappings.
4. Validate the plugin.

Click play to watch a video.

Mount a Directory Outside of the Core Container

Utilizing mounting allows data retrieval from the Core Container without the need to access the container itself. Moreover, in the event of the container shutting down, all data within the container is deleted.
Follow these steps to mount the container with folder:

1. Log in to the machine where CE is installed.
2. Create the folder where you want to store data outside the container.
   
3. Go to the folder you created, and run pwd command for the path of that folder, and copy that path because it will be required for mounting.
4. Go to the folder where Netskope CE is installed.
   
5. Run this command to edit the docker-compose.yml file: vi docker-compose.ymls.
   
6. Inside docker-compose.yml, go to core:, and below volumes:, add the new line with the path of the folder on the left separated with :, and then enter the path inside of the container in which you going to collect the WebTx data.
   Path would be like /home/devuser/WebTx_Data:/opt/Local_Export.
   
7. Now go inside your docker container to create a empty folder that will collect data inside it.
   To go inside the docker container, run docker-compose exec -u 0 core sh.
   
8. Inside the container, create a folder with the name you used in the docker-compose.yml file (like Local_Export as the folder name).
   
9. Give permission for Write and Executable using: chmod 777 -R Local_Export(Folder Name).
   
10. Exit the container using exit.
    
11. Restart the Core container using docker-compose restart core.
    
12. Configure the Plugin using the same directory created inside the container.

Configure the Local Export Plugin

1. Go to Settings > Plugins.
2. Search for and select the Local Export box to open the plugin creation dialog.
3. Enter a Configuration Name.
4. Click Next.
   
5. Enter these parameters:
   • Storage Path: Storage path where data objects will be stored.
   • Object Prefix: Object prefix for the data object name while pushing to the storage path. ‘/’ is not allowed in the object prefix.
   • Maximum File Size (in MBs): Maximum size of data object to be stored in the storage path. Value should be between 1 to 100.
   • Maximum Duration (in Seconds): Maximum duration after which the data object should be stored in the storage path.
6. Click Save. Your new plugin is be available on the Cloud Log Shipper > Plugins page.

Configure SIEM Mappings for the Local Export Plugin

1. Go to Log Shipper > SIEM Mappings and click Add SIEM Mapping.
2. Select the Source plugin (Netskope WebTx), Destination plugin (Local Export) and click Save.
3. After the SIEM mapping is added, the data will start to be pulled from the Netskope tenant, transformed, and ingested inside the container folder.

Validate the Local Export Plugin

Validate the Push

To validate the plugin workflow in Cloud Exchange.

1. Go to Logging and search for ingested events with the filter CLS Local Export [plugin name]Successfully.
2. The ingested logs will be filtered.
   

To validate the push from the docker container, follow these steps:

1. Log in to the machine where CE is installed.
2. Move to the folder where CE is installed.
   
3. Run docker-compose exec -u 0 core sh.
   
4. Go to the Storage Path you entered while configuring the plugin.
5. Run ls.
   
6. If you are using the Mounted Directory, then you can also check in that directory.
   

Troubleshooting

Check folder has enough permissions to ingest data.

If not, then you need to give permission to that folder inside the container.

1. Log in to the machine where CE is installed.
2. Move to the folder where CE is installed.
   
3. Run docker-compose exec -u 0 core sh.
   
4. To check permissions, run ls -l.
   
5. To give permission to the folder, use chmod 777 -R Local_export (The folder where you want to ingest the data)

Limitations

• This plugin’s functionality is limited to container-based deployments and it will not work with OVA deployments, as it does not allow editing the docker-compose.yml file. The changes required in the docker-compose.yml file will differ for each deployment. Refer to the respective deployment guides for the required changes.
• As this plugin stores the WebTx data locally, it would consume an outrageous amount of storage space and can cause storage issues. For an ingestion of 1 hour, it takes around ~2 GB storage space; for 24 hours, ~50 GB storage space.