Local Export Plugin for Log Shipper

This document explains how to configure the Local Export with the Log Shipper module of the Netskope Cloud Exchange platform. This plugin is used to deliver web transactions data to a designated location in your local storage.

Prerequisites

• A Netskope tenant (or multiple, for example, production and development/test instances).
• A Netskope Cloud Exchange tenant with the Tenant plugin and Log Shipper plugin already configured.
• A Netskope Cloud Exchange tenant with the AWS Netskope Log Streaming or Azure Netskope Log Streaming plugin already configured (for pulling WebTx from the Netskope Log Streaming plugins).

Plugin Scope

This plugin is used to deliver web transactions data to the Local Storage where your Netskope Cloud Exchange is installed.

Local Export Plugin Support

Data Type	Support
Events	No
Alerts	No
WebTx Logs	Yes (via Netskope Log Streaming)

Permissions

• Permission to create a folder in a container with Read, Write, and Execute rights.
• Permission to mount the folder in Docker File (required only if you want the data outside of the containers).

Performance Matrix

Workflow

1. Mount a directory outside of the core container.
2. Configure the Local Export plugin.
3. Configure Log Delivery for the Local Export plugin.
4. Validate the plugin.

Click play to watch a video.

Mount a Directory Outside of the Core Container

Utilizing mounting allows data retrieval from the Core Container without the need to access the container itself. Moreover, in the event of the container shutting down, all data within the container is deleted.
Follow these steps to mount the container with folder:

1. Log in to the machine where CE is installed.
2. Create the folder where you want to store data outside the container.
   

   

3. Go to the folder you created, and run pwd command for the path of that folder, and copy that path because it will be required for mounting.

   

4. Go to the folder where Netskope CE is installed.
   

   

5. Run this command to edit the docker-compose.yml file: vi docker-compose.yml.
   

   

6. Inside docker-compose.yml, go to core:, and below volumes:, add the new line with the path of the folder on the left separated with :, and then enter the path inside of the container in which you going to collect the WebTx data.
   Path would be like /home/devuser/WebTx_Data:/opt/Local_Export.
   

   

7. Now go inside your docker container to create a empty folder that will collect data inside it.
   To go inside the docker container, run docker-compose exec -u 0 core sh.
   

   

8. Inside the container, create a folder with the name you used in the docker-compose.yml file (like Local_Export as the folder name).
   

   

9. Give permission for Write and Executable using: chmod 777 -R Local_Export(Folder Name).
   

   

10. Exit the container using exit.
    

    

11. Restart the Core container using docker-compose restart core.
    

    

12. Configure the Plugin using the same directory created inside the container.

Configure the Local Export Plugin

1. In Cloud Exchange, go to Settings > Plugin Store.
2. Search for and select the Local Export v1.0.0 (CLS) plugin.

   

3. Enter a Configuration Name.
4. Click Next.
   

   

5. Enter these parameters:
   • Storage Path: Storage path where data objects will be stored.
   • Object Prefix: Object prefix for the data object name while pushing to the storage path. ‘/’ is not allowed in the object prefix.
   • Maximum File Size (in MBs): Maximum size of data object to be stored in the storage path. Value should be between 1 to 100.
   • Maximum Duration (in Seconds): Maximum duration after which the data object should be stored in the storage path.

     

6. Click Save. Your new plugin is be available on the Cloud Log Shipper > Plugins page.

Configure Log Delivery for the Local Export Plugin

1. Go to Log Shipper > Log Delivery and click Add Log Delivery Configuration.
2. Select the Source plugin (AWS LogStreaming or Azure LogStreaming), the Destination plugin (Local Export) and click Save.
3. After the Log Delivery configuration is added, the data will start to be pulled from the Netskope tenant, transformed, and ingested inside the container folder.

Validate the Local Export Plugin

Validate the Push

To validate the plugin workflow in Cloud Exchange.

1. Go to Logging and search for ingested events with the filter CLS Local Export [plugin name]Successfully.
2. The ingested logs will be filtered.
   

   

To validate the push from the docker container, follow these steps:

1. Log in to the machine where Cloud Exchange is installed.
2. Move to the folder where Cloud Exchange is installed.
   

   

3. Run docker-compose exec -u 0 core sh.
   

   

4. Go to the Storage Path you entered while configuring the plugin.
5. Run ls.
   

   

6. If you are using the Mounted Directory, then you can also check in that directory.
   

   

Troubleshooting the Local Export Plugin

Check folder has enough permissions to ingest data.

If not, then you need to give permission to that folder inside the container.

1. Log in to the machine where Cloud Exchange is installed.
2. Move to the folder where Cloud Exchange is installed.
   

   

3. Run docker-compose exec -u 0 core sh.
   

   

4. To check permissions, run ls -l.
   

   

5. To give permission to the folder, use chmod 777 -R Local_export (The folder where you want to ingest the data)

   

Limitations

• This plugin’s functionality is limited to container-based deployments and it will not work with OVA deployments, as it does not allow editing the docker-compose.yml file. The changes required in the docker-compose.yml file will differ for each deployment. Refer to the respective deployment guides for the required changes.
• As this plugin stores the WebTx data locally, it would consume an outrageous amount of storage space and can cause storage issues.