This document explains how to configure the v1.1.0 Web Page IoC Scraper plugin with the Threat Exchange module of the Netskope Cloud Exchange platform. This plugin is used to pull IoCs of types URL, Domain, IPv4, IPv6, MD5, and SHA256. The plugin does not support pushing of indicators.

Note that this plugin was previously named External Website; it has been renamed to Web Page IoC Scraper.

Prerequisites

To complete this configuration, you need:

• A Netskope tenant (or multiple, for example, production and development/test instances) that is already configured in Cloud Exchange.
• A Secure Web Gateway subscription for URL sharing. Refer to URL Lists for more information.
• A Netskope Cloud Exchange tenant with the Threat Exchange module already configured.
• A Publicly available Web Page URL.
• Connectivity to the following host: https://hub.docker.com.
• Connectivity to your Web Page IoC Scraper.

CE Version Compatibility

Netskope CE: v4.2.0, v5.0.1

Web Page IoC Scraper Plugin Support

This plugin supports pulling URL, IPv4, IPV6, Domains, SHA256, and MD5 types of indicators from any publicly available Website/Web Page. This plugin does not support pushing any indicators.

Fetched indicator types	URL, IPv4, IPv6, Domains, SHA256, MD5
Shared indicator types	Not Supported

API Details

Do not use any API to pull the data except for the file Endpoint/URL. You can use the File endpoint as your API Endpoint to pull the data.

Pull Data

API Endpoint: <URL of your Web Page IOC Scraper>

For example: https://bitbucket.org/abcd/netskope_ce_abcd/raw/0ee77838f1e1b0491c13e*********/ios.txt

Method: GET

Parameters: None

Sample API Response:

241.251.47.237
4e26f0a85c477205c552fb7dcff61f4b575c5424807252e197f5081877803a83
aa1b:0fb7:889f:7c3b:f2e0:6d17:0feb:0c09
19scuz0291fw7nzmvudlls.eu
201.219.216.230
mnjnzjcag.website
upbwbcfb93a.cn
http://branch.example.com/base/badge.aspx
https://www.example.org/

Performance Matrix

Here is the performance reading conducted after pulling 100K IoCs on a Large CE instance with the below specifications.

Stack details	Size: Large RAM: 32 GB CPU: 16 Cores
Indicators fetched from Web Page IoC Scraper	~35K per minute
Indicators shared with Web Page IoC Scraper	Not Supported

User Agent

netskope-ce-5.0.1-cte-web-page-ioc-scraper-v1.1.0

Workflow

1. Configure the Web Page IoC Scraper plugin.
2. Validate the Web Page IoC Scrapper plugin.

Click play to watch a video.

Configure the Web Page IoC Scraper Plugin

1. In Cloud Exchange, go to Settings > Plugins. Search for and select the Web Page IoC Scraper box to configure the plugin.
   
2. Enter values for the Basic Information:
   • Configuration Name: Unique name for the configuration
   • Sync Interval: Leave the default.
   • Aging Criteria: Expiry time of the plugin in days. (Default: 90)
   • Override Reputation: Set a value to override the reputation of indicators received from this configuration.
   • Enable SSL Validation: Enable SSL Certificate validation.
   • Use System Proxy: Enable if a proxy is required for communication.

   

3. Click Next.
4. Enter values for the Configuration Parameters:
   • Website URL: Add any publicly available URL from where the data is to be pulled.
   • Type of Threat data to pull: Select the type of Threat data you want to pull.

   

5. Click Save.
   

Add a Business Rule

Not Supported.

Add Sharing

Not Supported.

Validate the Web Page IoC Scraper Plugin

Validate the Pull

You can verify the pulling of IoCs from the plugin by going to Logging and checking the pulled logs from the Web Page IoC Scraper plugin.

You can check the pulled data stored in Cloud Exchange under Threat Exchange > Threat IoCs. Search the IoCs pulled from the plugin.

Validate the Push

NA

Troubleshooting

Unable to pull data from the plugin

If you are not able to pull IoCs from the platform it might be due to one of the following.

• Website URL is not public
• Available IOCs to pull are invalid or not supported

What to do: Check the website URL, it should be publicly accessible in order to pull IoCs. If that is the case check the data available to pull. The IoCs supported for pulling should be of type SHA256, ,MD5, URL, Domain, IPv4, IPv6, and valid.

Not able to pull IOCs of type IPv4, IPv6 and Domains after plugin update on CE versions below 5.0.1

If you are no longer able to pull the above mentioned IoC types it can be due to the URL bifurcation added in the plugin from CE version 5.0.1.

What to do: If you have updated your plugin on CE versions below 5.0.1 you might need to manually edit the plugin and select the IPv4, IPv6 and Domain, types of IoCs in the Type of Threat data to pull drop-down list. Earlier the plugin only supported the IoC types MD5, SHA256 and URL, where the URL itself included the subtypes. And new IoC type filters are added in the new plugin version hence they would not be selected in the existing configured plugin.

Note the IoCs of types IPv4, IPv6, and Domains will still be listed as type “URL” if the CE version is below 5.0.1 even though the latest plugin is configured. Since the support for URL bifurcation is available only from CE version 5.0.1.