Anomali ThreatStream XDR Plugin for Threat Exchange

This document explains how to configure the Anomali ThreatStream XDR plugin for the Threat Exchange module of the Netskope Cloud Exchange platform. This plugin fetches URL, Domain, IP (IPv4, IPv6), SHA256, and MD5 from Observables on the ThreatStream platform, and pushes the same to the Anomali ThreatStream XDR’s Observables.

Prerequisites

To complete this configuration, you need:

A Netskope tenant (or multiple, for example, production and development/test instances) that is already configured in Cloud Exchange.
A File Profile on your Netskope tenant.
A URL List on your Netskope tenant.
A Netskope Cloud Exchange tenant with the Threat Exchange module already configured.
An Anomali Threatstream Cloud Platform or On-Premises instance.
Connectivity to the following hosts:
- Anomali Threatstream Cloud Platform or On-Premises Setup.
- For Cloud Platform : https://api.threatstream.com.
- For On-Premises: https://<ThreatStream_On-Premises_IP_or_FQDN>.

CE Version Compatibility

Netskope CE: v4.2.0, v5.0.1

Anomali ThreatStream XDR Plugin Support

The plugin is used to pull and push IoCs of type URL, IPv4, IPv6, Domains, MD5, and SHA256 from/to Anomali ThreatStream XDR’s Observables.

Fetched indicator types	URL (Domains, URLs, IP(IPv4), IPv6), SHA256, MD5
Shared indicator types	URL (Domains, URLs, IP(IPv4), IPv6), SHA256, MD5

Performance Matrix

This reading is conducted on a Large CE Stack with these specs by pulling and pushing 100K IoCs.

Stack details	Size: Large RAM: 32 GB CPU: 16 Cores
Indicators fetched from Anomali ThreatStream XDR	~33K per minute
Indicators shared with Anomali ThreatStream XDR	~50K per minute

Mappings

Severity Mappings

Netskope Severity	Anomali ThreatStream XDR Severity
low	low
medium	medium
high	high
critical	very-high
unknown	–

Indicator Mappings for Pull and Push

Netskope Fields	Anomali ThreatStream XDR Observable Fields
value	value
type	type
firstSeen	created_ts
lastSeen	modified_ts
reputation	confidence
severity	meta.severity
comments	description
tags	tags.name

Reputation-Confidence Mapping

Netskope Reputation	Anomali ThreatStream XDR Confidence
1	1-10
2	11-20
3	21-30
4	31-40
5	41-50
6	51-60
7	61-70
8	71-80
9	81-90
10	91-100

Permissions

These permissions are needed for the plugin configuration.

Approval Intel
Show API for Users

API Details

List of APIs used

API Endpoint	Method	Use Case
/api/v2/intelligence	GET	To pull indicators.
/api/v2/intelligence/	PATCH	To push indicators.

Pull Indicators

API Endpoint:

/api/v2/intelligence

Method: GET

Parameters:

limit: 1000

modified_ts__gt: 2024-01-23T13:33:40.314Z

order_by:”update_id”

update_id__gt: 0

type: domain, ip, ipv6, hash, url

subtype: MD5, SHA256

The IoCs are pulled using the modified_ts field.

Headers:

Authorization: apikey <USERNAME>:<PASSWORD>

Accept: application/json

Content-Type: application/json

API Request Endpoint:

/api/v2/intelligence

Sample API Response:

{
    "objects": [
        {
            "source": "Botscout BOT IPs",
            "threatscore": 17,
            "threat_type": "bot",
            "trusted_circle_ids": [
                146,
                211,
                388
            ],
            "description": null,
            "workgroups": [],
            "sort": [
                17729579514
            ],
            "resource_uri": "/api/v2/intelligence/60654060239/",
            "modified_ts": "2023-10-18T11:11:57.115Z",
            "update_id": 17729579514,
            "source_reported_confidence": 65,
            "type": "ipv6",
            "uuid": "39a5d9b0-c48e-4831-afdb-511f57c039ec",
            "feed_id": 141,
            "retina_confidence": -1,
            "created_ts": "2023-10-18T10:52:26.484Z",
            "id": 60654060239,
            "value": "2401:4900:171:609:11:98:550:76",
            "itype": "bot_ipv6",
            "org": "",
            "confidence": 65,
            "expiration_ts": "2024-01-16T10:52:23.000Z",
            "owner_organization_id": 2,
            "meta": {
                "severity": "medium",
                "detail2": "imported by user 668"
            },
            "is_anonymous": false,
            "is_public": false,
            "asn": "",
            "status": "active",
            "tags": [
                {
                    "id": "1yx",
                    "name": "childfirstbehaviortherapy.seo"
                },
                {
                    "id": "3t7",
                    "name": "childfirstbehaviortherapy.seo@gmail.com"
                }
            ],
            "can_add_public_tags": false,
            "subtype": null,
            "tlp": null,
            "created_by": null,
            "rdns": null,
            "is_editable": false
        }
    ],
    "meta": {
        "offset": 0,
        "limit": 1,
        "total_count": 1499,
        "next": "/api/v2/intelligence/?limit=1&modified_ts__gt=2023-04-08T19%3A46%3A42.345Z&order_by=update_id&status=active&type=ipv6&update_id__gt=0&search_after=17729579514",
        "took": 34
    }
}

Push Indicators

API Endpoint:

/api/v2/intelligence/

Method: PATCH

Request Body:

{
    "meta":{
       "classification":"private",
       "allow_unresolved":true
    },
    "objects":[
       {
          "srcip":"2.1.0.6",
          "tags":[
             {
                "name":"private_tag",
                "tlp":"red"
             },
             {
                "name":"public_tag",
                "tlp":"white"
             }
          ],
          "severity":"high"
       },
       {
          "domain":"idfsdszqylwjzq2106.biz",
          "severity":"very-high"
       },
       {
          "url":"https://test.com",
          "severity":"high"
       },
       {
          "md5":"58fc769228711b86d905a8deb6f390799a8c1dc86dcc4b4b831734c3b25df470",
          "severity":"high",
          “confidence”:60,
          "expiration_ts":"2017-01-26T00:00:00",
          "subtype": "SHA256"
       }
    ]
 }

Headers:

Authorization: apikey <USERNAME>:<PASSWORD>

Accept: application/json

Content-Type: application/json

API Request Endpoint:

/api/v2/intelligence/

Sample API Response:

202 Accepted

User Agent

netskope-ce-5.0.1-cte-anomali-threatstream-xdr-v1.2.0

Workflow

Get the API Key and set necessary permissions.
Configure the Anomali ThreatStream XDR plugin.
Add a Business Rule for Anomali ThreatStream XDR.
Configure Sharing between Netskope and Anomali ThreatStream XDR.
Validate the Anomali ThreatStream XDR plugin.

Click play to watch a video.

Get your Anomali ThreatStream XDR API Token and Set Permissions

Log in to your Anomali ThreatStream XDR platform and click on the Settings icon from the top left corner.
In MY PROFILE, under Account Information, click Reveal for the API Key. Save the API key for the platform to use later in the plugin configuration.

Get your Username and Check the Permissions

For Username, use your email address that you have used while logging in the platform. Make sure your user has these permissions added for the plugin to run.

Approval Intel
Show API for Users

Go to Settings > User Admin to check if the above permissions are added to your user account. If not, provide these permissions.

Configure the Anomali ThreatStream XDR Plugin

Log in to Cloud Exchange and go to Settings > Plugins.
Search for and select the Anomali ThreatStream XDR plugin box to configure the plugin.
Enter these values:
- Configuration Name: Unique name for the configuration
- Sync Interval: Leave default
- Aging Criteria: Expiry time of the plugin in days. (Default: 90)
- Override Reputation: Set a value to override the reputation of indicators received from this configuration.
- Enable SSL Validation: Enable SSL Certificate validation.
- Use System Proxy: Enable if a proxy is required for communication.
Click Next.
Enter these values:
- Base URL: The ThreatStream API Base URL based on your Cloud or On-Premises instance.
- Username: The ThreatStream platform username you use when logging in.
- API Key: API Key you got from the ThreatStream Anomali XDR platform.
- Remote Observables: Select Yes or No to retrieve the Remote Observables.
- Type of Threat Data to Pull: Select based on the data you want to pull. The plugin supports type and subtype filtering from Anomali for the IoC types supported in CE
- Minimum Confidence: Provide a value for confidence (1-100) IoCs whose confidence is greater than or equal to the provided value that will be pulled.
- Severity: IoCs with the provided severity will be pulled.
- Status: IoCs with the matching status will be fetched.
- Tags: Indicators matching the Tags will be pulled. Enter single or multiple comma separated Tag names. Leave blank to pull IoCs with all Tags.
- Feed ID: Indicators matching the Feed ID will be pulled. Enter single or multiple comma separated numeric values for the Feed ID. Leave blank to pull IoCs from all feeds.
  Note
  - When multiple Tags or Feed IDs are added, the plugin will pull data from each mentioned tag using the OR operation.
  - If you have added both Tags and Feed ID in the plugin for pulling the data, the IoCs will be pulled using AND operation between the tags and Feed ID. The IoCs will only be pulled if the specified Feed ID has the mentioned tags.
- Enable Polling: Select Yes to pull data, or No if plugin is used for pushing.
- Enable Tagging: Select Yes to pull tags along with all IoCs from the platform.
- Initial Range: Number of days Threat IoCs will be pulled from the initial range.
Click Save.

Add a Threat Exchange Business Rule for Anomali ThreatStream XDR

To share indicators fetched from the Anomali ThreatStream XDR to Netskope CE, and vice versa, you will need to have a business rule that will filter out the indicators that you want to share. To configure a business rule, follow these steps:

Go to Threat Exchange > Business Rule and click Create New Rule.
Add the filter according to your requirements in the rule, and then click Save.

Configure Sharing for Netskope and Anomali ThreatStream XDR

To share IoCs from the Netskope Cloud Exchange to the Anomali ThreatStream XDR platform, and vice versa, follow these steps:

Go to Threat Exchange > Sharing and click Add Sharing Configuration.
Select the Source Configuration (Netskope CTE), the Business Rule, the Destination Configuration (Anomali ThreatStream XDR), and Target(s). Enter these values:
- URL iType: Select the iType that you want to assign to your URL.
- IP iType: Select the iType that you want to assign to your IP.
- IPv6 iType: Select the iType that you want to assign to your IPv6.
- Domain iType: Select the iType that you want to assign to your Domain.
- Hash (MD5, SHA256]) iType: Select the iType that you want to assign to your Hash (MD5, SHA256).
Click Save.
Repeat these steps, but with the configurations reversed, for sharing Anomali ThreatStream XDR IoCs to Netskope. Select the Source Configuration (Anomali ThreatStream XDR), the Business Rule, the Destination Configuration (Netskope CTE), and Target(s). Select the existing IoC List Name, or create a new IoC.
Click Save.

Validate the Anomali ThreatStream XDR Plugin

Validate the Pull

Pulled data will be listed on the Threat Exchange Threat IoCs page. You can filter the IoCs pulled from the platform using the filter: sources.source, contains <plugin name>.

To verify pulled logs on CE, go to Logging and search logs for the CTE Anomali ThreatStream plugin.

To verify the data available for pulling on Anomali ThreatStream XDR, log in to Anomali ThreatStream XDR and go to Analyze > Observables.

The plugin pulls IoCs seen on the Observables page.

Validate the Push

To validate the push in CE, go to Logging and filter shared logs for the Anomali plugin.

You will receive a summary log like below by the end of a pull cycle:

CTE Anomali ThreatStream XDR [CTE Anomali ThreatStream XDR]: Successfully shared 5 indicator(s).

Go to Threat IoCs and filter logs shared with Anomali plugin.

Note

The Anomali has a limit to ingest 10MB data in one push cycle, so if the data to be ingested is more than 10MB it will be divided in 10MB chunks to push.

On the Anomali platform, go to Analyze > Observables > IoC. Pushed IoCs from CE will have netskope-ce added as a tag.

All IoCs pushed from the Netskope CE will have the Visibility as private, like My Organization, and it will not be a part of the Anomali community.

Troubleshooting

Unable to pull IoCs from the Anomali ThreatStream platform

After the plugin configuration, if the IoCs are not pulled from the platform it might be due to one of these reasons:

No IoCs are available on the platform to pull.
IoCs are not available for the given time range or does not match the configuration parameters.

What to do: Identity your root cause from above and follow one of these steps to resolve the issue.

No IoCs are available on the platform to pull

Check if the IoCs are available on the platform to pull. If available, check the resolution for the next point.

IoCs are not available for the given time range

If the IoCs are available on the platform to pull, but the plugin has not pulled the IoCs in CE, check the number of days mentioned in the initial range parameter of the plugin configuration. On the Anomali platform, check if you have data for the given time range.

If the data is still available for the given time range it might be possible that the IoCs for the provided filter in the plugin configuration are not available, so check the values from the plugin configuration parameter and filter the same on the Anomali platform.

Also make sure that you have Yes selected in the enable polling, to pull the IoCs.

Unable to push the IoCs to Anomali

If you are not able to push the IoCs on the platform and receive an error while pushing, it might be due to insufficient permission to the user.

IoCs are pushed from CE but not available on the Anomali platform

If IoCs are pushed and not reflected on the platform it might be due to the IoCs count being too big, when data is shared to Anomali it sometimes takes time to reflect on the platform, at max 10MB data can be shared in a page at a time.

What to do: The pushed data will be reflected but it might take some time, sometimes minutes, sometimes hours, so we suggested waiting for the data to be reflected.

Known Behaviors

These factors have been noticed while verifying the plugin workflow.

Sometimes multiple IoCs for a single record are created on the Anomali ThreatStream platform after the IoCs are shared.
Sometimes ingested IoCs take time to reflect on the Anomali platform after IoCs are pushed, this might take minutes or in some cases hours.
Unknown severity from CE is not mapped in Anomali, so when IoCs are shared with Unknown severity, the platform calculates its own severity and assigns it to the IoC.
The iType for the IoCs won’t be updated if the IoCs are reshared with updated iType.

Anomali ThreatStream XDR Plugin for Threat Exchange