Anomali ThreatStream XDR Plugin for Threat Exchange

This document explains how to configure the Anomali ThreatStream XDR plugin for the Threat Exchange module of the Netskope Cloud Exchange platform. This plugin fetches URL, Domain, IP (IPv4), IPv6, SHA256, and MD5 and pushes the same to the Anomali ThreatStream XDR platform.

Prerequisites

To complete this configuration, you need:

A Netskope tenant (or multiple, for example, production and development/test instances) that is already configured in Cloud Exchange.
A File Profile on the Netskope tenant.
A URL List on the Netskope tenant.
A Netskope Cloud Exchange tenant with the Threat Exchange module already configured.
An Anomali Threatstream Cloud Platform or OnPrem instance.
Connectivity to the following hosts:
Anomali Threatstream Cloud Platform or OnPrem Setup
For Cloud Platform : https://apii.threatstream.com
For OnPrem: https://<ThreatStream_OnPrem_IP_or_FQDN>.

CE Version Compatibility

Netskope CE: v4.2.0, v5.0.0

Anomali ThreatStream XDR Plugin Support

Fetched indicator types	URL (Domains, URLs, IP(IPv4), IPv6), SHA256, MD5
Shared indicator types	URL (Domains, URLs, IP(IPv4), IPv6), SHA256, MD5

Performance Matrix

This reading is conducted on a Large CE Stack with below mentioned specs by pulling and pushing 100K IOCs.

Stack details

Size: Large

RAM: 32 GB

CPU: 16 Cores

Indicators fetched from third-party product

~20K per minute

Indicators shared with third-party product

~20K per minute

Mappings

Severity Mapping

Netskope Severity	Anomali ThreatStream XDR Severity
low	low
medium	medium
high	high
critical	very-high
unknown	–

Indicator Mapping for Pull and Push

Netskope field	Anomali ThreatStream XDR Observable field
value	value
type	type
firstSeen	created_ts
lastSeen	modified_ts
reputation	confidence
severity	meta.severity
comments	description
tags	tags.name

Reputation-Confidence Mapping

Netskope Reputation	Anomali ThreatStream XDR Confidence
1	1-10
2	11-20
3	21-30
4	31-40
5	41-50
6	51-60
7	61-70
8	71-80
9	81-90
10	91-100

Permissions

Below are the permissions needed for the plugin workflow.

Approval Intel
Show API for Users

API Details

List of APIs used

API Endpoint	Method	Use case
/api/v2/intelligence	GET	To pull indicators.
/api/v2/intelligence/	PATCH	To push indicators.

Pull Indicators

API Endpoint:

/api/v2/intelligence

Method: GET

Parameters:

limit: 1000

modified_ts__gt: 2024-01-23T13:33:40.314Z

order_by:”update_id”

update_id__gt: 0

type: domain, ip, ipv6, hash, url

subtype: MD5, SHA256

Headers:

Authorization: apikey <USERNAME>:<PASSWORD>

Accept: application/json

Content-Type: application/json

API Request Endpoint:

https://api.threatstream.com/api/v2/intelligence

Sample API Response:

{
    "objects": [
        {
            "source": "Botscout BOT IPs",
            "threatscore": 17,
            "threat_type": "bot",
            "trusted_circle_ids": [
                146,
                211,
                388
            ],
            "description": null,
            "workgroups": [],
            "sort": [
                17729579514
            ],
            "resource_uri": "/api/v2/intelligence/60654060239/",
            "modified_ts": "2023-10-18T11:11:57.115Z",
            "update_id": 17729579514,
            "source_reported_confidence": 65,
            "type": "ipv6",
            "uuid": "39a5d9b0-c48e-4831-afdb-511f57c039ec",
            "feed_id": 141,
            "retina_confidence": -1,
            "created_ts": "2023-10-18T10:52:26.484Z",
            "id": 60654060239,
            "value": "2401:4900:171:609:11:98:550:76",
            "itype": "bot_ipv6",
            "org": "",
            "confidence": 65,
            "expiration_ts": "2024-01-16T10:52:23.000Z",
            "owner_organization_id": 2,
            "meta": {
                "severity": "medium",
                "detail2": "imported by user 668"
            },
            "is_anonymous": false,
            "is_public": false,
            "asn": "",
            "status": "active",
            "tags": [
                {
                    "id": "1yx",
                    "name": "childfirstbehaviortherapy.seo"
                },
                {
                    "id": "3t7",
                    "name": "childfirstbehaviortherapy.seo@gmail.com"
                }
            ],
            "can_add_public_tags": false,
            "subtype": null,
            "tlp": null,
            "created_by": null,
            "rdns": null,
            "is_editable": false
        }
    ],
    "meta": {
        "offset": 0,
        "limit": 1,
        "total_count": 1499,
        "next": "/api/v2/intelligence/?limit=1&modified_ts__gt=2023-04-08T19%3A46%3A42.345Z&order_by=update_id&status=active&type=ipv6&update_id__gt=0&search_after=17729579514",
        "took": 34
    }
}

Push Indicators

API Endpoint:

/api/v2/intelligence/

Method: PATCH

Request Body:

{
    "meta":{
       "classification":"private",
       "allow_unresolved":true
    },
    "objects":[
       {
          "srcip":"2.1.0.6",
          "tags":[
             {
                "name":"private_tag",
                "tlp":"red"
             },
             {
                "name":"public_tag",
                "tlp":"white"
             }
          ],
          "severity":"high"
       },
       {
          "domain":"idfsdszqylwjzq2106.biz",
          "severity":"very-high"
       },
       {
          "url":"https://test.com",
          "severity":"high"
       },
       {
          "md5":"58fc769228711b86d905a8deb6f390799a8c1dc86dcc4b4b831734c3b25df470",
          "severity":"high",
          “confidence”:60,
          "expiration_ts":"2017-01-26T00:00:00",
          "subtype": "SHA256"
       }
    ]
 }

Headers:

Authorization: apikey <USERNAME>:<PASSWORD>

Accept: application/json

Content-Type: application/json

API Request Endpoint:

https://api.threatstream.com/api/v2/intelligence/

Sample API Response:

202 Accepted

User Agent

netskope-ce-5.0.0-cte-anomali-threatstream-xdr-v1.0.1

Workflow

Get the API Key and set necessary permissions.
Configure the Anomali ThreatStream XDR plugin.
Add a Business Rule for Anomali ThreatStream XDR.
Configure Sharing between Netskope and Anomali ThreatStream XDR.
Validate the Anomali ThreatStream XDR plugin.

Click play to watch a video.

Get your Anomali ThreatStream XDR API Token and Set Permissions

Log in to your Anomali ThreatStream XDR platform and click on the Settings icon from the top left corner.
In MY PROFILE, under Account Information, click Reveal for the API Key. Save the API key for the platform to use it later in the plugin configuration.

Get your Username and Check the Permissions

For Username, use your email address that you have used while logging in the platform. Make sure your user has below permissions added for the plugin to run.

Approval Intel
Show API for Users

Go to Settings > User Admin to check if the above permissions are added to your user account. If not, provide these permissions.

Configure the Anomali ThreatStream XDR Plugin

Log in to Cloud Exchange and go to Settings > Plugins.
Search for and select the Anomali ThreatStream XDR plugin box to configure the plugin.
Enter these values:
- Configuration Name: Unique name for the configuration
- Sync Interval: Leave default
- Aging Criteria: Expiry time of the plugin in days. (Default: 90)
- Override Reputation: Set a value to override the reputation of indicators received from this configuration.
- Enable SSL Validation: Enable SSL Certificate validation.
- Use System Proxy: Enable if the proxy is required for communication.
Click Next.
Enter these values:
- Base URL: The ThreatStream API Base URL based on your Cloud or OnPrem instance.
- Username: The ThreatStream platform username you use while logging in.
- API Key: API Key fetched from the ThreatStream Anomali XDR platform.
- Remote Observables: Select Yes or No to retrieve the Remote Observables.
- Type of Threat Data to Pull: Select based on the data you want to pull.
- Minimum Confidence: Provide value for confidence(in range of 1-100) IoCs whose confidence is greater than or equal to the provided value will be pulled.
- Severity: IoCs with provided severity will be pulled.
- Status: IoCs with matching Status will be fetched.
- Enable Polling: Keep Yes to pull data, eep No if plugin is used for pushing.
- Enable Tagging: Keep Yes to pull tags along with all IoCs from the platform.
- Initial Range: Number of days Threat IoCs to pull from initial range.
Click Save.

Add a Business Rule for Anomali ThreatStream XDR

To share indicators fetched from the Anomali ThreatStream XDR to the Netskope CE and vice versa you will need to have a business rule that will filter out the indicators that you want to share. To configure a business rule, follow these steps:

Go to Threat Exchange > Business Rule and click Create New Rule.
Add the filter according to your requirement in the rule.

Configure Sharing for Netskope and Anomali ThreatStream XDR

To share IOCs from the Netskope Cloud Exchange to the Anomali ThreatStream XDR platform, and vice versa, follow these steps:

Go to Threat Exchange > Sharing and click Add Sharing Configuration.
Select the Source Configuration (Netskope CTE), the Business Rule, the Destination Configuration (Anomali ThreatStream XDR), and Target(s). Enter these values:
- URL iType: Select the iType that you want to assign to your URL.
- IP iType: Select the iType that you want to assign to your IP.
- IPv6 iType: Select the iType that you want to assign to your IPv6.
- Domain iType: Select the iType that you want to assign to your Domain.
- Hash [MD5, SHA256] iType: Select the iType that you want to assign to your Hash [MD5, SHA256].
Follow the same steps, but with the configurations reversed, for sharing Anomali ThreatStream XDR IoCs to Netskope. Select the Source Configuration as Anomali ThreatStream XDR, the Business Rule, the Destination Configuration (Netskope CTE), and Target(s). Select the existing IoC List Name, or create a new IoC.
Click Save.

Validate the Anomali ThreatStream XDR Plugin

Validate the Pull

Pulled data will be listed on the Threat Exchange Threat IoCs page. You can filter the IoCs pulled from the platform using the filter: sources.source, like <plugin name>.

To verify pulled logs on CE, go to Logging and search logs for the CTE Anomali ThreatStream XDR plugin.

To verify the data available for pulling on Anomali ThreatStream XDR, log in to Anomali ThreatStream XDR and go to Analyze > Observables.

The plugin pulls IoCs seen on the Observables page.

Validate the Push

To validate the push in CE, go to Logging and filter shared logs for the Anomali plugin.

You will receive a summary log like below by the end of a pull cycle:

CTE Anomali ThreatStream XDR [CTE Anomali ThreatStream XDR]: Total indicator(s) fetched 545, 6 SHA256, 150 MD5, 108 URL, 15 Domain, 259 IP, and 7 IPv6 indicator(s) fetched. skipped 0 indicator(s), total 0 tag(s) skipped.

Go to Threat IoCs and filter logs shared with Anomali plugin.

Note

The Anomali has a limit to ingest 10MB data in one push cycle, so if the data to be ingested is more than 10MB it will be divided in 10MB chunks to push.

On the Anomali platform, go to Analyze > Observables > IoC. Pushed IoCs from CE will have netskope-ce added as a tag.

Troubleshooting

Unable to pull IoCs from the Anomali ThreatStream platform

After the plugin configuration if the IoCs are not pulled from the platform it might be due to one of the following.

No IoCs are available on the platform to pull.
IoCs are not available for the given time range or does not match the configuration parameters.

What to do: Identity your root cause from above and follow below steps to resolve the issue.

No IoCs are available on the platform to pull

Check if the IoCs are available on the platform to pull. If available, check the resolution for the next point.

IoCs are not available for the given time range

If the IoCs are available on the platform to pull, but the plugin has not pulled the IoCs in CE, check the number of days mentioned in the initial range parameter of the plugin configuration. On the Anomali platform check if you have data for the given time range.

If the data is still available for the given time range it might be possible that the IoCs for the provided filter in the plugin configuration are not available, so check the values from the plugin configuration parameter and filter the same on the Anomali platform.

Also make sure that you have Yes selected in the enable polling, to pull the IoCs.

Unable to push the IoCs to Anomali

If you are not able to push the IoCs on the platform and receive an error while pushing it might be due to insufficient permission to the user.

IoCs are pushed from CE but not available on the Anomali platform

If IoCs are pushed and not reflected on the platform it might be due to the IoCs count being too big, when data is shared to Anomali it sometimes takes time to reflect on the platform, at max 10MB data can be shared in a page at a time.

What to do: The pushed data will be reflected but it might take some time, sometimes minutes, sometimes hours, so it suggested waiting for the data to be reflected.

Known Behavior

Following are the things that have been noticed while verifying the plugin workflow.

Sometimes multiple IoCs for a single record are created on the Anomali ThreatStream platform after the IOCs are shared.
Sometimes ingested IoCs take time to reflect on the Anomali platform after IoCs are pushed, this might take minutes or in some cases hours.
Unknown severity from CE is not mapped in Anomali so when IoCs are shared with Unknown severity the platform calculates its own severity and assigns it to the IoC.
The iType for the IoCs won’t be updated if the IoCs are reshared with updated iType.

Anomali ThreatStream XDR Plugin for Threat Exchange