This document explains how to configure the Anomali ThreatStream XDR v1.3.2 plugin for the Threat Exchange module of the Netskope Cloud Exchange platform. This plugin is used to fetch the indicators of type URL, IP (IPv4, IPv6), Domain, SHA256, and MD5 from the Observables on Anomali ThreatStream XDR. This plugin supports sharing MD5, SHA256, URL, Domain, IPv4 and IPv6 to Observables on the Anomali ThreatStream XDR platform using the Share Indicators action.
Prerequisites
To complete this configuration, you need:
- A Netskope tenant (or multiple, for example, production and development/test instances).
- A File Profile on your Netskope tenant.
- A URL List on your Netskope tenant.
- A Netskope Cloud Exchange tenant with the Tenant plugin and Threat Exchange plugin already configured.
- An Anomali Threatstream Cloud Platform or On-Premises instance.
- Connectivity to the following hosts:
- For Cloud Platform : https://api.threatstream.com.
- For On-Premises: https://<ThreatStream_On-Premises_IP_or_FQDN>.
Anomali Threatstream XDR Plugin Supports
The plugin is used to pull and push IoCs of type URL, IPv4, IPv6, Domains, MD5, and SHA256 from/to Anomali ThreatStream XDR’s Observables.
| Fetched Indicator Types | Shared Indicator Types |
|---|---|
| URL, Domains, IP(IPv4), IPv6, SHA256, MD5 | URL, Domains, IP(IPv4), IPv6, SHA256, MD5 |
IoC Retraction
Anomali Threatstream XDR plugin supports pull retraction. Pull retraction will be based on the Retraction Interval field under configuration parameters and configured plugin filters. This means the IoCs present under the Retraction Interval range will not be marked as retracted and rest of the IoCs will be marked as retracted in Cloud Exchange.
IoC Retraction (Pull): Indicators will be fetched from Anomali and in the subsequent pull cycles. If some indicators are deleted from Anomali, they will be marked as Retracted in Cloud Exchange.
| Type | Description |
|---|---|
| IoC Retraction (Pull) | Yes |
| IoC Retraction (Push) | No |
IoC Source Labeling
Indicator shared to the third party will have a field describing the source plugin which actually fetched this IoC.
| Type | Description |
|---|---|
| IoC Source Labelling | Supported (Netskope CE | <Source Plugin>) |
Mappings
Severity Mappings
| Netskope Severity | Anomali ThreatStream XDR Severity |
|---|---|
| low | low |
| medium | medium |
| high | high |
| critical | very-high |
| unknown | – |
Indicator Mappings for Pull and Push
| Netskope Fields | Anomali ThreatStream XDR Observable Fields |
|---|---|
| value | value |
| type | type |
| firstSeen | created_ts |
| lastSeen | modified_ts |
| reputation | confidence |
| severity | meta.severity |
| comments | description |
| tags | tags.name |
Reputation-Confidence Mappings
| Netskope Reputation | Anomali ThreatStream XDR Confidence |
|---|---|
| 1 | 1-10 |
| 2 | 11-20 |
| 3 | 21-30 |
| 4 | 31-40 |
| 5 | 41-50 |
| 6 | 51-60 |
| 7 | 61-70 |
| 8 | 71-80 |
| 9 | 81-90 |
| 10 | 91-100 |
Push Indicators Mappings
| Anomali ThreatStream XDR Observable Field | Netskope Indicator Field |
|---|---|
| url for URL IoC Type domain for Domain IOC type scrip for IPv4 IoC type ipv6 for IPV6 IoC Type md5 for MD5 and SHA256 IoC Types | value |
| subtype(For SHA256 and MD5 only) | SHA256 for SHA256 MD5 for MD5 |
| itype | ITypes selected for their Respective IOC type below are the Action parameters used for each IOC Type: URL iType for URLs IP iType for IPv4 IPv6 iType for IPv6 Domain iType for domains Hash [MD5, SHA256] iType for SHA256 and MD5 |
| severity | Severity very-high if Critical high if high medium -> medium low -> low unknown -> “” |
| confidence | reputation |
| description | comments |
| tags.name | Tags + “netskope-ce” + “Netskope CE | <Source Plugin Name>” |
Performance Matrix
This reading is conducted on a Large Cloud Exchange Stack with these specifications by pulling and pushing 100K IoCs.
| Description | Specification |
|---|---|
| Stack details | Size: Large RAM: 32 GB CPU: 16 Cores |
| Indicators fetched from Anomali ThreatStream XDR | ~33K per minute |
| Indicators shared with Anomali ThreatStream XDR | ~180K per minute |
Permissions
These permissions are needed for the plugin configuration.
- Approval Intel
- Show API for Users
API Details
List of APIs used
| API Endpoint | Method | Use Case |
|---|---|---|
| /api/v2/intelligence | GET | To pull indicators and check for modified indicators |
| /api/v2/intelligence/ | PATCH | To push indicators |
Pull indicators and check for modified indicators
API Endpoint: /api/v2/intelligence
Method: GET
Parameters:
| Key | Value |
|---|---|
| limit | 1000 |
| modified_ts__gt | 2024-01-23T13:33:40.314Z |
| update_id__gt | 0 |
| order_by | update_id |
| remote_api | true |
| type | domain,ip,ipv6,hash,url |
| hash$subtype(For SHA256 and MD5) | MD5,SHA256 |
| confidence__gte | 1 |
| status | active |
| meta.severity | low,medium,high,very-high |
| tags | tag1, tag2 |
| feed_id | 0,1 |
Headers:
| Key | Value |
|---|---|
| Authorization | apikey <username>:<password> |
| Content-Type | application/json |
| Accept | application/json |
| User-Agent | netskope-ce-6.0.1-cte-anomali-threatstream-xdr-v1.3.2 |
Sample API Response:
{
“objects”: [
{
“source”: “Botscout BOT IPs”,
“threatscore”: 17,
“threat_type”: “bot”,
“trusted_circle_ids”: [
146,
211,
388
],
“description”: null,
“workgroups”: [],
“sort”: [
17729579514
],
“resource_uri”: “/api/v2/intelligence/60654060239/”,
“modified_ts”: “2023-10-18T11:11:57.115Z”,
“update_id”: 17729579514,
“source_reported_confidence”: 65,
“type”: “ipv6”,
“uuid”: “39a5d9b0-c48e-4831-afdb-511f57c039ec”,
“feed_id”: 141,
“retina_confidence”: -1,
“created_ts”: “2023-10-18T10:52:26.484Z”,
“id”: 60654060239,
“value”: “2401:4900:171:609:11:98:550:76”,
“itype”: “bot_ipv6”,
“org”: “”,
“confidence”: 65,
“expiration_ts”: “2024-01-16T10:52:23.000Z”,
“owner_organization_id”: 2,
“meta”: {
“severity”: “medium”,
“detail2”: “imported by user 668”
},
“is_anonymous”: false,
“is_public”: false,
“asn”: “”,
“status”: “active”,
“tags”: [
{
“id”: “1yx”,
“name”: “childfirstbehaviortherapy.seo”
},
{
“id”: “3t7”,
“name”: “childfirstbehaviortherapy.seo@gmail.com”
}
],
“can_add_public_tags”: false,
“subtype”: null,
“tlp”: null,
“created_by”: null,
“rdns”: null,
“is_editable”: false
}
],
“meta”: {
“offset”: 0,
“limit”: 1,
“total_count”: 1499,
“next”: “/api/v2/intelligence/?limit=1&modified_ts__gt=2023-04-08T19%3A46%3A42.345Z&order_by=update_id&status=active&type=ipv6&update_id__gt=0&search_after=17729579514”,
“took”: 34
}
}
Push Indicators
API Endpoint: /api/v2/intelligence/
Method: PATCH
Request Body:
{
"meta": {
"classification": "private",
"allow_unresolved": true,
"allow_update": true,
"enrich": false
},
"objects": [
{
"url": "https://185.67.81.10",
"severity": "high",
"itype": "compromised_url",
"type": "url",
"tags": [
{
"name": "Netskope CE | MISp",
"tlp": "white"
},
{
"name": "netskope-test",
"tlp": "white"
}
]
}
]
}
Headers:
| Key | Value |
|---|---|
| Authorization | apikey <username>:<password> |
| Content-Type | application/json |
| Accept | application/json |
| User-Agent | netskope-ce-6.0.1-cte-anomali-threatstream-xdr-v1.3.2 |
Sample API Response:
202 Accepted
User Agent
netskope-ce-6.0.1-cte-anomali-threatstream-xdr-v1.3.2
Workflow
- Get your Anomali ThreatStream XDR credentials.
- Configure the Anomali ThreatStream XDR plugin.
- Configure a Business Rule for Anomali ThreatStream XDR.
- Configure Sharing between Netskope and Anomali ThreatStream XDR.
- Validate the Anomali ThreatStream XDR plugin.
Watch a Video
Click play to watch a video.
Get your Anomali ThreatStream XDR Credentials
Contact your Anomali Support team to get your Base URL, Username, and API Key.
Configure the Anomali ThreatStream XDR Plugin
- In Cloud Exchange, go to Settings > Plugin Store.
- Search for and select the Anomali ThreatStream XDR v1.3.2 (CTE) plugin box.
- Enter the Basic Information:
- Configuration Name: Unique name for the configuration.
- Sync Interval: Leave default.
- Aging Criteria: Expiry time of the plugin in days (Default: 90).
- Override Reputation: Set a value to override the reputation of indicators received from this configuration.
- Enable SSL Validation: Enable SSL Certificate validation.
- Click Next and enter the Configuration Parameters:
- Base URL: The ThreatStream API Base URL based on your Cloud or On-Premises instance.
- Username: ThreatStream platform username (used while logging to platform).
- API Key: API Key fetched from the ThreatStream Anomali XDR platform.
- Remote Observables: Select Yes or No to retrieve the Remote Observables.
- Type of Threat Data to pull: Select based on the data you want to pull. The plugin supports type and subtype filtering from Anomali for the IoC types supported in Cloud Exchange.
- Minimum Confidence: Provide value for confidence(in range of 1-100) IoCs whose confidence is greater than or equal to the provided value will be pulled.
- Severity: IoCs with provided severity will be pulled.
- Status: IoCs with matching Status will be fetched.
- Tags: Indicators matching the Tags will be pulled, enter single or multiple comma separated Tag names. Keep blank to pull IoCs with all Tags.
- Feed ID: Indicators matching the Feed ID will be pulled, enter single or multiple comma separated numeric values for Feed ID. Keep blank to pull IoCs from all feeds.
Note
- When multiple Tags or Feed IDs are added, the plugin will pull data from each mentioned tag using the OR operation.
- If you have added both Tags and Feed ID in the plugin for pulling the data, the IoCs will be pulled using AND operation between the tags and Feed ID, i.e. IoCs will only be pulled if the specified Feed ID has the mentioned tags.
- Enable Polling: Keep Yes to pull data, Keep No if plugin is used for pushing.
- Enable Tagging: Keep Yes to pull tags along with all IoCs from the platform.
- Retraction Interval: Specify the number of days for which IoC retraction should be run for Anomali ThreatStream XDR indicators. Note: This parameter is applicable only for Cloud Exchange version 5.1.0 or later, and if IoC(s) Retraction is enabled in Threat Exchange Settings.
- Initial Range: Number of days Threat IoCs to pull from initial range.
- Click Save. Your plugin configuration is shown in Plugins.
Congfigure a Threat Exchange Business Rule for Anomali ThreatStream XDR
To share indicators fetched from the Anomali ThreatStream XDR to the Cloud Exchange and vice versa, you will need to have a business rule that will filter out the indicators that you want to share.
To configure a business rule:
- In Threat Exchange, go to Business Rules and click Create New Rule.
- Add the filter according to your requirement in the rule, and click Save.
Configure Sharing for Netskope and Anomali ThreatStream XDR
To share IoCs from Cloud Exchangeto the Anomali ThreatStream XDR platform, or vice versa, follow these steps:
- In Threat Exchange, go to Sharing and click Add Sharing Configuration.
- Select your Source Configuration (Netskope CTE), Business Rule, Destination Configuration (Anomali ThreatStream XDR), and Target.
- Select these parameters:
- URL iType: Select the iType that you want to assign to your URL.
- IP iType: Select the iType that you want to assign to your IP.
- IPv6 iType: Select the iType that you want to assign to your IPv6.
- Domain iType: Select the iType that you want to assign to your Domain.
- Hash[MD5, SHA256] iType: Select the iType that you want to assign to your Hash [MD5, SHA256].
- Follow the same steps but vice versa for sharing Anomali ThreatStream XDR IoCs to Netskope select your Source Configuration as Anomali ThreatStream XDR, Business Rule, Destination Configuration(Netskope CTE), and Target. Refer to the Netskope plugin guide for more details.
- Click Save.
Validate the Anomali ThreatStream XDR Plugin
Validate the Pull
Pulled data will be listed on the Threat IoCs page. You can filter the IoCs pulled from the platform using the filter: sources.source Like “<plugin configuration name>”.
To verify pulled logs in Cloud Exchange, go to Logging and search logs from the CTE Anomali ThreatStream XDR plugin.
To verify the data available for pulling in Anomali ThreatStream XDR, go to Analyze > Observables.
To verify the Retracted IoCs, check the logs for IoC Retraction example: message Like CTE Anomali ThreatStream XDR [CTE Anomali ThreatStream XDR] [Retraction]:
You can filter the retracted IoCs from the platform using the filter: sources.source Is equal “<plugin configuration name>” && sources.retracted Is equal true
Note
The IoCs that fall under the Retraction Interval will be marked as Retracted: Yes in Cloud Exchange.
Here you can see IoCs were added to the URL list on Netskope Tenant.
Then some of the shared IoCs got marked as retracted so it was deleted from the URL list.
Note
Anomali does not support the deletion (retraction) of IoCs on their platform. As a result, any previously shared IoCs will not be deleted (retracted) on Anomali.
Validate the Push
Shared IoCs to Anomali can be verified from logs available at Logging in Cloud Exchange.
Note
The Anomali has a limit to ingest 10MB data in one push cycle, so if the data to be ingested is more than 10MB it will be divided in 10MB chunks to push.
On the Anomali platform, go to Analyze > Observables, IOC. Pushed IoCs from Cloud Exchange will have netskope-ce added as a tag.
Note
All IoCs pushed from Cloud Exchange will have the Visibility as private, like My Organization, and it will not be a part of the Anomali community.
Note
You can also use the IoC source labeling to filter the shared IOCs according to the source plugin. For example to filter shared IoCs which were pulled from Netskope tenant you can use this filter as a tag:
Tag format: “Netskope CE |<source plugin name>”
Troubleshooting the Anomali ThreatStream Plugin
Unable to pull IoCs from the Anomali ThreatStream platform
After the plugin configuration, if the IoCs are not pulled from the platform, it might be due to one of these reasons:
- No IoCs are available on the platform to pull.
- IoCs are not available for the given time range or does not match the configuration parameters.
What to do: Identity your root cause noted above, and then follow one of the next two sections to resolve the issue.
No IoCs are available on the platform to pull
Check if the IoCs are available on the platform to pull. If available, check the resolution mentioned in the next section.
IoCs are not available for the given time range
If the IoCs are available on the platform to pull, but the plugin has not pulled the IoCs in Cloud Exchange, check the number of days mentioned in the initial range parameter of the plugin configuration. On the Anomali platform, check if you have data for the given time range.
If the data is still available for the given time range, it might be possible that the IoCs for the provided filter in the plugin configuration are not available, so check the values from the plugin configuration parameter and filter the same on the Anomali platform.
Also, make sure that you have Yes selected in the enable polling to pull the IoCs.
Unable to push the IoCs to Anomali
If you are not able to push the IoCs on the platform and receive an error while pushing, it might be due to insufficient permissions for the user.
IoCs are pushed from CE but not available on the Anomali platform
If IoCs are pushed and not reflected on the platform, it might be due to the IoCs count being too big. When data is shared to Anomali, it sometimes takes time to reflect on the platform. At the most 10MB data can be shared in a page at a time.
What to do: The pushed data will be reflected, but it might take some time, sometimes minutes, sometimes hours, so we suggest waiting for the data to be reflected.
Known Behaviors
Following are the things that have been noticed while verifying the plugin workflow.
- Sometimes multiple IoCs for a single record are created on the Anomali ThreatStream platform after the IoCs are shared.
- Sometimes ingested IoCs take time to reflect on the Anomali platform after IoCs are pushed. This might take minutes or in some cases hours.
- Unknown severity from Cloud Exchange is not mapped in Anomali, so when IoCs are shared with Unknown severity, the platform calculates its own severity and assigns it to the IoC.
- The iType for the IoCs won’t be updated if the IoCs are reshared with updated iType.
- Sometimes if an IoC with a matching value already exists on the Anomali ThreatStream platform as an expired indicator, the platform does not modify or update the existing IoC, even when a successful API call is made to push the same IoC.
- Certain URLs shared via API calls may not be displayed in the Anomali ThreatStream UI, despite being accepted by the API. This could be due to additional filtering or validation applied on the platform’s end.
- Anomali API provides multiple IoC objects in the API response, and in some cases, the IoC values are the same, but some other parameters like severity, reputation, etc. are different.