ThreatConnect Plugin for Threat Exchange

This document explains how to configure the v1.2.0 ThreatConnect plugin with the Threat Exchange module of the Netskope Cloud Exchange platform. The ThreatConnect plugin fetches indicators of types File (MD5 and SHA256), and URL, Host and Address (IPv4 and IPv6) from Intelligence Requirements, Indicators available on ThreatConnect platform. The plugin also supports sharing the Netskope CE indicators using the Add to Group action on the ThreatConnect’s Groups under Intelligence Requirements.

Prerequisites

To complete this configuration, you need:

Compatibility

Netskope CE: v4.2.0, v5.0.1

ThreatConnect Plugin Support

This plugin is used to pull IoCs of types File (MD5 and SHA256), URL, Host and Address (IPv4 and IPv6) from the Intel Requirements from ThreatConnect. This plugin also supports pushing File (MD5 and SHA256), URL, Host and Address (IPv4 and IPv6) to ThreatConnect’s Intel Requirements. This plugin supports performing actions such as Add to Group.

Fetched indicator types	URL, Host, Address (IPv4, IPv6) SHA256, MD5
Shared indicator types	URL, Host, Address (IPv4, IPv6) SHA256, MD5

Permissions

To create an API user make sure you have the organization administrator role of the Organization Administrator for creating an API User for getting the Access ID and the Secret Key.

To pull and push data from/to ThreatConnect make sure your user has the below rights

Functionality	API Endpoint	Request Type	Permissions
Pull Indicators	/api/v3/indicators?	GET	Select any one of the following: – App Developer – Organization Administrator – Read Only Commenter – Read Only User – Sharing User – Standard User
List Groups	/api/v3/groups/	GET	Select any one of the following: – App Developer – Organization Administrator – Read Only Commenter – Read Only User – Sharing User – Standard User
Create Group	/api/v3/groups/	POST	Select any one of the following: – App Developer – Organization Administrator – Sharing User – Standard User
Push Indicators	/api/v3/indicators	POST	Select any one of the following: – App Developer – Organization Administrator – Sharing User – Standard User
Get Owner	/api/v2/owners/mine	GET	Select any one of the following: – App Developer – Organization Administrator – Read Only Commenter – Read Only User – Sharing User – Standard User

API Details

List of APIs used

API Name	Method	API Endpoint
Pull Indicators	GET	/api/v3/indicators?
List Groups	GET	/api/v3/groups/
Create Group	POST	/api/v3/groups/
Push Indicators	POST	/api/v3/indicators
Get Owners Mine	GET	/api/v2/owners/mine
Get Owners	GET	/api/v3/security/owners
Update Indicators	PUT	/api/v3/indicators/{value}

Pull Indicators

API Endpoint: /api/v3/indicators

Method: GET

Headers:

Key	Value
User-Agent	netskope-ce-5.0.1-cte-threatconnect-v1.2.0
Authorization	TC <Token>
Timestamp	Unix Epoch of current date and time.

Query Parameters:

Parameter	Value
typeName IN	(“File”, “URL”, “Host”, “Address”,)
lastModified	String representation of Datetime in %Y-%m-%dT%H:%M:%SZ Format.
sorting	lastModified asc
fields	tags
resultStart	0
resultLimit	1000

Sample API Response:

{
    "data": [
        {
            "id": 10,
            "ownerId": 1,
            "ownerName": "Demo Organization",
            "dateAdded": "2021-11-02T13:07:08Z",
            "webLink": "https://app.threatconnect.com/#/details/indicators/10/overview",
            "type": "File",
            "lastModified": "2021-11-02T14:04:55Z",
            "summary": "F5A2496CF66CB8CFFE66CB1B27D7DEDE",
            "privateFlag": false,
            "active": true,
            "activeLocked": false,
            "md5": "F5A2496CF66CB8CFFE66CB1B27D7DEDE",
            "legacyLink": "https://app.threatconnect.com/auth/indicators/details/file.xhtml?file=F5A2496CF66CB8CFFE66CB1B27D7DEDE&owner=Demo+Organization"
        },
	]
}

List Groups

API Endpoint: /api/v3/groups

Method: GET

Headers:

Key	Value
User-Agent	netskope-ce-5.0.1-cte-threatconnect-v1.2.0
Authorization	TC <Token>
Timestamp	Unix Epoch of current date and time.

Query Parameters:

Parameter	Value
tql	ownerName == <ownerName>
resultLimit	1000

Sample API Response:

{
    "data": [
        {
            "id": 10,
            "ownerId": 1,
            "ownerName": "Demo Organization",
            "dateAdded": "2021-10-21T19:54:59Z",
            "webLink": "https://app.threatconnect.com/auth/document/document.xhtml?document=10",
            "type": "Document",
            "name": "Bad Document",
            "createdBy": {
                "id": 3,
                "userName": "11112222333344445555",
                "firstName": "John",
                "lastName": "Smith",
                "pseudonym": "jsmithAPI",
                "owner": "Demo Organization"
            },
            "upVoteCount":"0",
            "downVoteCount":"0",
            "fileName": "indicators.txt",
            "fileSize": 36,
            "status": "Success",
            "documentType": "Text",
            "documentDateAdded": "2021-10-21T19:54:59Z",
            "lastModified": "2022-03-09T12:44:04Z",
            "legacyLink": "https://app.threatconnect.com/auth/document/document.xhtml?document=10"
        },
	]
}

Create a Group

API Endpoint: /api/v3/groups

Method: POST

Headers:

Key	Value
User-Agent	netskope-ce-5.0.1-cte-threatconnect-v1.2.0
Authorization	TC <Token>
Timestamp	Unix Epoch of current date and time.

Payload:

{
    "type": "Incident",
    "name": "Bad Incident",
    "tags": {
        "data": [
            {
                "name": "Netskope CEa"
            }
        ]
    }
}

Sample API Response:

{
    "data": {
        "id": 3,
        "ownerId": 1,
        "ownerName": "Demo Organization",
        "dateAdded": "2021-11-03T14:57:45Z",
        "webLink": "https://app.threatconnect.com/#/details/groups/3/overview",
        "type": "Incident",
        "name": "Bad Incident",
        "createdBy": {
            "id": 3,
            "userName": "11112222333344445555",
            "firstName": "John",
            "lastName": "Smith",
            "pseudonym": "jsmithAPI",
            "owner": "Demo Organization"
        },
        "upVoteCount":"0",
        "downVoteCount":"0",
        "status": "New",
        "eventDate": "2021-11-03T00:00:00Z",
        "lastModified": "2021-11-03T14:57:4511:04:12Z",
        "legacyLink": "https://app.threatconnect.com/auth/incident/incident.xhtml?incident=3"
    },
    "message": "Created",
    "status": "Success"
}

Push Indicators

API Endpoint: /api/v3/indicators

Method: POST

Headers:

Key	Value
User-Agent	netskope-ce-5.0.1-cte-threatconnect-v1.2.0
Authorization	TC <Token>
Timestamp	Unix Epoch of current date and time.

Payload:

{
    "type": "Host",
    "hostName": "ultrabadguy.com",
    "associatedGroups": {
        "data": [
            {
                "id": 12
            }
        ]
    },
    "confidence": 85,
    "rating": 5,
    "tags": {
        "data": [
            {
                "name": "Netskope CE"
            }
        ]
    }
}

Sample API Response:

{
    "data": {
        "id": 4,
        "ownerId": 1,
        "ownerName": "Demo Organization",
        "dateAdded": "2021-11-05T16:43:17Z",
        "webLink": "https://app.threatconnect.com/#/details/indicators/4/overview",
        "type": "Host",
        "lastModified": "2021-11-05T16:43:17Z",
        "rating": 5.00,
        "confidence": 85,
        "description": "This host is very dangerous",
        "summary": "ultrabadguy.com",
        "privateFlag": false,
        "active": true,
        "activeLocked": false,
        "hostName": "ultrabadguy.com",
        "dnsActive": true,
        "whoisActive": true,
        "legacyLink": "https://app.threatconnect.com/auth/indicators/details/host.xhtml?host=ultrabadguy.com&owner=Demo+Organization"
    },
    "message": "Created",
    "status": "Success"
}

Get Owners Mine

API Endpoint: /api/v2/owners/mine

Method: GET

Headers:

Key	Value
User-Agent	netskope-ce-5.0.1-cte-threatconnect-v1.2.0
Authorization	TC <Token>
Timestamp	Unix Epoch of current date and time.

Sample API Response

{
  "status": "Success",
  "data": {
    "resultCount": 1,
    "owner": {
      "id": 1,
      "name": "Demo Organization",
      "type": "Organization"
    }
  }
}

Get Owners

API Endpoint: /api/v3/security/owners

Method: GET

Headers: 

Key	Value
User-Agent	netskope-ce-5.0.1-cte-threatconnect-v1.2.0
Authorization	TC <Token>
Timestamp	Unix Epoch of current date and time.

Sample API Response:

{
	"data": [
		{
            "id": 1,
            "name": "Demo Organization",
            "type": "Organization",
            "ownerRole": "Organization Administrator",
            "permIndicator": "FULL",
            "permGroup": "FULL",
            "permPost": "FULL",
            "permTrack": "FULL",
            "permVictim": "FULL",
            "permAttribute": "FULL",
            "permApps": "BUILD",
            "permUsers": "FULL",
            "permSecurityLabel": "FULL",
            "permTag": "FULL",
            "permAttributeType": "FULL",
            "permSettings": "FULL",
            "permMembers": "READ",
            "permCopyData": "FULL",
            "permInvite": "FULL",
            "permTask": "FULL",
            "permCaseTag": "FULL",
            "permArtifact": "FULL",
            "permComment": "FULL",
            "permTimeline": "FULL",
            "permWorkflowTemplate": "FULL",
            "permPublish": "FULL",
            "permPlaybooksExecute": "FULL",
            "permPlaybooks": "FULL"
        },
	]
}

Update Indicator

API Endpoint: /api/v3/indicators/{value}

Method: PUT

Headers

Key	Value
User-Agent	netskope-ce-5.0.1-cte-threatconnect-v1.2.0
Authorization	TC <Token>
Timestamp	Unix Epoch of current date and time.

Payload:

{
	"associatedGroups": {
		"data": [
			{"id": group_id},
		],
		"mode": "append",
	},
}

Sample API Response:

{
    "data": {
        "id": 4,
        "ownerId": 1,
        "ownerName": "Demo Organization",
        "dateAdded": "2021-11-05T16:43:17Z",
        "webLink": "https://app.threatconnect.com/#/details/indicators/4/overview",
        "type": "Host",
        "lastModified": "2021-11-05T17:21:06Z",
        "rating": 5.00,
        "confidence": 92,
        "description": "This host is very dangerous",
        "summary": "ultrabadguy.com",
        "privateFlag": false,
        "active": true,
        "activeLocked": false,
        "hostName": "ultrabadguy.com",
        "dnsActive": false,
        "whoisActive": true,
        "legacyLink": "https://app.threatconnect.com/auth/indicators/details/host.xhtml?host=ultrabadguy.com&owner=Demo+Organization"
    },
    "message": "Updated",
    "status": "Success"
}

Performance Matrix

Here is the performance reading conducted for fetching and sharing 100K IoCs in each plugin lifecycle on a Large CE instance with these specifications.

Stack details	Size: Large RAM: 32 GB CPU: 16 Cores
Indicators fetched from ThreatConnect	~14K per minute
Indicators shared to ThreatConnect	~5K per minute

User Agent

The user agent added for this plugin is in the following format: netskope-ce-<CE VERSION>-<MODULE NAME>-<PLUGIN NAME>-v<PLUGIN VERSION>.

For example: netskope-ce-5.0.1-cte-threatconnect-v1.2.0.

Mappings

Default tags mapping for the Indicators pulled from the ThreatConnect platform.

Tag Mapping in Cloud Exchange based on IoC Type

Type of Indicator pulled from ThreatConnect	Tag added in Netskope CE for each type of indicator
URL	ThreatConnect-URL
Address(IPV4)	ThreatConnect-Address-IPV4
Address(IPV6)	ThreatConnect-Address-IPV6
Host	ThreatConnect-Host
File(MD5)	ThreatConnect-File-MD5
File(SHA256)	ThreatConnect-File-SHA256

Mappings for Pull

Netskope CE Fields	ThreatConnect Fields
Value	md5, sha256, text(URL), ip(Address), hostName(host)
Active	active
Severity	rating
Reputation	confidence
Comments	description
Firstseen	dateAdded
Lastseen	lastModified
Tags	Tags

Mappings for Push

Netskope CE Fields	ThreatConnect Fields
value	md5, sha256, text (URL), ip (Address), hostName (host)
type	url, File, Host, Address
Fetched existing_group_id	associatedGroups
Netskope CE	tags
severity	rating
reputation	confidence

Severity Mappings

Netskope CE Severity	ThreatConnect Severity
Unknown	0
Low	1
Low	2
Medium	3
High	4
Critical	5

Workflow

Click play to watch a video.

Validate the ThreatConnect Plugin

Validate the Pull

Threat Exchange pulls Address (IPv4, IPv6), Files (SHA256, MD5), URLs, and Hosts from Browse > Indicators in the ThreatConnect plugin.

Go to Threat Exchange > Threat IoCs and search for IoCs pulled from the ThreatConnect plugin.


The Host or domain values will be considered as Host on ThreatConnect, and will be stored as hostname in CE.


Go to Logging and search for IoCs pulled from the ThreatConnect plugin.

Validate the Push

To verify pushed IoCs in ThreatConnect, Go to ThreatConnect Platform > Browse > Indicators.
IoCs pushed from Netskope will be tagged with Netskope.

Go to Browse > Groups. Select the Group type under which you have your group added. Select your group name and check the shared data.

Data shared in the Group will be appended and not replaced.
To validate the pushed indicators on Threat Exchange, go to Threat IoCs and search for IoCs that are shared with ThreatConnect.

Troubleshooting

Unable to Pull data from the ThreatConnect platform

If you are not able to pull data from the ThreatConnect platform, it might be due to one of the following things.

• Data is not available on the platform to pull.
• Data is available on platform, but not in the given time range provided in the plugin configuration.

What to do: If data is not pulled from the plugin, first check if you have any data available on the ThreatConnect platform. If you have data to pull, check the date for the available data and check if the data falls under the time interval mentioned in the plugin configuration.

Unable to push IoCs to ThreatConnect

When sharing IoCs to ThreatConnect if you are receiving any error messages, check the error logs.
CTE ThreatConnect [CTE ThreatConnect]: Received exit code 400, Bad Request while Pushing indicator on ThreatConnect platform having indicator value: 1bfbefa4ff4d0df3ee0090b5079cf84ed2e8d5377ba5b7a30afd88367d57b9ff.

What to do: The IoCs that were not shared will be added in the error log as shown above. All IOCs that will be a part of the exclusion list on ThreatConnect will not be shared along with the IOCs that are already present on the ThreatConnect platform. To confirm if the IOC is not shared because of the exclusion list or being already present, go to ThreatConnect > Create. Select the type of IOC that was failed to be pushed and try adding it.

Receiving the 400 Client Error in logs while executing the plugin life cycle

If you are receiving the above-mentioned error in logs while pulling indicators from the ThreatConnect platform, try increasing the value of the Custom TQL Timeout.

You can increase the TQL Timeout value using the following steps:

1. Log in to your ThreatConnect account.
2. Go to Settings > Org Settings.

   

3. Edit the account that you use for generating the configuration parameters and increase the value for the Custom TQL Timeout.

   

Receiving an error while updating the plugin using the plugin repository

If you are facing an issue updating the configured ThreatConnect plugin, follow these steps:

1. Close the plugin repo page once pulling and downloading the plugin updates.
2. Go to Threat Exchange > Plugins.
3. Edit the plugin, on the Configuration Parameters page. Remove the selected value from the Type of Threat Indicator field, and reselect the IoC type that you want to pull.
4. Click Save.
5. Click on the enable plugin icon and enable the plugin. The plugin will be updated with the latest changes and should start working as expected.