Release Notes
1.3.0 (Requires minimum Cloud Exchange version 6.0.0)
Added
- Added support for Custom Region and Externally Provisioned Bucket in the configuration parameters.
- Added resolutions for error logs.
- Added the JSON format option.
Changed
- Updated the object file name format to <epoch>_<thread_id>_<uuid>.gz to prevent data overwrite when data is ingested at the exact same time.
Fixed
- Fixed the error observed while sharing WebTx data using NLS.
- Fixed proxy settings not being applied to AWS IAM Roles Anywhere requests.
1.2.0
Added
- Added two new authentication methods.
Removed
- Removed secret credentials for authentication.
Changed
- Updated object structure in push functionality.
1.1.0
Changed
- Added support for a few regions.
- Changed the plugin name
1.0.0
Added
- Initial release.
This document explains how to configure the AWS S3 WebTx v1.3.0 plugin with the Log Shipper module of the Netskope Cloud Exchange platform. This plugin supports ingestion of WebTx data to an AWS S3 bucket. This plugin supports ingestion in JSON format only.
Prerequisites
To complete this configuration, you need:
- A Netskope tenant (or multiple, for example, production and development/test instances).
- A Netskope Cloud Exchange tenant with the Tenant plugin and Log Shipper plugin already configured.
- A Netskope Cloud Exchange tenant with the AWS Netskope LogStreaming or Azure Netskope LogStreaming plugin already configured, for ingesting WebTx from the Netskope Log Streaming plugins.
- Amazon S3 bucket permissions to the IAM user.
- ListBucket
- CreateBucket
- GetBucketLocation
- PutObject
- To use a Secrets Manager, refer to this documentation. To set up Azure Key Vault, refer to this documentation.
- Connectivity to the following hosts: AWS S3 Bucket access.
Note
Verify your bucket permissions are secure and not set up for open public access. Only allow access to your cloud storage instance from your Cloud Exchange Host and any other addresses that need access.
AWS S3 WebTx Plugin Support
The AWS S3 WebTx plugin support is:
| Data Type | Support |
|---|---|
| Events | No |
| Alerts | No |
| WebTx | Yes (via Netskope Log Streaming) |
Note
- CLS WebTX based on Google Pub Sub Lite is deprecated. Please refer to Netskope Product EOL/EOS Announcements – Netskope Knowledge Portal
- For ingesting WebTX logs to your Log delivery destinations like SIEM, SOAR, XDR, Data Lake, use the AWS Netskope LogStreaming or Azure Netskope LogStreaming plugin.
Permissions
The IAM user needs these Amazon S3 bucket permissions to send WebTx (via Netskope LogStreaming) data to buckets:
- ListBucket
- CreateBucket
- GetBucketLocation
- PutObject
API Details
List of APIs Used
This plugin uses Python libraries to create file objects in AWS S3.
Library: The AWS SDK for Python (Boto3)
Usage: The AWS SDK for Python (Boto3) to create, configure, and manage AWS services, such as Amazon Elastic Compute Cloud (Amazon EC2) and Amazon Simple Storage Service (Amazon S3). The SDK provides an object-oriented API as well as low-level access to AWS services.
Create a Client
s3_client = boto3.client(
"s3",
aws_access_key_id=self.aws_access_key_id,
aws_secret_access_key=self.aws_secret_access_key,
aws_session_token=self.aws_session_token,
region_name=get_effective_region(self.configuration),
verify=self._get_verify(),
config=Config(
proxies=self._get_proxies() or None,
user_agent=self.user_agent,
connect_timeout=CONST.BOTO_CONNECT_TIMEOUT,
read_timeout=CONST.BOTO_READ_TIMEOUT,
retries=dict(
total_max_attempts=CONST.BOTO_MAX_ATTEMPTS,
mode=CONST.BOTO_RETRY_MODE,
),
),
)
Create an STS Client
sts_client = boto3.client(
"sts",
aws_access_key_id=self.aws_access_key_id,
aws_secret_access_key=self.aws_secret_access_key,
aws_session_token=self.aws_session_token,
region_name=get_effective_region(self.configuration),
verify=self._get_verify(),
config=Config(
proxies=self._get_proxies() or None,
user_agent=self.user_agent,
connect_timeout=CONST.BOTO_CONNECT_TIMEOUT,
read_timeout=CONST.BOTO_READ_TIMEOUT,
retries=dict(
total_max_attempts=CONST.BOTO_MAX_ATTEMPTS,
mode=CONST.BOTO_RETRY_MODE,
),
),
)
Generate Temporary Credentials (AWS IAM Roles Anywhere Only)
response = requests.post(
"https://rolesanywhere.<region>.amazonaws.com/sessions",
data=body,
headers=headers,
proxies=self.proxy,
verify=self.verify,
timeout=CONST.ROLES_ANYWHERE_TIMEOUT,
)
Request Headers
headers = {
"Content-Type": CONST.CONTENT_TYPE,
"X-Amz-Date": amz_date,
"X-Amz-X509": amz_x509,
"Authorization": authorization,
"User-Agent": self.user_agent,
}
Request Body
{
"durationSeconds": 900,
"profileArn": "<Profile ARN>",
"roleArn": "<Role ARN>",
"sessionName": "Session",
"trustAnchorArn": "<Trust Anchor ARN>"
}
Response
{
"credentialSet": [
{
"credentials": {
"accessKeyId": "<accessKeyId>",
"secretAccessKey": "<secretAccessKey>",
"sessionToken": "<sessionToken>",
"expiration": "<UTC time, e.g. 2026-10-21T10:15:00Z>"
}
}
]
}
Create a Bucket
s3_client.create_bucket(Bucket=bucket_name) # us-east-1
s3_client.create_bucket(
Bucket=bucket_name,
CreateBucketConfiguration={"LocationConstraint": region_name},
)
Upload a File into the Bucket
s3_client.upload_file(
file_name,
bucket_name,
object_name,
)
Check Bucket Existence and Accessibility (HeadBucket)
s3_client.head_bucket(Bucket=bucket_name)
Verify Bucket Ownership (HeadBucket with ExpectedBucketOwner)
s3_client.head_bucket(
Bucket=bucket_name, ExpectedBucketOwner=own_account_id
)
Get the Caller Identity
sts_client.get_caller_identity()
Verify the Bucket Region
location = s3_client.get_bucket_location(Bucket=bucket_name)
Performance Matrix
This performance reading is for a Large Stack CE tested with these VM specifications.
| Description | Specification |
|---|---|
| Stack details | Size: Large RAM: 32 GB CPU: 16 Cores |
| WebTx (via Netskope LogStreaming) ingested to SIEM | ~170K EPM |
User Agent
APN/1.1 (ahq9d89xj9gspapczzdb59goq)
Workflow
- Create a bucket, a bucket policy, and configure an authentication method.
- Configure the AWS S3 WebTx plugin.
- Configure Log Shipper Business Rules for AWS S3 WebTx.
- Configure Log Shipper Log Delivery for AWS S3 WebTx.
- Validate the AWS S3 WebTx plugin.
Click play to watch a video.
Configure AWS S3
Create a AWS S3 Source Bucket
-
To create a Bucket and set the required policies, go to the AWS console and enter credentials to log in.

-
From All Services, select S3, or search for S3 in the search bar.

-
Click Create Bucket.

-
Enter a Bucket Name (like netskope-ce-source-bucket), scroll to the bottom, and click Create Bucket.

-
Search for your bucket and click on the bucket name to open it.

-
Click Permission to open the permission tab and to set the policy.

-
Click Edit in the Block public access (bucket settings) section. Uncheck all checkboxes and click Save Changes. It will ask for confirmation. Confirm it and click Confirm.

-
Under the Permissions tab in the bucket, click Edit in the Bucket Policy section.

-
Click Policy Generator.

-
Select S3 Bucket Policy as the policy type, add statement details, and generate a policy. Replace the <user-arn> with the user ARN used for accessing the source bucket, and <bucket-name> with the source bucket created above. Example:
<user-arn>: arn:aws:iam::7111xxxxxxxx:user/xxxxxxxx, and<bucket-name>: netskope-ce-source-bucket.- Select Type of Policy: S3 Bucket Policy
- Effect: Allow
- Principal: <user-arn>
- Actions:
- GetBucketAcl
- GetBucketPolicy
- ARN: arn:aws:s3:::<bucket-name>
- Click Add Statement.

- Scroll back up and add another statement.
- Select Type of Policy: S3 Bucket Policy
- Effect: Allow
- Principal: <user-arn>
- Actions:
- PutObject
- PutObjectAcl
- ARN: arn:aws:s3:::<bucket-name>/*
- Click Add Statement.


- Click Generate Policy. Make sure to copy the policy generated in this step as it will be used in the next step to attach in the source bucket.

-
Add this policy to the Textbox, scroll to the bottom, and click Save Changes.

Create a Bucket Policy
A Policy required for accessing the source bucket by the plugin.
-
Go to Policy Generator and Select IAM Policy as the policy type and generate policy.
Replace the <bucket-name> with the AWS S3 Source Bucket created previously.
(like netskope-ce-source-bucket).- Select Type of Policy: IAM Policy
- Effect: Allow
- AWS Service: Amazon S3
- Actions:
- ListAllMyBuckets
- ARN: *
- Click Add Statement.

- Scroll back up to add another statement.
- Select Type of Policy: IAM Policy
- Effect: Allow
- AWS Service: Amazon S3
- Actions:
- GetBucketAcl
- GetBucketPolicy
- GetBucketLocation
- ARN: arn:aws:s3:::<bucket-name>
- Click Add Statement.

- Scroll back up to add another statement
- Select Type of Policy: IAM Policy
- Effect: Allow
- AWS Service: Amazon S3
- Actions:
- PutObject
- PutObjectAcl
- ARN: arn:aws:s3:::<bucket-name>/*
- Click Add Statement.

- Click Generate Policy. Make sure to copy the policy generated in this step, as it will be used in the next step to create a policy that will be attached to the user for accessing the source bucket.

-
Go to AWS Console and select IAM from All Services.

-
Click Policies in the left panel and click Create Policy.

-
Paste the policy created in the previous step to the JSON tab, click Next: Tags, and then click Next: Review.

-
Enter a Name (like netskope-ce-s3-policy) and click Create Policy.

-
Attach this policy to the user, using the following steps:
-
Go to IAM > Users. Select the user for which you want to attach a policy, click Add permissions, and click Add permissions.

-
Select Attach policies directly under Permissions options. Search and select the policy created in the previous step for source bucket (netskope-ce-s3-policy).

-
Click Next and then click Add permissions. The Policy will be attached to the user.

-
Plugin Authentication Methods
IAM Role Anywhere Configuration
Prerequisites
The AWS Certificate Manager service is required to be enabled to authenticate the plugin using the AWS IAM Roles Anywhere Authentication Method.
Create a Policy
This Policy contains the required permissions for creating a Private CA Certificate(including Permissions for creating Trust Anchor and Profile) and using the IAM Roles Anywhere.
-
Go to Policy Generator, select IAM Policy as the policy type, and generate a policy.
- Select Type of Policy: IAM Policy
- Effect: Allow
- AWS Service: AWS Private Certificate Authority
- Actions:
- CreateCertificateAuthority
- DescribeCertificateAuthority
- GetCertificate
- GetCertificateAuthorityCertificate
- GetCertificateAuthorityCsr
- ImportCertificateAuthorityCertificate
- IssueCertificate
- ListCertificateAuthorities
- ARN: *
- Click Add Statement.

- Select Type of Policy: IAM Policy
- Effect: Allow
- AWS Service: AWS Identity and Access Management (IAM)
- Actions:
- AttachRolePolicy
- CreateAccessKey
- CreateRole
- DeleteRole
- PassRole
- ARN: *
- Click Add Statement.

- Select Type of Policy: IAM Policy
- Effect: Allow
- AWS Service: AWS Certificate Manager
- Actions:
- DescribeCertificate
- ExportCertificate
- GetCertificate
- ListCertificates
- ListTagsForCertificate
- RequestCertificate
- ARN: *
- Click Add Statement.

- Select Type of Policy: IAM Policy
- Effect: Allow
- AWS Service: AWS Identity and Access Management Roles Anywhere
- Actions:
- CreateProfile
- CreateTrustAnchor
- GetProfile
- GetTrustAnchor
- ListProfiles
- ListTrustAnchors
- ARN: *
- Click Add Statement.

- Click Generate Policy.

- Copy the Policy as it will be used in the next step for creating the policy required for creating the Private CA certificates.
-
Go to AWS Console and select IAM from All Services. Click Policies in the left panel, and then click Create Policy.

-
Copy the policy to the JSON tab. Click Next: Tags, and then click Next: Review.

-
Enter a Name (like netskope-ce-rolesAnywhere-policy) and click Save Changes.

Create Private Certificate Authority
-
Log in to the AWS Console.
-
Search for Certificate Manager.

-
Click AWS Private CA.
-
Click Create a private CA.

-
For Mode Options, select General-purpose.
-
For CA Type Options, select Root.

-
Enter an Organization (O).

-
For Key Algorithm Options, select RSA 2048.


-
Add tags if any (optional)
-
Check the checkbox in the CA Permissions Options section.
-
Check the checkbox in the Pricing section.
-
Click Create to generate the CA certificate.


-
From Actions, select Install CA certificate.

-
Click Confirm and Install.


Create Trust Anchor
-
Go to IAM Services in the AWS Console and click Roles under Access management. Scroll down to Roles Anywhere and select Manage.

-
Click Create a Trust anchor.

-
Enter a Trust anchor name (like netskope-ce-trust-anchor).

-
Select the AWS Certificate Manager Private CA (created in the previous steps) as a Certificate authority (CA) source.
-
Add tags if required.
-
Click Create a trust anchor.


-
Click on the created Trust Anchor and copy the Trust Anchor ARN.

Create an IAM Role
-
Go to IAM Services in the AWS Console.
-
Click Role under Access management.
-
Click Create Role.

-
For the Trusted entity type, select Custom Trust Policy.

-
Go to Policy Generator.
-
Replace the Custom trust Policy with this Trust Policy.This policy contains the permissions for using the roles anywhere service:
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": [ "rolesanywhere.amazonaws.com" ] }, "Action": [ "sts:AssumeRole", "sts:TagSession", "sts:SetSourceIdentity" ] } ] }
-
Click Next.
-
In the Permissions policies, select the previously created Bucket Policy.

-
Click Next.
-
Provide a Role name (like netskope-ce-roleAnywhere), and add a Description for the role.
-
Click Create role.

-
Make a note of the Role ARN as this will be required in the Plugin configuration parameter: Role ARN for the authentication method AWS IAM Roles Anywhere.

Create a Profile
-
Select Roles under Access management.
-
Scroll down to Roles Anywhere and click Manage.

-
Expand the Setup steps.
-
Click Configure roles.
-
Click Configure a profile.

-
Enter a Profile name (like netskope-ce-profile).
-
Select the role created in Create IAM Role section: netskope-ce-roleAnywhere.

-
Remove the Inline Policy.
-
Click Create profile.



-
Click the created Profile and copy the Profile ARN.

Request a Private Certificate
-
Go to AWS Certificate Manager > Request certificate.
-
Select Request a private certificate.

-
Click Next.
-
Select the Certificate authority created in the previous steps.

-
Provide a domain name in the Fully qualified domain name field (like netskope-ce.com).
-
For Key Algorithm, select RSA 2048.

-
Add tags if required.
-
Acknowledge the Certificate renewal permissions.
-
Click Request.

-
Go to List certificates from the navigation panel of AWS Certificate Manager.
-
Select the certificate created previously.

-
Click Export.

-
Enter the passphrase. Make a note of the passphrase as it will be required for the Configuration of the AWS S3 Plugin using the AWS IAM Roles Anywhere Authentication method.
-
Click Generate PEM Encoding.

-
Download all the Certificates because they won’t be visible again. For new certificates, you will need to Export it again.

For more info, go to AWS IAM Role Anywhere
Deployed on AWS Configuration
Create a Role
-
Go to IAM services in the AWS Console.
-
Click Create role.

-
Select the AWS Service.
-
For Use Case, select EC2.
-
Click Next.

-
Select the permission policy created in your Bucket Policy.
Policy:
{ "Version": "2012-10-17", "Statement": [ { "Sid": "VisualEditor0", "Effect": "Allow", "Action": [ "s3:GetBucketLocation", "s3:PutObject", "s3:ListBucket", "s3:CreateBucket" ], "Resource": "*" } ] } -
Click Next.

-
Enter a Role Name (like netskope-ce-instance-role), and add a Description.
-
Click Create Role.

For this configuration, both the Netskope instance and S3 Bucket should be in the same region.
Assign a Role to an EC2 Instance
-
Open your EC2 instance console.
-
Click Instances under Instances.

-
Go to Action > Security > Modify IAM Role.

-
Select the Role that you created previously (like netskope-ce-instance-role).
-
Click Update IAM Role.

Assign a Role to a Fargate Instance
-
Open your CFN script.
-
Get the ExistingECSTaskRole Parameter value if present.

-
Go to IAM > Roles.
-
Search for the ExistingECSTaskRole parameter value. Otherwise, search for NetskopeCloudExchangeTaskRole-<CFN name> and select the role.

-
Attach the previously created Create Bucket Policy to this role.
-
Click Add permission > Attach policies.

-
Search and select the bucket policy previously created (netskope-ce-s3-policy).
-
Click Add permissions.

Assign the Role to a K8s Instance
-
Open your Role created for ServiceAccount while creating K8s instance.
-
Attach the Bucket policy created previously.
Configure an Externally Provisioned Bucket
If the bucket is in an external AWS account, you work with two accounts:
- Account A (Plugin Account): The account that runs the Cloud Exchange instance (for example, an EC2 or Ubuntu VM that uses IAM Roles Anywhere).
- Account B (Core Account): The account that owns the externally provisioned S3 bucket.
Both policies below are required. AWS evaluates the policy on the Cloud Exchange role and the bucket policy on the external bucket, and the request fails with a 403 (AccessDenied) error if either one is missing.
On Account A (Plugin Account):
-
Go to IAM > Roles > <CE_ROLE_NAME>.
-
Add this policy:
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:ListBucket", "s3:GetBucketLocation" ], "Resource": "arn:aws:s3:::<external bucket name>" }, { "Effect": "Allow", "Action": [ "s3:PutObject" ], "Resource": "arn:aws:s3:::<external bucket name>/*" } ] }s3:ListBucket and s3:GetBucketLocation are bucket-level actions, so the resource is the bucket ARN without /*. s3:PutObject is an object-level action, so the resource is the bucket contents with /*.
On Account B (Core Account):
-
Go to S3 > Bucket > Permissions > Bucket Policy and add this policy to grant the Cloud Exchange role in Account A access:
{ "Version": "2012-10-17", "Statement": [ { "Sid": "AllowCECrossAccountAccess", "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::<ACCOUNT_A_ID>:role/<CE_Account_A_ROLE_NAME>" }, "Action": [ "s3:ListBucket", "s3:GetBucketLocation", "s3:PutObject" ], "Resource": [ "arn:aws:s3:::<bucket name>", "arn:aws:s3:::<bucket name>/*" ] } ] } -
Go to S3 > Bucket > Permissions > Object Ownership, click Edit, and select Bucket owner enforced. Objects uploaded by Account A are then owned by Account B.
Configure the AWS S3 WebTX Plugin
-
In Cloud Exchange, go to Settings > Plugin Store.
-
Search for and select the AWS S3 WebTx v1.3.0 (CLS) plugin.

-
Enter a Configuration Name and keep the Mappings and Format as default. The Format is JSON.
-
Click Next.

-
Enter the Configuration Parameters:
- Authentication Method: Select the method to use for authentication: Deployed on AWS or AWS IAM Roles Anywhere.
- Private Key: PEM-encoded private key used to decrypt the AWS Private CA Certificate. Required only when Authentication Method is AWS IAM Roles Anywhere.
- Certificate Body: PEM-encoded X.509 certificate body issued by your AWS Private or Public CA. Required only when Authentication Method is AWS IAM Roles Anywhere.
- Password Phrase: Passphrase used to decrypt the Private Key. Required only when Authentication Method is AWS IAM Roles Anywhere. Required only if the Private Key is encrypted.
- Profile ARN: ARN of the IAM Roles Anywhere profile. Required only when Authentication Method is AWS IAM Roles Anywhere.
- Role ARN: ARN of the IAM role to be assumed. Required only when Authentication Method is AWS IAM Roles Anywhere.
- Trust Anchor ARN: ARN of the IAM Roles Anywhere trust anchor. Required only when Authentication Method is AWS IAM Roles Anywhere.
- Externally Provisioned Bucket: Select Yes if the S3 bucket is in an external AWS account, or No if it is in your own AWS account. If you select No and the bucket does not exist, the plugin creates it. If you select Yes, the bucket must already exist, because the plugin does not create it.
- AWS S3 Bucket Region Name: AWS region where the AWS S3 Bucket resides. If you use AWS IAM Roles Anywhere, the region must match the region in the Profile ARN and Trust Anchor ARN. Select Custom Region to provide a region that is not in the list.
- AWS Custom Region: AWS region code of the bucket. Required only when Custom Region is selected in AWS S3 Bucket Region Name.
- AWS S3 Bucket Name: AWS S3 Bucket Name in which the WebTx data objects are stored.
The AWS IAM Roles Anywhere parameters appear only after you select AWS IAM Roles Anywhere as the Authentication Method.


-
Click Save. Your plugin configuration appears on the Log Shipper > Plugins page.

Configure a Log Shipper Business Rule for AWS S3 WebTx
-
In Log Shipper, go to Business Rules.
-
By default, there is a business rule that filters all alerts and events. To filter a specific type of alert or event, click Create New Rule and configure a new business rule by adding the rule name and filter(s).

-
Click Save.

Configure Log Shipper Log Delivery for AWS S3 WebTx
-
In Log Shipper, go to Log Delivery and click Add Log Delivery Configuration.
-
Select the Source plugin (AWS Netskope LogStreaming or Azure Netskope LogStreaming), Destination plugin (AWS S3 WebTx), and click Save.
-
After the Log Delivery configuration is added the data will start to be pulled from the Netskope tenant, transformed, and ingested into the AWS S3 buckets.

Validate the AWS S3 WebTx Plugin
Validate the Pull
To validate the pull of WebTx from the Netskope tenant or any source platform, go to Logging in Cloud Exchange and search for the pulled logs.

Validate the Push
To validate the plugin workflow in Cloud Exchange, go to Logging and search for ingested WebTx with the filter message contains ingested. The ingested logs are filtered.

To validate the push from the AWS S3 platform:
-
Go to Amazon S3 > Buckets. In the search bar, search for and click Buckets.

-
Search for the bucket you used while creating the plugin.
-
Click the bucket name.


Example: Webtxdemobucket/webtx/feedname=2.0.0/year=2023/month=11/day=30/hour=9/1792588265_132874188526208_37862b2b68994795874100a37cc8e2cf.gz
Here is the sample content of the downloaded file:

Troubleshooting the AWS S3 WebTx Plugin
Facing issues in the existing plugin configuration after the plugin update
If you’ve recently updated your AWS S3 plugin, there might be an issue where editing the existing plugin configuration leads to a blank Cloud Exchange interface. This problem may occur in Cloud Exchange versions 4.2.0 and 5.0.0 if users try to modify the plugin configuration after the update and go to the Configuration Parameters page.
What to do:
For versions 4.2.0 and 5.0.0, the only solution available is to delete the current plugin configuration and set up a new one from scratch.
Getting an error while upgrading the AWS S3 WebTx plugin
If you see the error Only the JSON format is supported by this plugin while upgrading the plugin, CEF may be selected as the format under Basic Information.

What to do:
- Click Skip while upgrading the plugin.
- Edit the plugin configuration and select JSON from the Format dropdown.
- Click Save.
- Enable the plugin configuration.

Facing issues while configuring the new plugin
If you are creating a new plugin with AWS IAM Roles Anywhere and face an error of 400, you can find the root cause for not being able to save the plugin in the logs. Go to Logging and expand the logs.


What to do:
When you expand the log, you see At least one of the Trust Anchor ARN, Role ARN, and Profile ARN has a different account ID. Check that the provided parameters are generated from the same account and have the same region.

