AWS SQS Plugin for Log Shipper

This document explains how to configure the AWS SQS plugin for the Log Shipper module of the Netskope Cloud Exchange platform.This plugin supports ingestion of Alerts (DLP, Malware, Policy, Compromised Credential, Malsite, Quarantine, Remediation, Security Assessment, Watchlist, CTEP, UBA) data to the AWS SQS platform. To access the plugin, you would need the credentials of AWS. Note: This plugin supports ingestion in JSON format on the AWS SQS Queue.

Prerequisites

To complete this configuration, you need:

A Netskope tenant (or multiple, for example, production and development/test instances) that is already configured in Cloud Exchange.
A Netskope Cloud Exchange tenant with the Log Shipper module already configured.
Connectivity to AWS SQS.
Connectivity to the following host: https://<aws_region>.console.aws.amazon.com/. For example: https://us-east-1.console.aws.amazon.com/
AWS SQS service access.

CE Version Compatibility

Netskope CE v4.2.0, v5.0.1

AWS SQS Plugin Support

AWS SQS plugin is used to ingest Netskope Alerts data to AWS SQS Queue. This plugin supports sending data in the JSON format to the SQS Queue.

Event Types	Not Supported
Alert Types	Yes (DLP, Malware, Policy, Compromised Credential, Malsite, Quarantine, Remediation, Security Assessment, Watchlist, CTEP, UBA)
Log Types	Not Supported
WebTx Support	Not Supported

Mappings

Netskope Field	AWS SQS Field
alert_name	Name
alert_type	Type

Permissions

Amazon SQS Queue permissions to the IAM user to send Alerts data to Queue. Follow the steps mentioned in Create a SQS Queue Policy.

API Details

List of APIs Used

This plugin uses Python libraries to create and ingest data in AWS SQS.
Library: The AWS SDK for Python (Boto3).
Usage: The AWS SDK for Python (Boto3) to create, configure, and manage AWS services, such as Amazon Simple Queue. Service (Amazon SQS), Amazon Elastic Compute Cloud (Amazon EC2) and Amazon Simple Storage Service (Amazon S3). The SDK provides an object-oriented API as well as low-level access to AWS services.
Creating the SQS Client:

sqs_client= boto3.client(
                "sqs",
                aws_access_key_id=self.aws_public_key,
                aws_secret_access_key=self.aws_private_key,
                aws_session_token=self.aws_session_token,
                region_name=self.region_name,
                config=Config(proxies=self.proxy, user_agent=self.useragent),
            )

Creating a Queue:

queue= sqs_client.create_queue(
                        QueueName=queue_name
                    )

Getting a Queue URL:

queue_url = sqs_client.get_queue_url(
                        QueueName=queue_name
                    )

Sending Messages in a Batch:

response = sqs_client.send_message_batch(
                         QueueUrl=queue_url, Entries=batches
                      )

Performance Matrix

This performance reading is for a Large Stack CE tested on the below-mentioned VM specifications. The below readings are added with the consideration that it will ingest around 10K file size in 21 seconds.

Stack details

Size: LargeRAM: 32 GB

CPU: 16 Cores

Events, Alerts ingested to third-party SIEM

200K EPM

User Agent

netskope-ce-5.0.1-cls-aws_sqs-v1.0.0

Workflow

Create an AWS SQS Queue and Queue Policy.
Generate a Private Key, Certificate Body, Password Phrase, Profile ARN, Role ARN, Trust Anchor ARN, if using AWS IAM Role Anywhere for authentication. Or use another AWS authentication configuration.
Create a Queue.
Configure the AWS SQS Alerts plugin.
Configure a Business Rule.
Configure a SIEM Mapping.
Validate the plugin.

Click play to watch a video:

Create AWS SQS Queue

The queue name created here will be used in the Queue Name parameter while configuring the plugin.

To create a queue and set the required policies, log in to the AWS console.
From All Services, search for and select Simple Queue Service.
Click Create queue.
For Type, select Standard. Enter a name for the queue.
Provide the Configuration info according to your needs.
Enable Server-side encryption, and select Amazon SQS Key for the Encryption key type.
Select the Access policy, Redrive allow policy, Dead-letter queue, and Tags per your requirements. Click on Create.
The Queue is created successfully.

Create a SQS Queue Policy

Search for IAM, and on the left panel, click on Policies.
Click Create policy.

Click on the JSON tab, and enter the below-mentioned policy. Click Next: Tags and then Next: Review.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "sqs:GetQueueUrl",
                "sqs:SendMessage",
                "sqs:CreateQueue"
            ],
            "Resource": "arn:aws:sqs:*:XXXXXXXXXXX:*"
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": "sqs:ListQueues",
            "Resource": "*"
        }
    ]
}

Enter a Name and click Create Policy.
Attach this policy to the user. Go to IAM > Users. Select the user that you want to attach a policy to, click Add permissions, and then click on Add permissions options.
Select Attach policies directly under Permissions, and search and select the policy created in the previous step for the source queue.
Click Next, and then click Add permissions. The policy will be attached to the user.

Plugin Authentication Methods

IAM Role Anywhere Authentication

Prerequisites

The AWS Certificate Manager service is required to be enabled to authenticate the plugin using the AWS IAM Roles Anywhere Authentication Method.

Make sure you create the Private Certificate Authority, Trust Anchor and Profile in the same region in which your AWS S3 Source Bucket resides.

Create a Policy

This Policy contains the required permissions for creating Private CA Certificate (including Permissions for creating Trust Anchor and Profile) and using the IAM Roles Anywhere.

Go to Policy Generator and select IAM Policy as policy type, enter Add Statement details, and generate a policy.
- Select Type of Policy: IAM Policy
- Effect: Allow
- AWS Service: AWS Private Certificate Authority
- Actions:
  - CreateCertificateAuthority
  - DescribeCertificateAuthority
  - GetCertificate
  - GetCertificateAuthorityCertificate
  - GetCertificateAuthorityCsr
  - ImportCertificateAuthorityCertificate
  - IssueCertificate
  - ListCertificateAuthorities
- ARN: *
Click Add Statement.
Scroll back up to add another statement.
- Select Type of Policy: IAM Policy
- Effect: Allow
- AWS Service: AWS Identity and Access Management (IAM)
- Actions:
  - AttachRolePolicy
  - CreateAccessKey
  - CreateRole
  - DeleteRole
  - PassRole
- ARN: *
Click Add Statement.
Scroll back up to add another statement.
- Select Type of Policy: IAM Policy
- Effect: Allow
- AWS Service: AWS Certificate Manager
- Actions:
  - DescribeCertificate
  - ExportCertificate
  - GetCertificate
  - ListCertificates
  - ListTagsForCertificate
  - RequestCertificate
- ARN: *
Click Add Statement.
Scroll back up to add another statement.
- Select Type of Policy: IAM Policy
- Effect: Allow
- AWS Service: AWS Identity and Access Management Roles Anywhere
- Actions:
  - CreateProfile
  - CreateTrustAnchor
  - GetProfile
  - GetTrustAnchor
  - ListProfiles
  - ListTrustAnchors
- ARN: *
Click Add Statement.
Click Generate Policy.
Copy the Policy as it will be used in the next step for creating the policy required for creating the Private CA certificates.
Go to AWS Console and select IAM from All Services. Click Policies in the left panel, and then click Create Policy.
Copy the policy to the JSON tab, click Next:Tags, and then click Next:Review
.
Enter a name (like netskope-ce-rolesAnywhere-policy) and click Save Changes.

Create a Private Certificate Authority

Log in to AWS Console.
Search for Certificate Manager.
Click AWS Private CA.
Click Create a private CA.
For Mode Options, select General-purpose.
For CA type options, select Root.
Enter Organization (O).
For Key algorithm options, select RSA 2048.
Add tags if any (optional).
Check the checkbox in the CA permissions options section.
Check the checkbox in the Pricing section.
Click Create to create the CA certificate.
From Actions, select Install.
Click Confirm and Install.

Create a Trust Anchor

Search for the IAM service, go to Roles under Access management scroll down to Roles Anywhere and select Manage.
Click Create a Trust anchor
Enter a Trust anchor name, like netskope-ce-trust-anchor.
Select AWS Certificate Manager Private CA (created in the previous steps) as a Certificate authority (CA) source
Add tags if required.
Click Create a trust anchor.
Click on created Trust Anchor and copy the Trust Anchor ARN.

Create an IAM Role

Go to IAM services in the AWS Console.
Click Role in the Access Management submenu.
Click Create Role.
In the Trusted entity type, select Custom Trust Policy.
Go to Policy Generator.

Replace the Custom trust Policy with the below Trust Policy. This policy contains the permissions for using the roles anywhere service:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": [
                    "rolesanywhere.amazonaws.com"
                ]
            },
            "Action": [
                "sts:AssumeRole",
                "sts:TagSession",
                "sts:SetSourceIdentity"
            ]
        }
    ]
}

Click Next.
In the Permissions policies, select the policy created in Create a SQS Queue Policy.
Click Next.
Provide a Role name (like AWS-SQS-Role) and Description for the role.
Click Create role.
Make a note of the Role ARN as this will be required in the Plugin configuration parameter Role ARN for the authentication method AWS IAM Roles Anywhere.

Create a Profile

Select Roles under Access management.
Scroll down to Roles Anywhere and click Manage.
Expand the Setup steps.
Click Step 2: Configure roles.
Click Configure a profile.
Enter a Profile name, like netskope-ce-profile.
Select the role created in Create IAM Role netskope-ce-roleAnywhere.
Remove the Inline Policy.
Click Create profile.
Select the created Profile and copy the Profile ARN.

Request a Private Certificate

Go to AWS Certificate Manager > Request certificate.
Select Request a private certificate.
Click Next.
Select the Certificate authority created in the previous step.
Provide a domain name in the Fully qualified domain name field, like netskope-ce.com.
Select RSA 2048 as the Key algorithm.
Add tags if required.
Acknowledge the Certificate renewal permissions.
Click Request.
Go to List certificates from the navigation pane of AWS Certificate Manager.
Select the certificate created previously.
Click Export.
Enter the passphrase. Make a note of the passphrase as it will be required for the Configuration of the AWS Security Lake Plugin using the AWS IAM Roles Anywhere Authentication method.
Click Generate PEM Encoding.
Download all the Certificates as it won’t be visible again. For new certificates you will need to Export them again.

For More Info go to AWS IAM Role Anywhere

AWS Authentication

Create a Role

Go to IAM services in the AWS Console.
Click Create role.
Select the AWS Service.
Under Use case, select EC2.
Click Next.
Select the permission policy created in Create an AWS SQS Policy.
Click Next.
Enter a Role Name (like AWS-SQS-Role) and add a Description.
Click Create Role.

Assign a Role to an EC2 Instance

Log in to your EC2 instance console.
Click Instances under Instances.
Go to Action > Security > Modify IAM Role.
Select the Role that you created above in Create a Role. (netskope-ce-instance-role).
Click Add IAM Role or Modify IAM Role.

Assign a Role to a K8s Instance

Open your Role created for ServiceAccount while creating K8s instance.
Attach the policy created in Create a AWS SQS Policy.

Configure the AWS SQS Plugin

In Cloud Exchange, go to Settings > Plugins.
Search for and select the AWS SQS Plugin to open the plugin creation dialog.
Enter a Configuration Name. Disable the transformation toggle button, as only sending JSON data is supported.
Click Next.
Enter values for these parameters:
- Authentication Method: Select the method to be used for authentication (Deployed on AWS/AWS IAM Roles Anywhere)
- Private Key: Private Key for decrypting the AWS Private CA Certificate. Required for AWS IAM Roles Anywhere authentication type.
- Certificate Body: Certificate Body for AWS Public/Private CA Certificate. Required for AWS IAM Roles Anywhere authentication type.
- Password Phrase: Password Phrase for decrypting the CA Certificate. Required for AWS IAM Roles Anywhere authentication type.
- Profile ARN: AWS Profile ARN for AWS client authentication. Required for AWS IAM Roles Anywhere authentication type.
- Role ARN: AWS Role ARN for AWS client authentication. Required for AWS IAM Roles Anywhere authentication type.
- Trust Anchor ARN: AWS Trust Anchor ARN for AWS client authentication. Required for AWS IAM Roles Anywhere authentication type.
- AWS SQS Region Name: AWS SQS Region Name from where to get the AWS queue. Make sure that the region name matches the region in the Profile ARN and Trust Anchor ARN.
- AWS SQS Queue Name: AWS SQS Queue Name in which the data object will be stored.
Click Save. The new plugin will be available on the Log Shipper > Plugins page.

Configure a Log Shipper Business Rule for AWS SQS

Go to the Business Rule page.
By default, we have a business rule that filters all alerts and events. If you want to filter out any specific type of alert, click on ‘Create New Rule’ and configure a new business rule by adding the rule name and filter.

Configure a Log Shipper SIEM Mapping for AWS SQS

Go to Log Shipper > SIEM Mappings and click Add SIEM Mapping.
Select the Source plugin (CLS Netskope), the Destination plugin (CLS AWS SQS), the business rule, and then click Save.
After the SIEM mapping is added, the data will start to be pulled from the Netskope tenant, transformed, and ingested into the AWS SQS Queue.

Validate the AWS SQS Plugin

Validate the Pull

To validate the pulling of indicators from the Netskope tenant, go to Logging in Netskope CE. Search for the pulled logs..

Validate the Push

To validate the plugin workflow in Netskope Cloud Exchange:

Go to Logging and search for ingested events with the filter message contains ingested.
The ingested logs will be filtered.

To validate the push on the AWS SQS Queue:

Go to AWS Console. On the Search bar, search for Queue and click Simple Queue Service.
Search the Queue you used while creating the plugin.
Click on the queue name, and then click Send and receive messages.
Scroll down and click Poll for messages.
All the shared data will be visible as shown below.

Specific alerts can be filtered out using the Search messages text box. Click on the ID to view the Alert details. Click on the Attributes tab to see the Name and Type Mapping of the alert.

Troubleshooting

Unable to configure the CLS AWS SQS plugin.

If you are unable to configure the CLS AWS SQS plugin, then, it could be due to one of these reasons:

If the authentication method IAM Role Anywhere, the certificate and other configuration parameters may not be present in the same region.
If the Netskope CE is deployed on AWS, using the Deployed on AWS authentication method, the proper role is not attached to the EC2 instance.
Invalid values provided to the configuration parameters.
IAM permissions not provided to the user on the AWS platform.

To solve these issues, follow the below-mentioned steps.

Make sure that the certificate and other configuration parameters are created in the same region.
Make sure to follow the Role Assign section to assign a proper role to the EC2 instance.
Make sure valid values are provided in the configuration parameters. Navigate to the logging page and verify the log message.
Provide the IAM permissions to the user using which configuration parameters are created.

Unable to push alerts on the AWS SQS Queue.

If you are unable to push alerts on the AWS SQS Queue, then it could be due to the Queue being deleted from the AWS SQS platform after configuring the CLS AWS SQS plugin.

To solve this issue, update the plugin configuration with the new queue name.

Unable to find alerts on the AWS SQS Queue.

If you are unable to find alerts on the AWS SQS Queue, then it could be due to the Default message retention period being 4 days for the AWS SQS Queue. If alerts are older than the message retention period, alerts are automatically deleted.

To solve this issue, follow these steps:

Go to the AWS console and enter credentials to log in.
From All Services, Select Simple Queue Service or search for Simple Queue Service in the search bar.
Select your queue from the list and click on the Edit button.
In Configuration, update Message retention period as per your requirement.

AWS SQS Plugin for Log Shipper