AWS LogStreaming Plugin for Log Shipper

This document explains how to configure the AWS Log Streaming v1.0.0 plugin with the Log Shipper module of the Netskope Cloud Exchange platform. This plugin is used to fetch the Netskope Alerts (DLP, Malware, Policy, Compromised Credential, Malsite, Quarantine, Remediation, Security Assessment, Watchlist, UBA, CTEP, Device, and Content), Events (Page, Application, Audit, Infrastructure, Network, Incident, Endpoint and Client Status), and WebTx logs from an AWS SQS queue-enabled S3 Bucket.

Prerequisites

To complete this configuration, you need:

• A Netskope tenant (or multiple, for example, production and development/test instances) with the AWS Netskope Log Streaming service enabled.
• A Netskope Cloud Exchange tenant with the Tenant plugin and Log Shipper plugin already configured.
• A Netskope Cloud Exchange tenant with a 3rd-party plugin (like Syslog) already configured.
• Amazon SQS Queue should be enabled on S3 bucket with following permissions to the IAM user. Make sure your user has admin access for the AWS account.
  • GetObject (Read)
  • GetObjectAttributes (Read)
  • GetQueueAttributes (Read)
  • GetQueueUrl (Read)
  • ReceiveMessage (Read)
  • DeleteMessage (Write)
• Connectivity to a host with AWS S3 Bucket access.

AWS LogStreaming Plugin Support

This plugin is used to fetch the Netskope Alerts (DLP, Malware, Policy, Compromised Credential, Malsite, Quarantine, Remediation, Security Assessment, Watchlist, UBA, CTEP, Device, and Content), Events (Page, Application, Audit, Infrastructure, Network, Incident, Endpoint and Client Status), and WebTx logs from an AWS SQS queue-enabled S3 Bucket.

Permissions

• Amazon S3 bucket permissions to the IAM user to send Events, Alerts, and WebTx data to buckets.
  • GetObject (Read)
  • GetObjectAttributes (Read)
• Amazon SQS Queue permissions to the IAM user to send Alerts data to Queue.
  • GetQueueAttributes (Read)
  • GetQueueUrl (Read)
  • ReceiveMessage (Read)
  • DeleteMessage (Write)

s3_client = boto3.client(
     "s3",
     aws_access_key_id=self.aws_public_key,
     aws_secret_access_key=self.aws_private_key,
     aws_session_token=self.aws_session_token,
     region_name=self.region_name,
     config=Config(proxies=self.proxy, user_agent=self.useragent),
)

s3_client = boto3.client(
     "sqs",
     aws_access_key_id=self.aws_public_key,
     aws_secret_access_key=self.aws_private_key,
     aws_session_token=self.aws_session_token,
     region_name=self.region_name,
     config=Config(proxies=self.proxy, user_agent=self.useragent),
)

response = sqs_client.get_queue_url(QueueName=str(queue_name))

response = sqs_client.receive_message(
     QueueUrl=queue_url,
     MaxNumberOfMessages=1,
     WaitTimeSeconds=2,
     MessageAttributeNames=["All"],
)

Performance Matrix

This performance reading is for a Large Stack CE tested with these VM specifications. These readings are added with the following data considerations:

• 50 MB compressed ~500MB uncompressed (~500k) Alerts/Events in ~3.3 minutes 
• 50 MB compressed ~500MB uncompressed (~500k) WebTx logs in ~2.5 minutes

User Agent

APN/1.1 (ahq9d89xj9gspapczzdb59goq)

Workflow

1. Generate credentials for AWS Netskope LogStreaming authentication methods
2. Configure CLS AWS Netskope LogStreaming Plugin
3. Add Business Rules
4. Add a SIEM Mapping
5. Validate the plugin

Watch a Video

Click play to watch a video:

Configure the Queue with Amazon S3 Buckets

To configure the plugin, you will need to configure one queue with an Amazon S3 bucket:

1. Log in to your AWS account and go to Amazon S3 > Buckets to find your configured buckets.

   

2. Click on any of the buckets and go to Properties > Event Notifications.

   

3. From here create event notifications to ingest the data from the specified folder in the S3 bucket to the configured SQS Queue.
4. And from Amazon SQS > Queues, you can find your configured queues.

   

Create a AWS Netskope LogStreaming Policy

1. Search for IAM in the search box and from the left panel and click Policies.

   

2. Click Create policy.

   

3. On the JSON tab, enter this policy. Click Next:Tags, and then click Next:Review.

   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Sid": "VisualEditor0",
               "Effect": "Allow",
               "Action": [
                   "s3:GetObject",
                   "s3:GetObjectAttributes"
               ],
               "Resource": [
                   "arn:aws:s3:::*/*",
                   "arn:aws:s3:*:932414340604:accesspoint/*"
               ]
           },
           {
               "Sid": "VisualEditor1",
               "Effect": "Allow",
               "Action": [
                   "sqs:DeleteMessage",
                   "sqs:GetQueueUrl",
                   "sqs:ReceiveMessage",
                   "sqs:GetQueueAttributes"
               ],
               "Resource": "*"
           }
       ]
   }

   Note

   Make sure to update your AWS account ID in the above policy template.

4. Enter a Name and click Create Policy.

   

5. Attach this policy to the user. Go to IAM > Users and select the user for which you want to attach a policy. Click Add permissions and click the Add permissions option:

   

6. Select Attach policies directly under Permissions options, and then search for and select the policy created in the previous step for the source queue.

   

7. Click Next, and then click Add permissions. The Policy will be attached to the user.

   

Plugin Authentication Methods

IAM Role Anywhere Configuration 

Prerequisites

The AWS Certificate Manager service needs to be enabled to authenticate the plugin using the AWS IAM Roles Anywhere authentication method.

Make sure you create the Private Certificate Authority, Trust Anchor, and Profile in the same region in which your AWS SQS Queue resides.

Create a Policy

This Policy contains the required permissions for creating Private CA Certificate (including Permissions for creating Trust Anchor and Profile) and using the IAM Roles Anywhere.

1. Go to Policy Generator and select IAM Policy as the policy type. Enter Add Statement details, and then generate the policy.
   • Select Type of Policy: IAM Policy
   • Effect: Allow
   • AWS Service: AWS Private Certificate Authority
   • Actions:
     • CreateCertificateAuthority
     • DescribeCertificateAuthority
     • GetCertificate
     • GetCertificateAuthorityCertificate
     • GetCertificateAuthorityCsr
     • ImportCertificateAuthorityCertificate
     • IssueCertificate
     • ListCertificateAuthorities
   • ARN: *
2. Click Add Statement.

   

3. Scroll back up to add another statement.
   • Select Type of Policy: IAM Policy
   • Effect: Allow
   • AWS Service: AWS Identity and Access Management (IAM)
   • Actions:
     • AttachRolePolicy
     • CreateAccessKey
     • CreateRole
     • DeleteRole
     • PassRole
   • ARN: *
4. Click Add Statement.

   

5. Scroll back up to add another statement.
   • Select Type of Policy: IAM Policy
   • Effect: Allow
   • AWS Service: AWS Certificate Manager
   • Actions:
     • DescribeCertificate
     • ExportCertificate
     • GetCertificate
     • ListCertificates
     • ListTagsForCertificate
     • RequestCertificate
   • ARN: *
6. Click Add Statement.

7. Scroll back up to add another statement.
   • Select Type of Policy: IAM Policy
   • Effect: Allow
   • AWS Service: AWS Identity and Access Management Roles Anywhere
   • Actions:
     • CreateProfile
     • CreateTrustAnchor
     • GetProfile
     • GetTrustAnchor
     • ListProfiles
     • ListTrustAnchors
   • ARN: *
8. Click Add Statement.

9. Click Generate Policy.

10. Copy the Policy as it will be used in the next step for creating the policy required for creating the Private CA certificates.
11. Go to AWS Console and select IAM from All Services. Click Policies in the left panel, and then click Create Policy.

    

12. Copy the policy to the JSON tab, click Next:Tags, and then click Next:Review.

    

13. Enter a name (like netskope-ce-rolesAnywhere-policy) and click Save Changes.

    

Create a Private Certificate Authority

1. Log in to the AWS Console.
2. Search for Certificate Manager.

   

3. Click AWS Private CA.
4. Click Create a private CA.

   

5. For Mode Options, select General-purpose.
6. For CA type options, select Root.

   

7. Enter the Organization (O).

   

8. For Key algorithm options, select RSA 2048.

   

9. Add tags if any (optional).
10. Check the checkbox in the CA permissions options section.
11. Check the checkbox in the Pricing section.
12. Click Create to create the CA certificate.

    

    

13. From Actions, select Install CA Certificate.

    

14. Click Confirm and Install.

    

    

Create a Trust Anchor

1. Search for the IAM service, go to Roles under Access Management and then scroll down to Roles Anywhere and select Manage.

   

2. Click Create a Trust anchor.

   

3. Enter a Trust anchor name, like netskope-ce-aws-verified-trust-anchor.

   

4. Select AWS Certificate Manager Private CA (created in the previous steps) as a Certificate authority (CA) source. Select the Certificate that was created in the previous step in AWS Private Certificate Authority.

   

5. Add tags if required.
6. Click Create a trust anchor.

   

7. Click on the created Trust Anchor and copy the Trust Anchor ARN.

   

Create an IAM Role

1. Go to IAM services in the AWS Console.
2. Click Roles under the Access Management
3. Click Create Role.

   

4. In the Trusted entity type, select Custom Trust Policy.

   

5. Replace the Custom trust Policy with this Trust Policy. This policy contains the permissions for using the roles anywhere service:

   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Effect": "Allow",
               "Principal": {
                   "Service": [
                       "rolesanywhere.amazonaws.com"
                   ]
               },
               "Action": [
                   "sts:AssumeRole",
                   "sts:TagSession",
                   "sts:SetSourceIdentity"
               ]
           }
       ]
   }

   

6. Click Next.
7. In the Permissions policies, select the policy created previously in Create a SQS Queue Policy.

   

8. Click Next.
9. Provide a Role name (like AWS-SQS-Role) and a Description for the role.

   

10. Click Create role. Search for the created role and click on the Role Name.

    

11. Make a note of the Role ARN because this will be required in the Plugin configuration parameter Role ARN for the authentication method AWS IAM Roles Anywhere.

    

Create a Profile

1. Go to IAM services in the AWS Console.
2. Select Roles under Access management.
3. Scroll down to Roles Anywhere and click Manage.

   

4. Expand the Setup steps.
5. Click Step 2: Configure roles.
6. Click Configure a profile.

   

7. Enter a Profile name, like AWS-SQS-Profile.
8. Select the role created in Create IAM Role.

   

9. Remove the Inline Policy.

   

10. Click Create a profile.

    

11. Select the created Profile and copy the Profile ARN.

    

    

12. Go to AWS Certificate Manager > Request certificate.
13. Select Request a private certificate.

    

14. Click Next.
15. Select the Certificate authority created in the previous step.

    

16. Provide a domain name in the Fully qualified domain name field, like netskope-ce-verified-access.com.
17. Select RSA 2048 as the Key algorithm.

    

18. Add tags if required.
19. Acknowledge the Certificate renewal permissions.
20. Click Request.

    

21. Go to List certificates from the navigation pane of AWS Certificate Manager.
22. Select the certificate created previously.

    

23. Click Export.

    

23. Enter a passphrase. Make a note of the passphrase as it will be required for the Configuration of the AWS S3 Plugin using the AWS IAM Roles Anywhere Authentication method.
24. Click Generate PEM Encoding.

    

25. Download all the Certificates because these won’t be visible again. For new certificates, you will need to Export them again.

    

26. The Certificate body, Certificate Private Key will be required for the Configuration of the AWS Verified Access Plugin using the AWS IAM Roles Anywhere authentication method. For more info, go to AWS IAM Role Anywhere.

Deployed on AWS Authentication

Create a Role

1. Go to IAM services in the AWS Console.
2. Click Create role.

   

3. Select the AWS Service.
4. Under Use case, select EC2.
5. Click Next.

   

6. Select the permission policy created in Create a SQS Queue Policy.

   

7. Click Next.
8. Enter a Role Name (like netskope-ce-instance-role) and Description.
9. Click Create Role.

   

   

Assign a Role to an EC2 Instance

1. Log in to your EC2 instance console.
2. Click Instances under Instances.

   

   

3. Go to Action > Security > Modify IAM Role.

   

4. Select the Role that you created above in Create a Role. (netskope-ce-instance-role).
5. Click Add IAM Role or Modify IAM Role. Note that both EC2 instance and Queue should be on the same region.

   

Assign a Role to a K8s Instance

1. Open your Role created for ServiceAccount while creating K8s instance.
2. Attach the policy created in Create a SQS Queue Policy.

Configure the AWS LogStreaming Plugin

1. In Cloud Exchange, go to Settings > Plugins.
2. Search for and select the AWS Netskope LogStreaming v1.0.0 (CLS) plugin box.

   

3. Enter a Configuration Name and set the pull interval per your requirements.

   

4. Click Next and enter the Configuration parameters:
   • Authentication Method: Select the method to be used for AWS client authentication.
   • Private Key: Private Key for decrypting the AWS Private CA Certificate. Required for the AWS IAM Roles Anywhere authentication type.
   • Certificate Body: Certificate Body for AWS Public/Private CA Certificate. Required for the AWS IAM Roles Anywhere authentication type.
   • Password Phrase: Password Phrase for decrypting the CA Certificate. Required for the AWS IAM Roles Anywhere authentication type.
   • Profile ARN: AWS Profile ARN for AWS client authentication. Required for the AWS IAM Roles Anywhere authentication type.
   • Role ARN: AWS Role ARN for AWS client authentication. Required for the AWS IAM Roles Anywhere authentication type.
   • Trust Anchor ARN: AWS Trust Anchor ARN for AWS client authentication. Required for the AWS IAM Roles Anywhere authentication type.
   • AWS Region Name: AWS Region Name in which the SQS queue enabled S3 bucket is set up. Make sure that the region name matches the region in the Profile ARN and Trust Anchor ARN if you have selected the authentication method as AWS IAM Roles Anywhere.
   • AWS SQS Queue Name: AWS SQS Queue Name with which the S3 bucket is set up. To get more details, go to AWS > SQS > QueueName. Make sure that the provided queue is pre-configured on your AWS instance. A new queue will not be created if the provided queue does not exist.

   Note

   AWS Region Name and AWS SQS Queue Name are the only required fields for the Deployed on AWS authentication method.

   

   

5. Click Save. Your Plugin configuration will be available at Log Shipper > Plugins.

   

   Note

   We have configured the SYSLOG with Splunk plugin to ingest the pulled data from the AWS Netskope LogStreaming plugin.

Configure a Log Shipper Business Rule for AWS LogStreaming

1. In Log Shipper, go to Business Rules.
2. By default, there is a business rule that filters all alerts and events. If you want to filter out any specific type of alert or event, click Create New Rule and configure a new business rule by adding the rule name and filter.

   

Configure Log Shipper SIEM Mappings for AWS LogStreaming

1. Go to SIEM Mappings and click Add SIEM Mapping.
2. Select the Source plugin (CLS AWS Netskope LogStreaming), a Destination plugin, and a business rule, and then click Save.
3. After the SIEM mapping is added, data will start to be pulled from the AWS Netskope LogStreaming tenant, transformed, and ingested into the 3rd-party SIEM platform.

   

   

Validate the AWS LogStreaming Plugin

Validate the Pull

To validate the pulling of indicators from the Netskope tenant, go to Logging in Cloud Exchange and search for the pulled logs with the filter message contains pulled.

Validate the Push

To validate the plugin workflow in Cloud Exchange, go to Logging and search for ingested events/alerts with the filter message contains ingested, or you can also use the destination plugin name as the filter. 

Troubleshooting the AWS LogStreaming Plugin

Facing issues while fetching Alerts/Events or WebTx data from the configured Queue

If you have configured AWS Netskope LogStreaming plugin and 3rd-party SIEM platform successfully and still you are getting error after configuring SIEM Mapping for the plugin, it can be due to one of these reasons:

• You do not have access to the configured Queue.
• The permissions provided to IAM User are insufficient.
• All alerts/events/webtx data are consumed and nothing is available to pull from the queue.

  

• The provided queue in the plugin might not be enabled in the S3 bucket.

What to do:

1. Verify if the Queue is available on the AWS SQS service.
2. Verify that IAM User has provided sufficient permissions provided. For the minimum permissions, you can refer to Permissions.
3. Verify that you have data to consume from the queue, for that you have to go to Amazon SQS > Queues.

   

4. To verify the enabled queue in the S3 bucket, follow the steps mentioned in Configure the Queue with Amazon S3 Buckets.

Facing issues while configuring the new plugin

If you’re creating a new plugin with AWS IAM Roles Anywhere and get an error like:

What to do:
After expanding the logger, you will see At least one of the Trust Anchor ARN, Role ARN, and Profile ARN has a different account ID, so you need to check that the provided parameters are generated from the same account and have the same region.

Data getting skipped while pulling through AWS Netskope LogStreaming Plugin

You might be facing this issue due to one of the following reasons:

• Invalid file or file having invalid data
• For Alerts/Events, alert_type and record_type are required fields and for WebTx, x-cs-timestamp is the required field. If the data does not have following fields then it will be skipped.

What to do: Validate your file is valid and has valid data. Also, validate that the data contains the required fields. For Alerts/Events, alert_type and record_type are required fields; for WebTx, x-cs-timestamp is the required field.

Known Behaviors

• If you have RBACv3 enabled in your Netskope tenant, you will not be able to use the Netskope LogStreaming services.
• If you restart Netskope Cloud Exchange, or due to some reason your Netskope Cloud Exchange restarts automatically, while the AWS NLS plugin is running, then the pull task may get stuck in progress. To fix this, you can reconfigure the SIEM mapping to start the pulling again.