Updated on March 14, 2024

AWS S3 WebTx Plugin for Log Shipper

AWS S3 WebTx Plugin for Log Shipper

This document explains how to configure your AWS S3 WebTx integration with the Log Shipper module of the Netskope Cloud Exchange platform. This integration allows pushing web transactions data into AWS S3 buckets.

Prerequisites

  • A Netskope tenant (or multiple, for example, production and development/test instances) that is already configured in Cloud Exchange.
  • A Netskope Cloud Exchange tenant with the Log Shipper module and WebTX plugin already configured.
  • AWS S3 credentials with create/read/write buckets permissions. Obtain your AWS S3 Access key ID and Secret Access Key with permission to read/write on specific buckets or permission to create buckets before proceeding.
  • Amazon S3 bucket permissions to the IAM user.
    • ListBucket
    • CreateBucket
    • ListAllMyBuckets
    • GetBucketPolicy
    • GetBucketPublicAccessBlock
    • PutEncryptionConfiguration
    • PutBucketPublicAccessBlock
    • PutBucketPolicy

    • Note

    Verify your bucket permissions are secure and not set up for open public access. Only allow access to your cloud storage instance from your Cloud Exchange Host and any other addresses that need access.

    AWS S3 WebTx Plugin Support

    The AWS S3 WebTx plugin support is:

    Event SupportNo
    Alert SupportNo
    WebTx SupportYes
    CE Version Compatibility

    This plugin is compatible with Netskope CE v4.2.0, v5.0.0

    Permissions

    Amazon S3 bucket permissions to the IAM user to send webtx data to buckets.

    Performance Matrix

    This performance reading is for a Large Stack CE tested with the below-mentioned VM specifications. The below readings are added with the consideration that it will ingest around 10MB file size in 2 seconds to the EC2 platform

    Stack detailsSize: Large
    RAM: 32 GB
    CPU: 16 Cores
    Webtx ingested to third-party SIEM~ 5 MBps
    API Details
    List of APIs Used

    This plugin uses Python libraries to create file objects in AWS S3.

    Library: The AWS SDK for Python (Boto3)

    Usage: The AWS SDK for Python (Boto3) to create, configure, and manage AWS services, such as Amazon Elastic Compute Cloud (Amazon EC2) and Amazon Simple Storage Service (Amazon S3). The SDK provides an object-oriented API as well as low-level access to AWS services.

    Create the Client

    s3_client = boto3.client(
                "s3",
                aws_access_key_id=self.aws_public_key,
                aws_secret_access_key=self.aws_private_key,
                aws_session_token=self.aws_session_token,
                region_name=self.region_name,
                config=Config(proxies=self.proxy, user_agent=self.useragent),
              )

    Create a Bucket 

    bucket = s3_client.create_bucket(
               Bucket=bucket_name,
               CreateBucketConfiguration=location,
             )

    Upload a File into the Bucket

    s3_client.upload_file(
                file_name,
                bucket_name,
                object_name,
            )

    Get an AWS Resource

    s3_resource = boto3.resource(
                "s3",
                aws_access_key_id=self.aws_public_key,
                aws_secret_access_key=self.aws_private_key,
                region_name=self.region_name,
                config=Config(proxies=self.proxy, user_agent=self.useragent),
            )
    User Agent

    The user-agent added in this plugin is in the following format

    netskope-ce-<ce_version>-<module>-<plugin_name>-v<plugin_version>

    For example:

    Netskope-ce-5.0.0-cls-aws_s3_webtx-v1.2.0

    Workflow

    1. Create a bucket, a bucket policy, and configure an authentication method.
    2. Configure the AWS S3 WebTx plugin.
    3. Configure Log Shipper Business Rules for AWS S3 WebTx.
    4. Configure Log Shipper SIEM mappings for AWS S3 WebTx.
    5. Validate the AWS S3 WebTx plugin.

    Click play to watch a video.

     

    Create an AWS S3 Bucket and Bucket Policy

    Create an AWS S3 Bucket

    1. To create bucket and set required policies, go to your AWS console and log in.
      image1.png
    2. From All Services, select S3.
      image2.png
    3. Click Create Bucket.
      image3.png
    4. Enter a Bucket Name, scroll to the bottom, and click Create Bucket.
      image4.png
    5. Search for your bucket and click on the bucket name to open it.
      image5.png
    6. Click on Permission to open the permission tab and to set a policy.
      image6.png
    7. Click Edit. Under Block public access (bucket settings), uncheck all checkboxes and click Save Changes. When prompted for a confirmation, confirm it, and then click Confirm.
      image7.png
    8. Under the Permissions tab, click Edit in the Bucket Policy section.
      image8.png
    9. Click Policy Generator.
      image9.png
    10. Select S3 Bucket Policy as policy type for Step 1, and Add Statement details for Step 2, and then click Generate Policy.
      • Select Type of Policy: S3 Bucket Policy
      • Effect: Allow
      • Principal: <user-arn>
      • Actions:
        1. GetBucketAcl
        2. GetBucketPolicy
      • ARN: arn:aws:s3:::<bucket-name>
      image10.png
    11. Click Add Statement.
      • Select Type of Policy: S3 Bucket Policy
      • Effect: Allow
      • Principal: <user-arn>
      • Actions:
        1. PutObject
        2. PutObjectAcl
      • ARN: arn:aws:s3:::<bucket-name>/*. Be sure to add /* after the second bucket name.
    12. Click Add Statement
      image11.png
      image12.png
    13. Click Generate Policy. Make sure to copy the policy generated in this step as it will be used in the next step to attach in the source bucket.
      image13.png
    14. Add this policy to this textbox.
      {
    "Id": "<policy ID>",
    "Version": "<version>",
    "Statement": [
        {
            "Sid": "<statement ID>",
            "Action": [
                "s3:GetBucketAcl",
                "s3:GetBucketPolicy"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:s3:::<bucket-name>",
            "Principal": {
                "AWS": [
                    "<user-arn>"
                ]
            }
        },
        {
            "Sid": "<statement ID>",
            "Action": [
                "s3:PutObject",
                "s3:PutObjectAcl"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:s3:::<bucket-name>/*",
            "Principal": {
                "AWS": [
                    "<user-arn>"
                ]
            }
        }
    ]
}
    15. Scroll to the bottom and click Save Changes.
      image14.png

    Create a Bucket Policy

    A Policy is required for accessing the source bucket by the plugin.

    1. Go to Policy Generator. 
      Replace the <bucket-name> with the AWS S3 Source Bucket created previously. (netskope-ce-source-bucket).
      • Select Type of Policy: IAM Policy
      • Effect: Allow
      • AWS Service: Amazon S3
      • Actions: ListAllMyBuckets
      • ARN: *
    2. Click Add Statement.
    3. Scroll back up and add another statement.
      • Select Type of Policy: IAM Policy
      • Effect: Allow
      • AWS Service: Amazon S3
      • Actions:
        • GetBucketAcl
        • GetBucketPolicy
        • GetBucketLocation
      • ARN: arn:aws:s3:::<bucket-name>
    4. Click Add Statement.
    5. Scroll back up and add another statement.
      • Select Type of Policy: IAM Policy
      • Effect: Allow
      • AWS Service: Amazon S3
      • Actions:
        • PutObject
        • PutObjectAcl
      • ARN: arn:aws:s3:::<bucket-name>
    6. Click Add Statement.
    7. Click Generate Policy. Make sure to copy the policy generated in this step as it will be used in the next step to create a policy that will be attached to the user for accessing the source bucket.
    8. Go to AWS Console and select IAM from All Services.
    9. Click Policies in the left panel, and then click Create Policy.
    10. Paste the policy created in the previous step to the JSON tab, and then click Next:Tags and Next:Review.
    11. Enter a name (like netskope-ce-s3-policy).
    12. Click Create Policy.
    13. Attach this policy to the user following these steps:
      1. Go to IAM > Users. Select the user on which you want to attach a policy, and click Add permissions, and then click Add permissions from the dropdown.
      2. Select Attach policies directly under Permissions, and then search for and select the policy created in the previous step (netskope-ce-s3-policy).
      3. Click Next and then click Add permissions. A Policy will be attached to the user.

    Plugin Authentication Methods

    IAM Role Anywhere Configuration

    Prerequisites

    The AWS Certificate Manager service is required to be enabled to authenticate the plugin using the AWS IAM Roles Anywhere Authentication Method.

    Note: Make sure you create the Private Certificate Authority, Trust Anchor, and Profile in the same region in which your AWS S3 Source Bucket resides.

    Create an IAM Policy

    This Policy contains the required permissions for creating a Private CA Certificate (including Permissions for creating a Trust Anchor and Profile) and using IAM Roles Anywhere.

    1. Go to Policy Generator and click Add Statement to generate a policy.
      • Select Type of Policy: IAM Policy
      • Effect: Allow
      • AWS Service: AWS Private Certificate Authority
      • Actions:
        • CreateCertificateAuthority
        • DescribeCertificateAuthority
        • GetCertificate
        • GetCertificateAuthorityCertificate
        • GetCertificateAuthorityCsr
        • ImportCertificateAuthorityCertificate
        • IssueCertificate
        • ListCertificateAuthorities
      • ARN: *
    2. Click Add Statement.
      • Select Type of Policy: IAM Policy
      • Effect: Allow
      • AWS Service: AWS Identity and Access Management (IAM)
      • Actions:
        • AttachRolePolicy
        • CreateAccessKey
        • CreateRole
        • DeleteRole
        • PassRole
      • ARN: *
    3. Click Add Statement.
      • Select Type of Policy: IAM Policy
      • Effect: Allow
      • AWS Service: AWS Certificate Manager
      • Actions:
        • DescribeCertificate
        • ExportCertificate
        • GetCertificate
        • ListCertificates
        • ListTagsForCertificate
        • RequestCertificate
      • ARN: *
    4. Click Add Statement.
      • Select Type of Policy: IAM Policy
      • Effect: Allow
      • AWS Service: AWS Identity and Access Management Roles Anywhere
      • Actions:
        • CreateProfile
        • CreateTrustAnchor
        • GetProfile
        • GetTrustAnchor
        • ListProfiles
        • ListTrustAnchors
      • ARN: *
    5. Click Add Statement.
    6. Click Generate Policy.
    7. Copy the Policy as it is used in the next step for creating the policy required for creating the Private CA certificates.
    8. Go to the AWS Console and select IAM from All Services. Click Policies from the left panel, and click Create Policy.
    9. Copy the policy to the JSON tab, and then click on Next: Tags and Next: Review.
    10. Enter a name, like netskope-ce-rolesAnywhere-policy, and click Save Changes.
    Create a Private Certificate Authority
    1. Log in to AWS Console.
    2. Search for Certificate Manager.
    3. Click AWS Private CA.
    4. Click Create a private CA.
    5. Select General-purpose for Mode Options.
    6. Select Root for CA type options.
    7. Enter the Organization (O).
    8. Select RSA 2048 for Key algorithm options.
    9. Add tags if any (optional).
    10. Enable the checkbox in the CA permissions options section.
    11. Enable the checkbox in the Pricing section.
    12. Click Create to create the CA certificate.
    13. From Actions, select Install.
    14. Click Confirm and Install.
    Create a Trust Anchor
    1. Search for the IAM service, and go to Roles under Access management. Scroll down to Roles Anywhere and select Manage.
    2. Click Create a Trust anchor.
    3. Enter the Trust anchor name, like netskope-ce-trust-anchor.
    4. Select AWS Certificate Manager Private CA (created in the previous steps) as a Certificate authority (CA) source
    5. Add tags if required.
    6. Click Create a trust anchor.
    7. Click on the created Trust Anchor and copy the Trust Anchor ARN.
    Create an IAM Role
    1. Go to IAM services in the AWS Console.
    2. Click Role under Access management.
    3. Click Create Role.
    4. For the Trusted entity type, select Custom Trust Policy.
    5. Go to Policy Generator.
    6. Replace the Custom trust Policy with the below Trust Policy; this policy contains the permissions for using the roles anywhere service: 
      {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": [
                    "rolesanywhere.amazonaws.com"
                ]
            },
            "Action": [
                "sts:AssumeRole",
                "sts:TagSession",
                "sts:SetSourceIdentity"
            ]
        }
    ]
}
    7. Click Next.
    8. In the Permissions policies, select the previously created Bucket Policy.
    9. Click Next.
    10. Provide a Role name, like netskope-ce-roleAnywhere, and add a Description for the role. 
    11. Click Create role.
    12. Make a note of the Role ARN as this will be required in the Plugin configuration parameter: Role ARN for the authentication method  AWS IAM Roles Anywhere.
    Create a Profile
    1. Select Roles under Access management.
    2. Scroll down to Roles Anywhere and click Manage.
    3. Expand the Setup steps.
    4. Click Step 2: Configure roles.
    5. Click Configure a profile.
    6. Enter a Profile name, like netskope-ce-profile.
    7. Select the role created in Create IAM Role section: netskope-ce-roleAnywhere.
    8. Remove the Inline Policy.
    9. Click Create profile.
    10. Click on the created Profile and copy the Profile ARN.
    Request a Private Certificate
    1. Go to AWS Certificate Manager > Request certificate.
    2. Select Request a private certificate.
    3. Click Next.
    4. Select the Certificate authority created in the previous steps.
    5. Provide a domain name in the Fully qualified domain name field, like netskope-ce.com.
    6. Select RSA 2048 for the Key algorithm.
    7. Add tags if required.
    8. Acknowledge the Certificate renewal permissions.
    9. Click Request.
    10. Go to List certificates from the navigation pane of AWS Certificate Manager.
    11. Select the certificate created previously.
    12. Click Export.
    13. Enter the passphrase. Make a note of the passphrase as it will be required for the Configuration of the AWS S3 Plugin using the AWS IAM Roles Anywhere Authentication method.
    14. Click Generate PEM Encoding.
    15. Download all the Certificates because they won’t be visible again. For new certificates, you will need to Export it again. For More Info visit AWS IAM Role Anywhere

    Deployed on AWS Configuration

    Create a Role
    1. Go to IAM services in the AWS Console.
    2. Click Create role.
    3. Select the AWS Service.
    4. For Use case, select EC2.
    5. Click Next.
    6. Select the permission policy created in your Bucket Policy.
    7. Click Next.
    8. Enter a Role Name, like netskope-ce-instance-role, and add a Description. 
    9. Click Create Role. Note: For this configuration, both Netskope instance and S3 Bucket should be in the same region.
    Assign a Role to an EC2 Instance
    1. Open your EC2 instance console.
    2. Click Instances under Instances.
    3. Go to Action > Security > Modify IAM Role.
    4. Select the Role that you created previously (like netskope-ce-instance-role).
    5. Click Update IAM Role.
    Assign a Role to a Farget Instance
    1. Open your CFN script.
    2. Get the ExistingECSTaskRole Parameter value if present.
    3. Go to IAM > Roles.
    4. Search for the ExistingECSTaskRole parameter value. Otherwise, search for NetskopeCloudExchangeTaskRole-<CFN name> and select the role.
    5. Attach the previously created Create Bucket Policy to this role.
    6. Click Add permission > Attach policies.
    7. Search and select the bucket policy previously created (netskope-ce-s3-policy).
    8. Click Add permissions.
    Assign the Role to a K8s Instance
    1. Open your Role created for ServiceAccount while creating K8s instance.
    2. Attach the Bucket policy created previously.

    Configure the AWS S3 WebTX Plugin

    1. Go to Settings > Plugins.
    2. Search for and select the AWS S3 Webtx box to open the plugin creation page.
    3. Enter the Configuration Name.
    4. Click Next.
    5. Enter these values for the below parameters:
      • Authentication Method: Select the method to be used for authentication (Deployed on AWS/AWS IAM Roles Anywhere)
      • Private Key: Private Key for decrypting the AWS Private CA Certificate. Required for ‘AWS IAM Roles Anywhere’ authentication type.
      • Certificate Body: Certificate Body for AWS Public/Private CA Certificate. Required for ‘AWS IAM Roles Anywhere’ authentication type.
      • Password Phrase: Password Phrase for decrypting the CA Certificate. Required for ‘AWS IAM Roles Anywhere’ authentication type.
      • Profile ARN: AWS Profile ARN for AWS client authentication. Required for ‘AWS IAM Roles Anywhere’ authentication type.
      • Role ARN: AWS Role ARN for AWS client authentication. Required for ‘AWS IAM Roles Anywhere’ authentication type.
      • Trust Anchor ARN: AWS Trust Anchor ARN for AWS client authentication. Required for ‘AWS IAM Roles Anywhere’ authentication type.
      • AWS S3 Bucket Region Name: AWS S3 Bucket Region Name from where to get the AWS S3 Bucket. Make sure that the region name matches the region in the Profile ARN and Trust Anchor ARN.
      • AWS S3 Bucket Name: AWS S3 Bucket Name in which the data object will be stored.
      • Maximum File Size (in MBs): Maximum size of WebTx data object to be stored in the bucket. (Value should be between 1 to 100.)
      • Maximum Duration (in Seconds): Maximum duration after which the WebTx data object should be stored in the bucket.
    6. Click Save. Your new plugin will be available on the Cloud Log Shipper > Plugins page.

    Configure Log Shipper SIEM Mappings for AWS S3 WebTx

    1. Go to Log Shipper > SIEM Mappings and click Add SIEM Mapping.
    2. For source, select the Netskope WebTx configuration.
    3. For destination, select the AWS S3 WebTx configuration.
      image39.png
    4. Click Save.

    Validate the AWS S3 WebTxPlugin

    To validate the plugin workflow in Cloud Exchange.

    1. Go to Logging and Search for ingested events with the filter “message contains ingested”.
    2. The ingested logs will be filtered.

    To validate the push from the AWS S3 platform, follow these steps:

    1. Go to AWS S3. In the Search bar, search for and click on Buckets.
    2. Search the bucket you used while creating the plugin.
    3. Click on the bucket name.

    Note: The folder structure will be like webtx/feedversion/year/month/day/hour/filename.gz

    Example: Webtxdemobucket/webtx/feedname=2.0.0/year=2023/month=11/day=30/hour=9/1701336881_139977276685128.gz

    Troubleshooting the S3 WebTx Plugin

    Facing issues in the existing plugin configuration after the plugin update

    If you’ve recently updated your AWS S3 plugin, there might be an issue where editing the existing plugin configuration leads to a blank CE interface. This problem may occur in CE versions 4.2.0 and 5.0.0 if users try to modify the plugin configuration after the update and navigate to the Configuration Parameters page.

    Here’s what you can do on your CE version:

    What to do:
    For version 4.2.0, 5.0.0: The only solution available is to delete the current plugin configuration and set up a new one from scratch.

    Facing issues while configuring the new plugin

    If you’re creating a new plugin with AWS IAM Roles Anywhere and face an error of 400.

    The root cause for not being able to save the plugin can be found in the logging section when expanding the logs

    What to do:
    Here while expanding the logger, we can see “At least one of the Trust Anchor ARN, Role ARN, and Profile ARN has a different account ID” – so we would have to check that the provided parameters are generated from the same account and have the same region.

Share this Doc

AWS S3 WebTx Plugin for Log Shipper

Or copy link

In this topic ...