Netskope Help

AWS S3 Plugin for Log Shipper

This document explains how to configure your AWS S3 integration with the Log Shipper module of the Netskope Cloud Exchange platform. This integration allows pushing web transactions into AWS S3 buckets.

Prerequisites
  • A Netskope Tenant (or multiple, for example, production and development/test instances).

  • A Netskope Cloud Exchange tenant with the Log Shipper module already configured.

  • AWS S3 credentials with create/read/write buckets permissions. Obtain your AWS S3 Access key ID and Secret Access Key with permission to read/write on specific buckets or permission to create buckets before proceeding.

Note

Verify your bucket permissions are secure and not set up for open public access. Only allow access to your cloud storage instance from your Cloud Exchange Host and any other addresses that need access.

Workflow
  1. Create a bucket, an IAM user and policy, and obtain the required credentials.

  2. Configure the AWS S3 plugin.

  3. Configure Log Shipper Business Rules.

  4. Configure Log Shipper SIEM mappings.

  5. Validate the plugin.

Create an AWS S3 Bucket
  1. To create bucket and set required policies, go to your AWS console and log in.

    image1.png
  2. From All Services, select S3.

    image2.png
  3. Click Create Bucket.

    image3.png
  4. Enter Bucket Name, scroll to bottom, and click Create Bucket.

    image4.png
  5. Search for your bucket and click on the bucket name to open it.

    image5.png
  6. Click on Permission to open the permission tab and to set a policy.

    image6.png
  7. Click Edit. Under Block public access (bucket settings), uncheck all checkboxes and click Save Changes. When prompted for a confirmation, confirm it, and then click Confirm.

    image7.png
  8. Under the Permissions tab, click Edit in the Bucket Policy section.

    image8.png
  9. Copy the Bucket ARN and click Policy Generator.

    image9.png
  10. Select S3 Bucket policy as policy type, in the Principal field, put in your user arn (refer to the prerequisites), select GetBucketACL and GetBucketPolicy for Actions, paste in the ARN that you copied from the last step (add /* at the end, see example below) and Add Statement.

    10.png
  11. You will add another Statement - Select S3 Bucket policy as policy type, in the Principal field put in your user arn (refer to the prerequisites), select PutObject and PutObjectACL for Actions, paste in the ARN with adding the /* again, Add Statement details and Generate policy. Copy the output.

    11.png

    Example of adding /* at the end of the bucket name.

    11a.png
  12. Policy required to access AWS S3 bucket.

    {
            "Id": "<policy>",
            "Version": "<version>",
            "Statement": [
                {
                    "Sid": "<statement ID>",
                    "Effect": "Allow",
                    "Principal": {
                        "AWS": "<user-arn>"
                    },
                    "Action": [
                        "s3:GetBucketAcl",
                        "s3:GetBucketPolicy"
                    ],
                    "Resource": "arn:aws:s3:::<bucket-name>"
                },
                {
                    "Sid": "<statement ID>",
                    "Effect": "Allow",
                    "Principal": {
                        "AWS": "<user-arn>"
                    },
                    "Action": [
                        "s3:PutObject",
                        "s3:PutObjectAcl"
                    ],
                    "Resource": "arn:aws:s3:::<bucket-name>/*"
                }
            ]
        }
    }
  13. Add copied policy to the textbox, scroll to bottom and click on Save Changes.

    image11.png
Create an IAM Policy
  1. Go to Policy Generator , select IAM Policy as the policy type, and then generate policy. Be sure to add the /* after the second bucket name.

    {
        "Policy": {
            "Version": "<version>",
            "Statement": [
                {
                    "Sid": "<statement ID>",
                    "Effect": "Allow",
                    "Principal": {
                        "AWS": "<user-arn>"
                    },
                    "Action": [
                        "s3:GetBucketAcl",
                        "s3:GetBucketPolicy"
                    ],
                    "Resource": "arn:aws:s3:::<bucket-name>"
                },
                {
                    "Sid": "<statement ID>",
                    "Effect": "Allow",
                    "Principal": {
                        "AWS": "<user-arn>"
                    },
                    "Action": [
                        "s3:PutObject",
                        "s3:PutObjectAcl"
                    ],
                    "Resource": "arn:aws:s3:::<bucket-name>/*"
                }
            ]
        }
    }
  2. Go to AWS Console and select IAM from All Services. Click Policies in the left panel, and then click Create Policy.

    image12.png
  3. Copy that policy to the JSON tab. Click Next:Tags, and then click Next:Review.

    image13.png
  4. Enter a Name and click Save Changes.

    4.png
Create IAM User
  1. To Create an IAM User, and get the Access ID and Secret Access Key, go to the AWS Console and select IAM from All Services.

    image2.png
  2. Click Users in the left panel.

    image15.png
  3. Click Add Users.

    image16.png
  4. Enter a User name, check the Access key - programmatic Access checkbox, and then click Next:Permissions.

    image17.png
  5. Click Attach existing policies directly, and then click on Create Policy.

    image18.png
  6. Search for your policy name that you created and then click on Next:Tags > Next:Reviews > Create User.

    image19.png
  7. Copy the Access Key ID and Secret Access Key.

    image20.png
To Set Policy from the AWS CLI

The account that owns the access keys should have the below policy.

To set a bucket policy from the AWS CLI
aws s3api put-bucket-policy --bucket MyBucket --policy file://policy.json

policy.json:

{
    "Policy": {
        "Version": "<version>",
        "Statement": [
            {
                "Sid": "<statement ID>",
                "Effect": "Allow",
                "Principal": {
                    "AWS": "<user-arn>"
                },
                "Action": [
                    "s3:GetBucketAcl",
                    "s3:GetBucketPolicy"
                ],
                "Resource": "arn:aws:s3:::<bucket-name>"
            },
            {
                "Sid": "<statement ID>",
                "Effect": "Allow",
                "Principal": {
                    "AWS": "<user-arn>"
                },
                "Action": [
                    "s3:PutObject",
                    "s3:PutObjectAcl"
                ],
                "Resource": "arn:aws:s3:::<bucket-name>/*"
            }
        ]
    }
}

To set an IAM policy from the AWS CLI
aws iam put-user-policy --user-name <username> --policy-name <policy_name> --policy-document file://iam_policy.json
iam_policy.json:

{
    "Policy": {
        "Version": "<version>",
        "Statement": [
            		     {
                "Sid": "<statement ID>",
                "Effect": "Allow",
                "Action": "s3:ListAllMyBuckets",
                "Resource": "*"
            },
            {
                "Sid": "<statement ID>",
                "Effect": "Allow",
                "Action": [
                    "s3:GetBucketAcl",
                    "s3:GetBucketPolicy"
                ],
                "Resource": "arn:aws:s3:::<bucket-name>"
            },
            {
                "Sid": "<statement ID>",
                "Effect": "Allow",
                "Action": [
                    "s3:PutObject",
                    "s3:PutObjectAcl"
                ],
                "Resource": "arn:aws:s3:::<bucket-name>/*"
            }
        ]
    }
}
  1. Go to Settings > Plugins.

  2. Select the AWS S3 box to open the plugin creation dialog.

  3. Enter a Configuration Name.

  4. Click Next.

    image1.png
  5. Enter your AWS Access Key ID, AWS Secret Access Key, and select a Region Name. Enter a Bucket Name, Object Prefix, Maximum File Size, and Maximum Duration.

    image2.png

    Object Prefix: Will be added to the Object key name while ingesting web transactions to the AWS S3 bucket.)

    Maximum File Size and Maximum Duration: Whichever hits first, data will be exported according to that.)

  6. Click Save.

    image3.png
  1. Go to Log Shipper > Business Rules.

    image4.png
  2. Click Create New Rule.

    image5.png
  3. Enter a Rule Name and select the filters to use.

  4. Click Save.

    image6.png
  1. Go to Log Shipper > SIEM Mappings and click Add SIEM Mapping.

  2. Select a Source Configuration, Business Rule, and Destination Configuration.

    image7.png
  3. Click Save.

To validate the plugin workflow, you can check from Netskope Cloud Exchange and from AWS CLI.

To validate from Netskope Cloud Exchange,

  • Go to Logging.

    image8.png

To validate from the AWS CLI.

  1. Install AWS CLI in your machine.

    (Reference: https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html)

  2. Configure AWS CLI in your machine.

    (Reference: https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-quickstart.html)

  3. Use these commands to view buckets and objects.

  4. To list all the buckets.

    aws s3 ls
  5. To list all the objects in a bucket.

    aws s3 ls s3://<bucket_name>
  6. To download an object from a bucket to a local machine.

    aws s3 cp s3://<bucket_name>/<object_path> <local_path>