AWS Configuration

S3 Configuration

Create an AWS S3 Source Bucket

This bucket will later be used in the AWS S3 Source Bucket configuration parameter while configuring the plugin. This will be the staging bucket where all the logs from Netskope CE will be dumped initially before the lambda processes them and sends them to the Security Lake Bucket. (Referred to as the source bucket in this document.)

1. To create a bucket and set required policies, log in to your AWS console.

   

2. From All Services, select S3, or search for S3 in the search bar.

   

3. Click Create Bucket.

   

4. Enter a Bucket Name (like netskope-ce-source-bucket), and then scroll down and click Create Bucket.

   

5. Search for your bucket and click on the bucket name to open it.

   

6. Click on Permission to open the permission tab and to set the policy.

   

7. Click Edit in the Block public access (bucket settings) section. Uncheck all checkboxes and click Save Changes. When asked for confirmation, confirm it and click Confirm.

   

8. Under the Permissions tab in bucket, click Edit in Bucket Policy section.

   

9. Click Policy Generator.

   

10. Next, select S3 Bucket policy as policy type, enter Add Statement details, and generate a policy. 
    Replace the <user-arn> with the user ARN used for accessing the source bucket and <bucket-name> with the source bucket created above.
    Sample <user-arn>: arn:aws:iam::7111xxxxxxxx:user/xxxxxxxx, and
    <bucket-name>: netskope-ce-source-bucket.
    • Select Type of Policy: S3 Bucket Policy
    • Effect: Allow
    • Principal: <user-arn>
    • Actions:
      • GetBucketAcl
      • GetBucketPolicy
    • ARN: arn:aws:s3:::<bucket-name>
11. Click Add Statement.

    

12. Scroll back up and add another statement.
    • Select Type of Policy: S3 Bucket Policy
    • Effect: Allow
    • Principal: <user-arn>
    • Actions:
      • PutObject
      • PutObjectAcl
    • ARN: arn:aws:s3:::<bucket-name>
13. Click Add Statement.

    

14. Click Generate Policy.

    

15. Make sure to copy the policy generated in this step as it will be used in the next step to attach in the source bucket.

    

16. Add this policy to the Textbox, scroll to the bottom, and click Save Changes.

    

Create a Bucket Policy

A Policy is required for accessing the source bucket by the plugin.

1. Go to Policy Generator. 
   Replace the <bucket-name> with the AWS S3 Source Bucket created previously. (netskope-ce-source-bucket).
   • Select Type of Policy: IAM Policy
   • Effect: Allow
   • AWS Service: Amazon S3
   • Actions: ListAllMyBuckets
   • ARN: *
2. Click Add Statement.

   

3. Scroll back up and add another statement.
   • Select Type of Policy: IAM Policy
   • Effect: Allow
   • AWS Service: Amazon S3
   • Actions:
     • GetBucketAcl
     • GetBucketPolicy
     • GetBucketLocation
   • ARN: arn:aws:s3:::<bucket-name>
4. Click Add Statement.

   

5. Scroll back up and add another statement.
   • Select Type of Policy: IAM Policy
   • Effect: Allow
   • AWS Service: Amazon S3
   • Actions:
     • PutObject
     • PutObjectAcl
   • ARN: arn:aws:s3:::<bucket-name>
6. Click Add Statement.

   

7. Click Generate Policy. Make sure to copy the policy generated in this step as it will be used in the next step to create a policy that will be attached to the user for accessing the source bucket.

   

8. Go to AWS Console and select IAM from All Services.

   

9. Click Policies in the left panel, and then click Create Policy.

   

10. Paste the policy created in the previous step to the JSON tab, and then click Next:Tags and Next:Review.

    

11. Enter a name (like netskope-ce-s3-policy).

    

12. Click Create Policy.

    

13. Attach this policy to the user following these steps:
    1. Go to IAM > Users. Select the user on which you want to attach a policy, and click Add permissions, and then click Add permissions from the dropdown.

       

    2. Select Attach policies directly under Permissions, and then search for and select the policy created in the previous step (netskope-ce-s3-policy).

       

    3. Click Next and then click Add permissions. A Policy will be attached to the user.

       

CFN Configuration

AWS CloudFormation is a service that helps you model and set up your AWS resources. Our main goal is to convert the logs/data received from Netskope CE to Parquet. To achieve this we need to create some resources such as lambda, eventbridge etc., for which we will use this CloudFormation service.

Security Lake Bucket aspects parquet files either in an interval of every 5 minutes or as soon as the data size reaches 256 MB. This can not be achieved by the current core architecture so to achieve this the cloud formation stack creates a lambda function which converts the data dumped in the AWS S3 Source Bucket by the plugin into parquet files which will be stored in the Security Lake Bucket (Destination Bucket).

Purpose

We will be using the CloudFormation service to create the required resources in order to fetch the objects from the AWS S3 Source Bucket (where Netskope CE dumps the data), merge those objects’ contents (to achieve the file size constraint), and create a Parquet file of the same which will be dumped in the Destination Bucket (Security Lake Bucket).

Create a CFN Policy

Create a Policy using  Policy Generator containing access to CloudFormation, CloudWatch Logs, Lambda, IAM, EventBridge and Lake Formation. The <Resource ARN> signifies the user ARNs you need to give permissions to.(IAM User ARN, SNS topic or VPC).

Replace the <Resource ARN> with the specific resource Amazon Resource Name being used. Replace the <region_name> with the region where you will be deploying the cloud formation stack and <user_name> with the user that will be deploying the cloud formation stack
(arn:aws:cloudformation:us-east-2:7111xxxxxxxx:user/tanushree.kurup/*).

1. Go to Policy Generator.
   • Select Type of Policy: IAM Policy
   • Effect: Allow
   • AWS Service: AWS CloudFormation
   • Actions:
     • CreateStack
     • CreateUploadBucket
     • DeleteStack
     • DescribeStackEvents
     • DescribeStacks
     • GetStackPolicy
     • GetTemplateSummary
     • ListStacks
     • RollbackStack
     • UpdateStack
   • ARN: <Resource ARN> (arn:aws:cloudformation:<region_name>:7111xxxxxxxx:user/<user_name>/*)
2. Click Add Statement.

   

3. Scroll back up and add another statement.
   • Select Type of Policy: IAM Policy
   • Effect: Allow
   • AWS Service: AWS Lambda
   • Actions:
     • AddPermission
     • CreateFunction
     • DeleteFunction
     • DeleteFunctionEventInvokeConfig
     • DeleteLayerVersion
     • GetAccountSettings
     • GetEventSourceMapping
     • GetFunction
     • GetFunctionConcurrency
     • GetFunctionEventInvokeConfig
     • GetLayerVersion
     • GetPolicy
     • GetRuntimeManagementConfig
     • InvokeFunction
     • ListEventSourceMappings
     • ListFunctionEventInvokeConfigs
     • ListFunctions
     • ListTags
     • PublishLayerVersion
     • PutFunctionEventInvokeConfig
     • RemovePermission
     • TagResource
     • UpdateFunctionCode
     • UpdateFunctionConfiguration
     • UpdateFunctionEventInvokeConfig
   • ARN: <Resource ARN> (arn:aws:lambda:<region_name>:7111xxxxxxxx:user/<user_name>/*)
4. Click Add Statement.

   

5. Scroll back up and add another statement.
   • Select Type of Policy: IAM Policy
   • Effect: Allow
   • AWS Service: AWS Identity and Access Management (IAM)
   • Actions:
     • CreateRole
     • DeleteRole
     • DeleteRolePolicy
     • GetRole
     • GetRolePolicy
     • ListRoles
     • PassRole
     • PutRolePolicy
     • TagRole
   • ARN: <Resource ARN> (arn:aws:iam::7111xxxxxxxx:user/<user_name>/*)
6. Click Add Statement.

   

7. Scroll back up and add another statement.
   • Select Type of Policy: IAM Policy
   • Effect: Allow
   • AWS Service: Amazon CloudWatch Logs
   • Actions:
     • GetLogEvents
     • FilterLogEvents
     • DescribeLogStreams
     • DescribeLogGroups
   • ARN: <Resource ARN>, (arn:aws:logs:<region_name>:711162259376:user/<user_name>/*)
8. Click Add Statement.

   

9. Scroll back up and add another statement.
   • Select Type of Policy: IAM Policy
   • Effect: Allow
   • AWS Service: Amazon EventBridge
   • Actions:
     • ListRules
     • RemoveTargets
     • ListTargetsByRule
     • PutEvents
     • DescribeRule
     • CreateEventBus
     • DescribeEventBus
     • PutRule
     • ListEventBuses
     • TagResource
     • PutTargets
     • DeleteRule
     • DeleteEventBus
   • ARN: <Resource ARN> (arn:aws:events:<region_name>:7111xxxxxxxx:user<user_name>/*)
10. Click Add Statement.

    

11. Scroll back up and add another statement.
    • Select Type of Policy: IAM Policy
    • Effect: Allow
    • AWS Service: Amazon SNS
    • Actions:
      • sns:GetTopicAttributes
      • sns:List*
    • ARN: <Resource ARN> (arn:aws:sns:<region_name>:7111xxxxxxxx:user/<user_name>/*)
12. Click Add Statement.

    

13. Scroll back up and add another statement.
    • Select Type of Policy: IAM Policy
    • Effect: Allow
    • AWS Service: Amazon Glue
    • Actions:
      • All Actions
    • ARN: <Resource ARN> (arn:aws:glue:<region_name>:7111xxxxxxxx:user/<user_name>/*)
14. Click Add Statement.

    

15. Click Generate Policy. Make sure to copy the policy generated in this step as it will be used in the next step to create a policy that will be attached to the user for creating the Cloud Formation Stack.

    

16. Go to your AWS Console and select IAM from All Services.

    

17. Click Policies in the left panel, and then click Create Policy.

    

18. Paste the policy created in the previous step to the JSON tab. Click Next:Tags and Next:Review.

    

19. Enter a name (like netskope-ce-cfn-policy) and click Create Policy.

    

Attach this Policy to the User

1. Go to IAM > Users. Select the user on which you want to attach a policy, and click on Add permissions and click on Add permissions options.
   

   

2. Select Attach policies directly under Permissions, and then search for and select the policy created in the previous step for source bucket.
   

   

3. Click Next and then click Add permissions. The Policy will be attached to the user.
   

   

Configure a Custom Source

1. Search for Security Lake in the AWS console search.
2. Select Custom Source from the Amazon Security Lake navigation menu.
   

   

3. Click Create custom source.
4. Enter a Data source name (with suffix as Netskope tenant name): ncte-<tenant-name>
   Replace the <tenant-name> with your Netskope Tenant Name. Note: The Custom Source name can only contain alphanumeric characters and/or underscore and dashes. If your tenant name contains any other special character, replace it with “-”
   like, ncte-crest-support-de.
5. Select Security Findings for the Event class.
6. Enter an Account ID: The Account where the CloudFormation stack will be deployed.
7. Enter an External ID: The unique string assigned to a customer in the partner service’s identity management system, like ncte-externalid-<tenant-name>.
   Replace the <tenant-name> with your Netskope tenant name, like  ncte-externalid-crest-support-de.
   

   

8. Create a new Service role, or use one that is already created.
   If Create is selected, then two new roles will be created named as:
   AmazonSecurityLake-Provider-<custom-source>-<region-name>
   AmazonSecurityLakeCustomDataGlueCrawler-<custom-source>
   Here the <custom-source> represents the Custom source name provided earlier.(In this case, ncte-crest-support-de).
   

   

9. Click Create.
   

   

10. Attach the policy in the AmazonSecurityLakeCustomDataGlueCrawler-<custom-source> role to your user to access the Security Lake Bucket. To attach a policy to a user follow the steps:

    1. Go to IAM > Users. Select the IAM user that you want to attach the policy to. Click Add Permissions and select Add permissions from the dropdown.
    2. Search the policy attached to the Role created with the name: AmazonSecurityLakeCustomDataGlueCrawler-<custom-source>Policy.
    3. Select the policy.
       

       

    4. Click Next and then click Add permissions.

Get the YML File

1. Go to https://github.com/netskopeoss/ta_cloud_exchange_plugins.
2. Clone the Repo.
3. Open the amazon_security_lake > cfn folder.
4. Inside that there will be an yaml file: AWSSecurityLake.yaml. Copy the file.

Configure YML

1. Sign in to your AWS Account.
   

   

2. Search for CloudFormation. and click on the Service named CloudFormation.
   

   

3. Click Create stack > With new resources (standard).
   

   

4. The Create stack page opens.
5. In the Create stack page Prerequisite – Prepare template section, select Template is ready.
   

   

6. Again, in the Create stack page Specify template section, select Upload a template file, and then browse the yml file (that you got from the steps Get YML File).
   

   

7. Click Next.
8. The Specify stack details page opens
9. In the Specify stack details page Stack name section, enter a name, like netskope-ce-cfn.
   

   

10. Again, in the Specify stack details page Parameter section, enter the values of these parameters.
    

    

    • Prefix for Parquet file: It will be added as a prefix to the Object key name while uploading the parquet to ASL.
    • Source Bucket Name: AWS S3 Source Bucket name that contains the logs sent from Netskope CE. Use the bucket name created in S3 Configuration.
    • Destination Bucket Name: Bucket name where the parquet files will be dumped. The bucket name is displayed as a Location present in Configure a Custom Source.

11. Click Next.
12. The Configure stack options page opens. Here you can keep all the default values.
    

    

13. Click Next.
    

    

14. Acknowledge the conditions and click Submit.
    

    

15. You will see the screen below.
    

    

16. As seen below, you will see the Create Complete message in green. The stack is successfully created with a lambda function named AWSSecurityLakeProcessParquet-<stack name>-<region name>, like AWSSecurityLakeProcessParquet-netskope-ce-cfn-us-east-2.