Databricks Plugin for Log Shipper

This document explains how to configure the Databricks v1.0.0 plugin with the Log Shipper module of the Netskope Cloud Exchange platform. This plugin is used to send Netskope Alerts (DLP, Malware, Policy, Compromised Credential, Malsite, Quarantine, Remediation, Security Assessment, Watchlist, UBA, CTEP, Device, Content), Events (Page, Application, Audit, Infrastructure, Network, Incident, Endpoint, Client Status) and WebTx data to an AWS S3 bucket. This plugin supports ingestion in JSON format only. To view the data on databricks, make sure the S3 bucket is linked with the Databricks External Location.

Prerequisites

To complete this configuration, you need:

• A Netskope tenant (or multiple, for example, production and development/test instances).
• A Netskope Cloud Exchange instance with the Tenant plugin and Log Shipper plugin already configured.
• Already configured AWS Netskope LogStreaming Plugin or Azure Log Streaming Plugin for pulling Webtx logs.
• A Databricks Workspace instance linked with an AWS account and Unity Catalog enabled.
• An AWS S3 bucket with the following IAM permissions granted to the authenticating role: s3:ListBucket, s3:GetBucketLocation, s3:PutObject.
• An AWS S3 bucket linked with Databricks External Location.

Databricks Plugin Support

This plugin is used to send Netskope Alerts (DLP, Malware, Policy, Compromised Credential, Malsite, Quarantine, Remediation, Security Assessment, Watchlist, UBA, CTEP, Device, Content), Events (Page, Application, Audit, Infrastructure, Network, Incident, Endpoint, Client Status) and WebTx data to an AWS S3 bucket. This plugin supports ingestion in JSON format only. To view the data on databricks, make sure the S3 bucket is linked with the Databricks External Location.

Permissions

These permissions are needed for the plugin configuration:

• s3:ListBucket on the target S3 bucket
• s3:GetBucketLocation on the target S3 bucket
• s3:PutObject on the target S3 bucket

{
  "durationSeconds": 900,
  "profileArn": "{profile_arn}",
  "roleArn": "{role_arn}",
  "sessionName": "Session",
  "trustAnchorArn": "{trust_anchor_arn}"
}

{
  "credentialSet": [
    {
      "credentials": {
        "accessKeyId": "ASIA...",
        "secretAccessKey": "...",
        "sessionToken": "...",
        "expiration": "2024-01-01T00:15:00Z"
      }
    }
  ]
}

s3_client = boto3.client(
    "s3",
    aws_access_key_id=self.aws_public_key,
    aws_secret_access_key=self.aws_private_key,
    aws_session_token=self.aws_session_token,
    region_name=self.configuration.get("region_name", "").strip(),
    config=Config(
        proxies=self.proxy,
        user_agent=self.useragent,
    ),
)

s3_client.head_bucket(Bucket=bucket_name)

location = s3_client.get_bucket_location(Bucket=bucket_name)
bucket_region = location.get("LocationConstraint") or "us-east-1"

s3_client.upload_file(
    file_name,      # local temp file path
    bucket_name,    # S3 bucket name
    object_name,    # S3 object key
)

Performance Matrix

This performance reading was conducted on a Large Stack in Cloud Exchange with these VM specifications. These readings are added with the consideration that it will ingest around 15k Netskope alerts/events in 5 seconds to the destination.

User Agent

APN/1.1 (ahq9d89xj9gspapczzdb59goq)

Workflow

1. Get credentials for configuring the Databricks plugin
2. Configure the Databricks plugin.
3. Configure a Business Rule.
4. Configure Log Delivery.
5. Validate the plugin.

Watch a Video

Click play to watch a video.

Configuration on AWS

Create a Policy

1. Log in to your AWS account, go to IAM > Policies and then click Create Policy.

   

2. Select json and paste this policy in it.

   {
   	"Version": "2012-10-17",
   	"Statement": [
   		{
   			"Sid": "VisualEditor0",
   			"Effect": "Allow",
   			"Action": [
   				"s3:GetBucketLocation",
   				"s3:PutObject",
   				"s3:ListBucket"
   			],
   			"Resource": "*"
   		}
   	]
   }

   

3. Enter a name and description.

   

4. Click Create Policy.

   

Deployed on AWS authentication

Create a Role

1. Go to IAM Services in the AWS Console.

2. Go to the Roles page under Access management and click Create Role.

   

3. Select the AWS Service.

4. Under Use case, select EC2.

5. Click Next.

   

6. Select the permission policy created in Create a Policy.

   

7. Click Next.

8. Enter a Role Name and Description. 

   

9. Click Create Role.

   

Assign a Role to EC2 Instance

1. Open your EC2 instance console.

2. Go to the Instances page and select the required instance where Cloud Exchange is deployed.

   

3. Go to Action > Security > Modify IAM Role.

   

4. Select the Role that you created above in Create a Role.

5. Click Add IAM Role/Update IAM Role. Note that both the EC2 instance and S3 bucket should be in the same region.

   

   Restart your EC2 instance if you get permission related errors even after updating the IAM role for the EC2 instance.

IAM Role Anywhere authentication

Prerequisites

The AWS Certificate Manager service is required to be enabled to authenticate the plugin using the AWS IAM Roles Anywhere authentication method.

Make sure you create the Private Certificate Authority, Trust Anchor, and Profile in the same region that your AWS S3 bucket resides.

Create a Policy

This Policy contains the required permissions for creating a Private CA Certificate(including Permissions for creating Trust Anchor and Profile) and using the IAM Roles Anywhere.

1. Go to Policy Generator and select IAM Policy as the policy type and generate policy.

   1. Select Type of Policy: IAM Policy

   2. Effect: Allow

   3. AWS Service: AWS Private Certificate Authority

   4. Actions:

      • CreateCertificateAuthority
      • DescribeCertificateAuthority
      • GetCertificate
      • GetCertificateAuthorityCertificate
      • GetCertificateAuthorityCsr
      • ImportCertificateAuthorityCertificate
      • IssueCertificate
      • ListCertificateAuthorities

   5. ARN: *

   6. Click Add Statement.

      

   7. Select Type of Policy: IAM Policy

   8. Effect: Allow

   9. AWS Service: AWS Identity and Access Management (IAM)

   10. Actions:

       • AttachRolePolicy
       • CreateAccessKey
       • CreateRole
       • DeleteRole
       • PassRole

   11. ARN: *

   12. Click Add Statement.

       

   13. Select Type of Policy: IAM Policy

   14. Effect: Allow

   15. AWS Service: AWS Certificate Manager

   16. Actions:

       • DescribeCertificate
       • ExportCertificate
       • GetCertificate
       • ListCertificates
       • ListTagsForCertificate
       • RequestCertificate

   17. ARN: *

   18. Click Add Statement.

       

   19. Select Type of Policy: IAM Policy

   20. Effect: Allow

   21. AWS Service: AWS Identity and Access Management Roles Anywhere

   22. Actions:

       • CreateProfile
       • CreateTrustAnchor
       • GetProfile
       • GetTrustAnchor
       • ListProfiles
       • ListTrustAnchors

   23. ARN: *

   24. Click Add Statement.

       

   25. Click Generate Policy.

       

   26. Copy the Policy as it will be used in the next step for creating the policy required for creating the Private CA certificates.

2. Go to the AWS Console and select IAM from All Services. Click Policies in the left panel, and then click Create Policy.

   

3. Paste the policy generated in the previous section to the JSON tab. Click Next: Tags and Next: Review.

   

4. Enter a Name and click Save Changes.

   

Create a Private Certificate Authority

1. Log in to the AWS Console.

2. Search for Certificate Manager.

   

3. Click AWS Private CA.

4. Click Create a private CA.

   

5. For Mode Options, select General-purpose, and for CA type options, select Root.

   

6. Enter the Organization (O).

   

7. For Key algorithm options, select RSA 2048.

   

   

8. Add tags if any (optional).

9. Enable the checkbox in the CA permissions options section.

10. Enable the checkbox in the Pricing section.

11. Click Create to create the CA certificate.

    

    

12. For Actions, select Install.

    

13. Click Confirm and Install.

    

    

Create a Trust Anchor

1. Search for IAM Service and go to Roles under Access management. Scroll down to Roles Anywhere and select Manage.

   

2. Click Create a trust anchor.

   

3. Enter a Trust anchor name.

   

4. Select the AWS Certificate Manager Private CA(created in the previous steps) as a Certificate authority (CA) source.

5. Select  the certificate that you created under AWS Certificate Manager Private CA.

6. Add tags if required.

7. Click Create a trust anchor.

   

   

8. Click on the created Trust Anchor and copy the Trust Anchor ARN. This will be required in the plugin Trust Anchor ARN configuration parameter for the AWS IAM Roles Anywhere authentication method.

   

Create an IAM Role

1. Go to IAM Services in the AWS Console.

2. Click Role in the Access management submenu.

3. Click Create Role.

   

4. For the Trusted entity type, select Custom Trust Policy.

   

5. Replace the Custom Trust Policy with this Trust Policy. This policy contains the permissions for using the Roles Anywhere service:

   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Effect": "Allow",
               "Principal": {
                   "Service": [
                       "rolesanywhere.amazonaws.com"
                   ]
               },
               "Action": [
                   "sts:AssumeRole",
                   "sts:TagSession",
                   "sts:SetSourceIdentity"
               ]
           }
       ]
   }

   

6. Click Next.

7. In the Permissions policies, select the created policy from the Create a policy section previously.

   

8. Click Next.

9. Enter a Role name and Description for the role.

   

10. Click Create role.

11. Search for the created role and click on the Role Name.

    

12. Make a note of the Role ARN because this will be required in the plugin Role ARN configuration parameter for the AWS IAM Roles Anywhere authentication method.

    

Create a Profile

1. Go to IAM Services in the AWS Console.

2. Select Roles present under Access management.

3. Scroll down to Roles Anywhere and click Manage.

   

4. Expand the Setup steps.

5. Click Step 2: Configure roles.

6. Click Configure a profile.

   

7. Enter a Profile name. 

8. Select the role created in the Create IAM Role step.

   

9. Remove the Inline Policy.

   

10. Click Create profile.

    

    

11. Click on the reated Profile and copy the Profile ARN.

    

Request a Private Certificate

1. Go to AWS Certificate Manager > Request certificate.

2. Select Request a private certificate.

   

3. Click Next.

4. Select the Certificate authority created in the previous steps.

   

5. Provide a domain name in the Fully qualified domain name field. For example: netskope-ce.com.

6. For Key algorithm, select RSA 2048.

   

7. Add tags if required.

8. Acknowledge the Certificate renewal permissions.

9. Click Request.

   

10. Go to List certificates from the navigation pane of AWS Certificate Manager.

11. Select the certificate created previously.

    

12. Click Export.

    

13. Enter the passphrase. Make a note of the passphrase as it will be required for the configuration of the AWS S3 Plugin using the AWS IAM Roles Anywhere authentication method.

14. Click Generate PEM Encoding.

    

15. Download all the Certificates because they won’t be visible again. For new certificates, you will need to Export them again.

    

16. Copy the Certificate body and Certificate Private Key, because they will be required for the Configuration of the AWS S3 Plugin using the AWS IAM Roles Anywhere Authentication method. For More Info visit AWS IAM Role Anywhere

Link an AWS S3 Bucket to a Databricks External Location

1. Log in to Databricks and go to Workspace.

   

2. In the top search bar, enter External Locations and select it from the results.

   

3. Click Create external location. 

   

4. Select AWS Quickstart and click Next.

   

5. Provide the bucket name in this format s3://<bucket-name>, and click Generate new token.

   

6. Copy the generated token, and then click Launch in Quickstart.

   

7. This will redirect you to the AWS console. Enter the Personal Access Token in the Databricks Personal Access Token field. 

   

8. Check the Acknowledgement, and then click Create stack.

   

9. After it is done, you will be able to see the external location on the Databricks after 10 minutes to 15 minutes.

   

Configure the Databricks Plugin

1. In Cloud Exchange, go to Settings > Plugin Store. Search for and select the Databricks v1.0.0 (CLS) plugin.

   

2. Enter the Basic Information:

   • Configuration Name: Provide a plugin configuration name.
   • Mapping: Choose the mapping configuration to be used. You can add a new one from Settings > Log Shipper.
   • Format: Choose the format of data you want to transform. When JSON is selected, raw JSON logs will be sent. The ingestion may be affected if the Log Delivery Configuration does not accept raw log format.

   

3. Click Next and enter the Configuration Parameters, depending on the authentication method:

   • Deployed on AWS authentication:

     • Authentication Method: Select the method to be used for AWS client authentication. ‘Deployed on AWS’ uses the IAM instance profile or role attached to the AWS environment. ‘AWS IAM Roles Anywhere’ uses an X.509 certificate and private key to obtain temporary AWS credentials — use this when CE is running outside AWS.
     • AWS S3 Bucket Region Name: AWS region where the target S3 bucket resides. Make sure the region matches the region in the Profile ARN and Trust Anchor ARN when using IAM Roles Anywhere.
     • AWS S3 Bucket Name: Name of the target AWS S3 bucket where Netskope Alerts and Events will be stored. This bucket must be linked with the Databricks External Location.
       Example: netskope-alerts-bucket.

     

   • IAM Roles Anywhere authentication:

     • Authentication Method: Select the method to be used for AWS client authentication. ‘Deployed on AWS’ uses the IAM instance profile or role attached to the AWS environment. ‘AWS IAM Roles Anywhere’ uses an X.509 certificate and private key to obtain temporary AWS credentials. Use this when CE is running outside AWS.
     • Private Key: PEM-encoded private key used to decrypt the AWS Private CA Certificate. Required for AWS IAM Roles Anywhere authentication.
     • Certificate Body: PEM-encoded X.509 certificate body issued by your AWS Private or Public CA. Required for AWS IAM Roles Anywhere authentication.
     • Password Phrase: Passphrase used to decrypt the CA certificate if it is encrypted. Required for AWS IAM Roles Anywhere authentication.
     • Profile ARN: ARN of the IAM Roles Anywhere profile. Format: arn:aws:rolesanywhere:{region}:{account-id}:profile/{profile-id}. Required for AWS IAM Roles Anywhere authentication.
     • Role ARN: ARN of the IAM role to be assumed. Format: arn:aws:iam::{account-id}:role/{role-name}. Required for AWS IAM Roles Anywhere authentication.
     • Trust Anchor ARN: ARN of the IAM Roles Anywhere trust anchor. Format:arn:aws:rolesanywhere:{region}:{account-id}:trust-anchor/{anchor-id}. Required for AWS IAM Roles Anywhere authentication.
     • AWS S3 Bucket Region Name: AWS region where the target S3 bucket resides. Make sure the region matches the region in the Profile ARN and Trust Anchor ARN when using IAM Roles Anywhere.
     • AWS S3 Bucket Name: Name of the target AWS S3 bucket where Netskope Alerts and Events will be stored. This bucket must be linked with the Databricks External Location. Example: netskope-alerts-bucket.

     

     

4. Click Save.

   

Configure a Log Shipper Business Rule for Databricks

1. In Log Shipper, go to Business Rules and click Add Business Rule.

2. By default, there’s a business rule that filters all alerts and events. If you want to filter out any specific type of alert or event, click Create New Rule and configure a new business rule by adding the rule name and filter(s).

   

3. Click Save.

Configure Log Shipper Log Delivery for the Databricks Plugin

1. In Log Shipper, go to Log Delivery and click Add Log Delivery Configuration.

2. Select a Source Configuration, Destination Configuration, and Business Rule.

   

3. Click Save.

   

Validate the Databricks Plugin

Validate the Pull

In Cloud Exchange, go to Logging and search for the pulled logs.

Validate the Push

To validate the plugin workflow in Cloud Exchange, go to Logging and search for ingested Events and Alerts with the filter: message contains “ingested” or message contains “<destination plugin configuration name>”.

Validate the Push on Databricks

Log in to the Databricks instance and go to the Data Ingestion page.

Click Amazon S3 under Databricks connectors. Select the External location used while configuring the plugin on Cloud Exchange.

Sample ingested alerts:

Sample ingested events:

Sample ingested Webtx logs:

Troubleshooting the Databricks Plugin

Unable to configure the plugin

It may be due to one of these reasons:

• Role not assigned to the EC2 instance where Cloud Exchange is hosted in case of Deployed on AWS authentication.
• Incorrect credentials in case of IAM roles anywhere authentication.

What to do:

1. In case you are using Deployed on AWS authentication then refer to the Deployed on AWS authentication section
2. In case you are using IAM roles anywhere authentication then refer to the IAM Role Anywhere authentication section.

AWS S3 bucket region mismatch

If you see the error AWS S3 bucket exists but is in region ‘{actual}’, not the configured region ‘{configured}’, it could be due to the AWS S3 Bucket Region Name in the plugin configuration does not match the bucket’s actual region.

What to do:

1. In the AWS S3 Console, go to the target bucket and verify its region under the bucket properties.
2. Update the AWS S3 Bucket Region Name in the plugin configuration to match the actual bucket region and save the configuration.

Known Behavior

• When using the AWS IAM Roles Anywhere authentication method, proxy settings configured in Cloud Exchange are not forwarded to the IAM Roles Anywhere credential requests. The plugin attempts a direct connection to rolesanywhere.{region}.amazonaws.com. Ensure direct connectivity to this endpoint is available when using IAM Roles Anywhere behind a proxy.
• The Cloud Exchange SSL Validation setting is applied only to Databricks API calls. IAM Roles Anywhere credential requests and AWS S3 (boto3) calls always use the library default SSL verification and are not affected by the Cloud Exchange SSL validation setting.