Cloud Exchange Troubleshooting

Scenario: In older versions of CE user might start seeing errors related to the importing of plugins continuously.

Impact: CE is functioning correctly but import errors are continuously getting logged in CE Audit logs.

Resolution: There is no functional impact of this error, so you can safely ignore this error.

Error observed while configuring the tenant in Cloud Exchange

- Error: Value error, please check the Tenant Name field has no special characters, ensure V1 and V2 tokens are valid.

Troubleshooting Steps

1. First, make sure that REST API Status is enabled for REST API v2 in the Netskope tenant.
   
2. Confirm that you have created the REST API v2 Token with the necessary Netskope Endpoint Permissions/Scopes as documented here.
3. Check your existing REST API v2 Token, and if required, modify the Netskope Endpoint Permissions/Scopes as per the above documentation. Try again to save the Netskope Tenant in the Cloud Exchange UI.
4. If the issue continue to persists, execute the following curl request in the host machine wherein Cloud Exchange is deployed:

   Note

   Before executing the above command, please make sure to replace the <tenant-name> with Netskope Tenant Name and <V2 token> with the existing REST API v2 token.

   curl --location https://<tenant-name>.goskope.com/api/v2/events/dataexport/events/page?index=tenant --header 'Netskope-Api-Token: <V2 token>'

Command Output
If you observed the output of the command as Unauthorized for the IP address x.x.x.x, then add the highlighted IP address in the Custom IP Addresses section (enable it, if disabled) under IP Allowlist in your Netskope tenant.

Path for IP Allowlist

Log in to the Netskope tenant and go to Settings > Administration > IP Allowlist > Custom IP Addresses.

Scenario: After migrating to 5.1.0, the CE core container is getting restarted and following errors are observed in the core container logs.

Impact: CE is not accessible.

Resolution: Reach out to Netskope Support team with the diagnostic logs.

Get errors like InvalidReplicaSetConfig while running ./start script while deploying Cloud Exchange with High Availability

During the initialization of a MongoDB replica set, an error message may surface, indicating: “No host described in new configuration with {version: 1, term: 0} for replica set mongo_replic_set maps to this node.” This issue commonly arises when the MongoDB container encounters difficulty connecting via the specified IP address or hostname in the setup script. To effectively resolve this issue, it is crucial to ensure that the necessary ports are allowed through the firewall. Facilitating seamless MongoDB initialization hinges on granting access to these essential ports, thereby enabling smooth communication and operation within the replica set. These are the ports to allow.

• 4369 (A peer discovery service used by RabbitMQ nodes and CLI tools)
• 5672 (Used by AMQP 0-9-1 and AMQP 1.0 clients without and with TLS)
• 15672 (HTTP API clients, management UI and rabbitmqadmin, without and with TLS)
• 25672 (Used for inter-node and CLI tools communication)
• 35672 (Used for CLI tools communication)
• 27017 (The default port for mongod and mongos instances.)
• Selected UI port (Default 443 for HTTPS and 80 for HTTP) (To access the UI and internode healthcheck)

Use these steps to allow port numbers:

1. Check the Status of Ports:

   For Ubuntu:

   $ sudo ufw status PORT_NUMBER

   For Redhat:

   $ sudo firewall-cmd --list-ports=PORT_NUMBER/tcp

2. If the port is not allowed then perform the following:

   For Ubuntu:

   $ sudo ufw allow PORT_NUMBER

   For Redhat:

   $ sudo firewall-cmd --zone=public --add-port=PORT_NUMBER/tcp --permanent

3. Reload the firewall:

   For Ubuntu:

   $ sudo ufw reload

   For Redhat:

   $ sudo firewall-cmd --reload

The maintenance password entered is not correct

The user encounters the error message mentioned below while executing the setup script during the migration or upgrade process. Error occurred while executing this command. Error: Command ‘docker exec mongo-migration mongo -u root –password <pass> admin –eval ‘db.adminCommand({setFeatureCompatibilityVersion: “5.0”})” returned non-zero exit status 1. To verify the error resulting from an incorrect maintenance password, the user must run the specified command using either podman or docker:

$ sudo docker ps

$ sudo podman ps

If you can observe a running container named “mongo-migration,” it indicates that an incorrect maintenance password was added during the migration process. 

If this is the situation, you should execute the setup script again based on the deployment type. Additionally, you need to enter the same maintenance password that was used during the old setup of Cloud Exchange. For Containerized CE or CE as HA, the command to rerun the setup script is:

$ sudo python3 ./setup

For CE as VM Instance, the command to rerun the setup script is:

$ sudo ./setup.

If the user is unable to locate the running container, they may need to refer to the troubleshooting steps provided below.

Incompatible Mongo Feature Compatibility Version

You encounter the error message mentioned below while executing the setup script during the migration or upgrade process. Error response from daemon: Container <Container_ID> is not runningError occurred while executing command. Error: Command ‘docker/podman exec mongo-migration mongo -u root –password <pass> admin –eval ‘db.adminCommand({setFeatureCompatibilityVersion: “5.0”})” returned non-zero exit status 1. To prevent this error from occurring, please follow the instructions below.
For those who are utilizing a Containerized CE, follow these steps:
1. Run this command:

$ echo MONGO_COMPATIBILITY=True >> .env

2. To verify the changes, use this command for Containerized CE instance:

$ cat .env

3. Run the setup script again. If you are using the below command:

$ sudo ./setup

If you are using CE as HA (Containerized) based instance, follow these steps:
1. Run this command:

$ echo MONGO_COMPATIBILITY=True >> <shared_drive>/config/.env

2. To verify the changes, use the following command for CE as an HA-based instance:

$ sudo cat <shared_drive>/config/.env

3. Run the setup script again on the current node. This time, the error should not appear.

$ sudo ./setup

If you are using CE as OVA instance, follow these steps:
1. Run this command:

$ sudo vi /opt/cloudexchange/cloudexchange/.env

If you have opted HA in CE as VM-based instances, proceed by executing this command:

$ sudo vi <shared_drive>/config/.env

2. Insert the following line into the .env file and remember to save the changes.

MONGO_COMPATIBILITY=True

3. Re-run the setup script by using the command below:

$ sudo ./setup

RabbitMQ Container status is showing down. To troubleshoot: Open CE UI and check container status for RabbitMQ container in Each node and if RabbitMQ Container status is down in majority of nodes, perform these steps: 

1. Open RabbitMQ Management UI this URL:

http://<CE_UI_IP>:15672/

Username: user Password: Maintenance password of CE  If you have observed these errors:Network partition Detected.mnesia reports that this RabbitMQ cluster has experienced a network partition. Steps to Resolve Network partition issue 1. Stop containers one by one using stop script:

$ sudo ./stop

2. Start CE nodes again one by one(Run Start script in primary node first):-

$ sudo ./start

3. Monitor CE Home dashboard and RabbitMQ Dashboard as well. Note: In queue ingestion/sharing tasks will be lost present in CE that are not yet performed by CE.

 Scenario: If you are modifying an existing cluster and experience clustering issues after halting with the stop script and then initiating with the start script, particularly during node addition/removal or migration, this problem may arise.

Resolution: If your message is as below that means there was some issues while removing instance2 from the cluster. “Node ‘rabbit@instance1’ thinks it’s clustered with node ‘rabbit@instance2’, but ‘rabbit@instance2’ disagrees”

1. Execute the stop script in the instance2.
   $ sudo ./stop
2. Go to the instance1 and execute below command. (make sure to replace the instance1 with the actual IP or hostname). If you are running docker, use this command:
   $ docker-compose -f docker-compose-ha.yml exec -- rabbitmq-stats rabbitmqctl forget_cluster_node rabbit@instance2
   In case of podman, use this command:
   $ podman-compose -f podman-compose-ha.yml exec -- rabbitmq-stats rabbitmqctl forget_cluster_node rabbit@instance2
3. Then execute start script in instance2.
   $ sudo ./start

Scenario: If you configure SIEM and delete that particular SIEM, and on reconfiguring it back while historical pulling is in progress for initial range mentioned in plugin configuration, CE is creating multiple tasks (One is on the creation of SIEM mapping time, and another one is on reconfiguration of SIEM Mapping). 

 To resolve this, restart CE using the stop and start scripts and perform a manual sync in Log shipper > SIEM Mappings manual sync with historical range.

Here’s how to troubleshoot the issue:

1. You need to check the status of the cloud exchange container. Check the status using these commands:
   • For Ubuntu/CentOS:

     sudo Docker ps

   • For RHEL:

     sudo podman-compose ps

2. If the containers are not running, use this command to start the containers:

   ./start

3. If you are using RHEL, make sure SELinux is disabled, because Cloud Exchange is not officially tested on SELinux. We recommend that you disable the SELinux.
4. Also check the status of ipv4 forwarder status using this command:

   systemctl net.ipv4.ip_forward

5. If the value of this command is 1, that means it is enabled. But if the value is 0, then you need to enable this by executing this command:

   systemctl net.ipv4.ip_forward=1

6. In order to check the container status, you need to go to the folder ta_cloud_exhange.
7. If your containers are running, then you need to check your network connectivity of the host machine. You can use these commands to check the connectivity:

   curl -v <IP of Local Host>:443 [i.e curl -v 127.0.0.1:443]

   curl -v www.github.com:443

8. If the connectivity is working, then you check by restarting CE. Use these command to restart CE:

   ./stop
   ./start

9. Also, there is a possibility that your SSL certificate has expired. If you are using any corporate SSL certificates, then validate the certificate. If you are using the default Cloud Exchange SSL certificate, then the certificate is valid for one year. Check that also.
10. If the certificate has expired, then follow the instructions in this document.
11. Now try rebooting the VM.
12. If the issue is not resolved, please contact Netskope Support.

Scenario: While sharing the Threat IoCs to Netskope, you might see this error.

Impact: IoCs will be shared to Netskope but the functionality of tagging the indicators will be affected. This might also impact the IoC retraction functionality.

Resolution:

1. Go to Settings > Repository and click Check for Updates to fetch the updates for the Default repository.
2. Click Update Plugins, select the Netskope Threat Exchange plugin, and click then Save and follow the prompts.
3. Wait for the next execution for sharing, and then check if the error is no longer visible in the audit logs.

Scenario: While fetching the updates from Default plugin Repository after migration from the old CE, this error occurs.

Impact: Not will be able to fetch plugin updates from Github for Default repository.

Resolution:

1. SSH into CE instance and go to the Cloud Exchange installation package directory.
2. Remove all untracked files using this command: (Use podman-compose instead of docker-compose in RHEL-based OS).

$ docker-compose exec -w /opt/netskope/repos/Default core git clean -df

Note that if you receive any error while running above commands, try this command to remove untracked files.

$ cd data/repos/Default (cd <shared_drive>/repos/Default for HA deployment) 
$ sudo rm -rf crowdstrike_identity_protect_ztre/ crowdstrike_ztre/ microsoft_entra_id_ztre/ mimecast_ztre/ netskope_ztre/ okta_ztre/

Scenario: While executing the setup script, you might see the following error due to a system specification that is not compatible with CE: CE Sizing Profile Check: Failed for Large profile” or “CE Sizing Profile Check: Failed for Medium profile”.

Impact: You will not be able to configure the CE until the minimum system requirements are met.

Resolution:

• Make sure your machine has exactly 8 or 16 CPU count to determine the medium or large profile respectively.
  Note that if the machine is deployed on an on-premises infrastructure, update the CPU count to match the profile as per your requirements. If the machine is deployed on a cloud infrastructure (like AWS, Azure), you will need to create a new instance.
• For medium profile minimum RAM and Total disk space is 16GB and 80 GB, respectively.
• For large profile minimum RAM and Total disk space is 32GB and 120 GB, respectively.
• Minimum 20 GB free disk space is required for both profiles.

If you encounter freezing or inaccessibility of the Cloud Exchange UI from your browser, this may be related to a configuration requirement in Docker/Podman environments. Docker specifically requires the net.ipv4.ip_forward setting to be enabled for outbound connectivity (https://github.com/moby/moby/issues/490). It’s important to note that while Docker expects users to enable this setting manually, Podman handles this differently, see https://github.com/containers/podman/issues/399.

If you encounter any errors during the installation, migration, or upgrade of Cloud Exchange, we’re here to assist you. Please follow the steps below to get help:

1. Open a Support Ticket: If you encounter any error during installation, migration, or upgrade of Cloud Exchange, please open a support ticket with Netskope support. Our team of engineers will promptly assist you in resolving the issue.
2. Provide Relevant Details: When opening a support ticket, please ensure to include the following details:
   • Platform logs: You can export platform logs by going to Cloud Exchange > Logging > Export Logs.
   • Diagnostic logs: Refer to the documentation here to collect diagnostic logs.

Additional Resources:

For detailed documentation on various aspects of Cloud Exchange, please refer to the following links:

• For Installation: Installation Documentation
• For Upgrading and Migration: Upgrading to the Latest Version Documentation
• For High Availability Deployment: High Availability Deployment Documentation

We’re committed to ensuring a smooth experience with Cloud Exchange and are here to support you every step of the way.

The error message is this:

Network netskope-cloud-threat-exchange-docker-compose_default Error 0.0s failed to create network netskope-cloud-threat-exchange-docker-compose_default: Error response from daemon: could not find an available, non-overlapping IPv4 address pool among the defaults to assign to the network.

The issue is specific to the network of the machine. The docker network conflicts with the host network and because of that the docker network is not created successfully. To change the network configuration of the docker you can refer to: https://docs.docker.com/compose/compose-file/06-networks/

There are 3 possible causes:

1. Mongo data directory permission issue.
2. The Core/Mongo-DB container is down.
3. The CE maintenance password is incorrect.

1. For the Mongo data directory permissions issue

Verify

Execute ls -lRn . inside the directory with docker-compose.yml.

The mongo-data dir should be read/write accessible to the user with UID 1001.

./data/mongo-data:
total 0
drwxr-xr-x. 3 1001 0 16 Apr 14 18:16 data

Solution to mongo data directory permissions issue:

Execute the setup script again using ./setup to fix the file permissions.

Restart CE.

2. For the core/mongo-db container is down issue

Verify

Check the container status using sudo docker-compose ps, all the containers should be Up.

$ sudo docker-compose ps
Name Command
State Ports
---------------------------------------------------------------------------------------------------------------- 
ce_330_core_1 /bin/sh start.sh Up 80/tcp
ce_330_mongodb-primary_1 /opt/bitnami/scripts/mongo ... Up 0.0.0.0:27018->27017/tcp ce_330_rabbitmq-stats_1 /opt/bitnami/scripts/rabbi ... Up 15671/tcp, 0.0.0.0:15672->15672/tcp, 25672 /tcp, 4369/tcp, 5551/tcp, 5552/tcp, 5671/tcp, 0.0.0.0:5672->5672/tcp
ce_330_ui_1 /bin/sh start.sh Up 0.0.0.0:443->3000/tcp, 80/tcp ce_330_watchtower_1 /watchtower --http-api-update Up 0.0.0.0:8080->8080/tcp

Solution to core/mongo-db container is down:

If any containers are down follow the below steps:

sudo docker-compose down
sudo ./start

3. For the maintenance password is incorrect Issue

Verify

Check the core logs using `sudo docker-compose logs core` for any “authentication error”.

Check if the customer is using CE version 3.2.0 or below 3.2.0 with the same MongoDB.

Solution to the maintenance password is incorrect:

Perform these steps:

sudo docker-compose down
sudo rm -rf .env
sudo ./setup
Add maintenance password as "cteadmin"
sudo ./start

SSO requires a top level domain (TLD). Not adding a TLD in the hostname while mapping the URL and upon enabling SSO in CE, a customer will receive the error “Invalid Hostname, Top Level Domain required”. This is resolved by adding proper TLD (like netskope.com).

If you receive this kind of error then it’s because Netskope CE by default supports TLSv1.3 only. To resolve this error, you should allow Netskope CE to run on TLSv1.2 along with TLSv1.3 and for that you have to change the TLS version from the setup script. Re-run the setup script again and enter ‘Yes’ to the following question.

Do you want to enable TLSv1.2 along with TLSv1.3 for CE UI.

Then execute the start script.

If your certificates are expired then follow below steps to regenerate the certificates.

1. Down all the containers.
2. Remove the certificate files (cte_cert.crt, cte_cert_key.key) from data/ssl_certs folder.
3. Re-run the setup script to regenerate the certs. Enter “https” to the below question:

   Do you want to access CE over HTTP, or HTTPS (HTTPS is recommended)? https

4. Execute the start script.

To address this issue, it is crucial to verify that the Business Rule and SIEM mapping used are correct. Additionally, ensure that all parameters are accurately filled out during the plugin configuration process. It is also important to review the sharing configuration to confirm it matches the IoCs you expect to share. If the sharing filter is incorrect, adjust the sharing criteria accordingly.

To retrieve any historical data that might have been missed due to a misconfiguration, consider removing and then re-adding the sharing configuration. By taking these steps, you can ensure that the IoCs are properly shared with the intended threat source.

Netskope only accepts URLs with wildcard characters that are in front of the domain, others will be rejected when Threat Exchange tries to send it. So *.google.com will be accepted by the Netskope tenant,but google.com/* will not. If your Threat Exchange database contains wildcards, you will need to manually tag to share.

By default, all your uploaded plugins are stored inside the ./data/custom_plugins directory. However, this can be changed from the docker-compose by mounting a different directory or the Admin can add an additional repository to download custom plugins from. This is configured within the Settings menu. This is the best method of adding additional plugins to your CE instance, and the only method for adding additional CTO, CLS, and CRE plugins.

To reset the administrator password, refer to Reset Password in the Account Settings section. Make sure to change the password from Account Settings after the CE administrator has reset the password.

To reset any other user’s password, the Super Admin can update a user password from Settings > Users, and then click the Edit icon on the right.

A special character was potentially used for the maintenance password during setup, which RabbitMQ does not support and causes issues with other services attempting to schedule tasks as cross-container communication fails to engage RabbitMQ.

Run the following commands from the directory where docker-compose.yml resides to reconfigure the maintenance password:

sudo docker-compose down
sudo rm -rf .env
sudo ./setup
Add maintenance password.
sudo ./start

Scenario: After migrating to 5.1.0 User might see the below error while updating the existing Business Rule for CTO module.

Impact: You will not be able to update the existing rule until the you redefine the rule.

Resolution: Redefine the business rule from the UI and update it.

The platform by default searches for the last 7 days of IoCs. If there are too many IoCs (more than 1 million) and no filter selected, the search performance will be slow.

Proposed solution: Consider applying the filters and narrowing the search criteria. Performance is best when the data set is ~100K records or less.

Verify you uploaded a custom plugin with active configuration to Netskope CE prior to upgrading or restarting the containers. In such a case, upload the custom plugin after the upgrade (Refer to Create a Custom CTE Plugin in the Supported 3rd-party Plugins). The configurations would be retained after uploading the custom plugin and normal operation is restored.

Verify if the outgoing API calls require a proxy. If your network deployment expects a proxy for HTTP API calls and a proxy is configured, the plugin operations would be impacted.

Proposed Solution:

1. Go to Settings > General > Proxy.
2. Edit the existing configuration and enable Use System Proxy.

CE relies on an internal scheduling mechanism for the plugin’s task. There are workers which execute the plugin tasks, by picking up a task from the queue one by one. The number of workers available in your system depends on the number of cores. If the available workers are busy serving plugin task, the already queued up task has to wait till the existing worker is available. This situation may usually occur during initial data ingestion, where there’s more data to be processed.

Proposed Solution: Consider increasing the cores of the system if you have a large number of configured plugins, and the configured plugins are consistently lagging behind. For initial ingestion, the system should pick up the backlog post initial ingestion and behave normally if the incremental data is not large.

If there is a red alert icon on one of the configurations, it indicates that there was one or more problems while polling the plugged-in system for data per that configuration. This could be related to API, proxy, or SSL settings.

Proposed Solution:

• Make sure the Plugin Configuration has correct parameters for API, Secret key, URL, etc.
• Make sure Enable Proxy is selected and your proxy is configured if outbound network calls require a proxy connection.
• Check logs for errors occurring around the last run time displayed on the configuration from the Audit section.

When a user tries to upload a plugin with a tar.gz package using the browse button, tar.gz files are not selectable by default.

Proposed Solution: Drag and drop plugin packages to the drop area of the UI.

Open the plugin configuration and set the Last Run value to an older date-time and save the configuration. Make sure that the configuration is currently not running when you update the Last Run value.

• For a Docker-based deployment:
  1. Retrieve the core container logs using this command:
     $ docker-compose logs core | grep WorkerLost
     If the logs containing WorkerLost appear around 5-6 times, this indicates there’s an issue. Restart the containers using these commands:
     $ ./stop
$ ./start
• For a Podman-based deployment:
  1. To access the core container logs, use this command:
     $ podman-compose logs core | grep WorkerLost. If the logs containing WorkerLost appear around 5-6 times, this indicates there’s an issue. Restart the containers using these commands:
     $ ./stop
$ /start

Scenario: In core logs, you might observe “Process ‘ForkPoolWorker-**’ pid:552 exited with ‘signal 9 (SIGKILL)’”

Impact: Often the core container might get restarted and you may experience delay in tasks performed by CE (like logs ingestion getting delayed).

Resolution:

• Make sure the CE is hosted on a machine having at least 2.2 GHz CPU frequency.
• If you’re not meeting the CPU frequency criteria please migrate to the machine which meets the criteria.
• If you’re already meeting the CPU frequency criteria, reach out to the support team.

This issue arises because Podman does not include the facility to automatically restart containers after a reboot. This behavior is particularly relevant for users of RHEL (Red Hat Enterprise Linux) servers where Podman is commonly used. Users will need to manually start the containers post-reboot. Here are the detailed steps to do so:

1. Go to the Cloud Exchange Directory:

$ cd <ce_directory>

Note that for CE as a VM, the Cloud Exchange directory is:

/opt/cloudexchange/cloudexchange

2. Check the Status of the Containers:

$ podman-compose ps

3. Start Cloud Exchange:

$ ./start

For users operating with Docker on other VMs like Ubuntu or CentOS, there’s no need to take any manual action for container restarts. Docker includes an auto-restart feature that ensures containers automatically restart after a reboot. This automatic restart functionality provides a seamless experience without the need for manual intervention post-reboot.

To change the SSO configuration, you need to disable SSO using the toggle on the SSO Configuration tab in the CE UI. To do so, follow the following steps:

1. First, save your details related to SSO configuration from Okta in a file (Identity Provider Issuer URL, Identity Provider SSO URL, Identity Provider SLO URL, and Public x509 Certificate).
2. Next disable the SSO using the toggle which can be seen on the SSO Configuration page in the CE UI.
3. After disabling SSO, the change in the domain should be reflected in the Service Provider Entity ID, Service Provider ACS URL, and Service Provider SLS URL as per your modified domain.
4. When you see this in the CE UI, re-enable SSO using the toggle.
5. Re-enter the details saved in the first step from the file.

Note that you would also have to update your IdP (like) Okta details pertaining to the Service Provider Entity ID, Service Provider ACS URL, and Service Provider SLS URL so that it also reflects the change with respect to your modified domain.