Cloud Exchange Troubleshooting

Docker/Podman requires the net.ipv4.ip_forward setting to be enabled in order to access the outside world.  According to reported issue on docker (https://github.com/moby/moby/issues/490), docker needs this to be  enabled but it does not enable this automatically as it is the user’s responsibility (podman seems to  have a different behavior in this regard, see https://github.com/containers/podman/issues/399).

Yes, please open a ticket with Netskope support and we’ll get an engineer to help you as soon as possible. Usually following the installation steps in this guide prevents most issues. If you do engage support, and you can access Cloud Exchange (the install got that far), please provide the following details with the ticket:

The error message is this:

Network netskope-cloud-threat-exchange-docker-compose_default Error 0.0s failed to create network netskope-cloud-threat-exchange-docker-compose_default: Error response from daemon: could not find an available, non-overlapping IPv4 address pool among the defaults to assign to the network.

The issue is specific to the network of the machine. The docker network conflicts with the host network and because of that the docker network is not created successfully. To change the network configuration of the docker you can refer to: https://docs.docker.com/compose/compose-file/06-networks/

For issues when checking for installed podman-compose version, you can reinstall podman-compose by running this command:
$ sudo pip3 install podman-compose.

There are 3 possible causes:

1. For the Mongo data directory permissions issue

Verify

Execute ls -lRn . inside the directory with docker-compose.yml.

The mongo-data dir should be read/write accessible to the user with UID 1001.

./data/mongo-data:
total 0
drwxr-xr-x. 3 1001 0 16 Apr 14 18:16 data

Solution to mongo data directory permissions issue:

Execute the setup script again using ./setup to fix the file permissions.

Restart CE.

2. For the core/mongo-db container is down issue

Verify

Check the container status using sudo docker-compose ps, all the containers should be Up.

$ sudo docker-compose ps
Name Command
State Ports
---------------------------------------------------------------------------------------------------------------- 
ce_330_core_1 /bin/sh start.sh Up 80/tcp
ce_330_mongodb-primary_1 /opt/bitnami/scripts/mongo ... Up 0.0.0.0:27018->27017/tcp ce_330_rabbitmq-stats_1 /opt/bitnami/scripts/rabbi ... Up 15671/tcp, 0.0.0.0:15672->15672/tcp, 25672 /tcp, 4369/tcp, 5551/tcp, 5552/tcp, 5671/tcp, 0.0.0.0:5672->5672/tcp
ce_330_ui_1 /bin/sh start.sh Up 0.0.0.0:443->3000/tcp, 80/tcp ce_330_watchtower_1 /watchtower --http-api-update Up 0.0.0.0:8080->8080/tcp

Solution to core/mongo-db container is down:

If any containers are down follow the below steps:

sudo docker-compose down
sudo ./start

3. For the maintenance password is incorrect Issue

Verify

Check the core logs using `sudo docker-compose logs core` for any “authentication error”.

Check if the customer is using CE version 3.2.0 or below 3.2.0 with the same MongoDB.

Solution to the maintenance password is incorrect:

Perform the following steps:

sudo docker-compose down
sudo rm -rf .env
sudo ./setup
Add maintenance password as "cteadmin"
sudo ./start

SSO requires a top level domain (TLD). Not adding a TLD in the hostname while mapping the URL and upon enabling SSO in CE, a customer will receive the error “Invalid Hostname, Top Level Domain required”. This is resolved by adding proper TLD e.g. (netskope.com).

If you receive this kind of error then it’s because Netskope CE by default supports TLSv1.3 only. To resolve this error, you should allow Netskope CE to run on TLSv1.2 along with TLSv1.3 and for that you have to change the TLS version from the setup script. Re-run the setup script again and give ‘Yes’ to following question.

Do you want to enable TLSv1.2 along with TLSv1.3 for CE UI.

Then execute the start script.

If your certificates are expired then follow below steps to regenerate the certificates.

Do you want to access CE over HTTP, or HTTPS (HTTPS is recommended)? https

While sharing the IoCs to a particular plugin, the sharing filters provided with the plugin’s configurations are considered. Ensure that the sharing configuration matches with the IoCs you are expecting to be shared. If the sharing filter is incorrect, fix the sharing criteria. To fetch the historical data that you may have missed due to misconfiguration, consider removing the sharing configuration and re-adding it.

Netskope only accepts URLs with wildcard characters that are in front of the domain, others will be rejected when Threat Exchange tries to send it. So *.google.com will be accepted by the Netskope tenant,but google.com/* will not. If your Threat Exchange database contains wildcards, you will need to manually tag to share.

Yes. Refer to Cloud Exchange Users.

By default, all your uploaded plugins are stored inside the ./data/custom_plugins directory. However, this can be changed from the docker-compose by mounting a different directory or the Admin can add an additional repository to download custom plugins from. This is configured within the Settings menu. This is the best method of adding additional plugins to your CE instance, and the only method for adding additional CTO, CLS, and CRE plugins.

To reset the administrator password, refer to Reset Password in the Account Settings section. Make sure to change the password from Account Settings after the CE administrator has reset the password.

To reset any other user’s password, the Super Admin can update a user password from Settings > Users, and then click the Edit icon on the right.

A special character was potentially used for the maintenance password during setup, which RabbitMQ does not support and causes issues with other services attempting to schedule tasks as cross-container communication fails to engage RabbitMQ.

Run the following steps from the directory where docker-compose.yml resides to reconfigure the maintenance password:

sudo docker-compose down
sudo rm -rf .env
sudo ./setup
Add maintenance password.
sudo ./start

The platform by default searches for the last 7 days of IoCs. If there are too many IoCs (more than 1 million) and no filter selected, the search performance will be slow.

Proposed solution : Consider applying the filters and narrowing the search criteria. Performance is best when the data set is ~100K records or less.

Verify you uploaded a custom plugin with active configuration to Netskope CE prior to upgrading or restarting the containers. In such a case, upload the custom plugin after the upgrade (Refer to Create a Custom CTE Plugin in the Supported 3rd-party Plugins). The configurations would be retained after uploading the custom plugin and normal operation is restored.

Verify if the outgoing API calls require a proxy. If your network deployment expects a proxy for HTTP API calls and a proxy is configured, the plugin operations would be impacted.

Proposed Solution:

CE relies on an internal scheduling mechanism for the plugin’s task. There are workers which execute the plugin tasks, by picking up a task from the queue one by one. The number of workers available in your system depends on the number of cores. If the available workers are busy serving plugin task, the already queued up task has to wait till the existing worker is available. This situation may usually occur during initial data ingestion, where there’s more data to be processed.

Proposed Solution: Consider increasing the cores of the system if you have a large number of configured plugins, and the configured plugins are consistently lagging behind. For initial ingestion, the system should pick up the backlog post initial ingestion and behave normally if the incremental data is not large.

If there is a red alert icon on one of the configurations, it indicates that there was one or more problems while polling the plugged-in system for data per that configuration. This could be related to API, proxy, or SSL settings.

Proposed Solution:

When a user tries to upload a plugin with a tar.gz package using the browse button, tar.gz files are not selectable by default.

Proposed Solution: Drag and drop plugin packages to the drop area of the UI.

Open the plugin configuration and set the Last Run value to an older date-time and save the configuration. Make sure that the configuration is currently not running when you update the Last Run value.

For STIX 1, the cybox observables of type URI, Domain, SHA256 and MD5 are extracted. For STIX 2, the same type of observables are extracted from the pattern string field of the indicators.

This is a known limitation. Indicator comment is only added the first time that file is encountered. In subsequent runs, if the file is renamed, only the external hits will be increased as long (as the contents of the file remains same).

This may happen if the API credentials used to create the tickets do not have access to the tickets anymore. Make sure that the credentials you are using have read access to the tickets.

When a Git repo contains a large number of files, a plugin configuration could timeout. The default timeout duration is set to 120 minutes. However, this can be increased by adding an environment variable in the docker-compose file.

core:
    image: crestsystems/netskope:core-latest
    container_name: "core"
    volumes:
    - ./data/custom_plugins:/opt/netskope/integrations/cte/custom_plugins:z
    environment:
    - PLUGIN_TIMEOUT_MINUTES=120

Log Shipper stops if something breaks, typically because one of the following conditions are considered a failure: code race condition, core dump, process errors, or TCP socket errors. If CE receives 5 of these, it stops. Note that Log Shipper does NOT restart when it receives API errors, including the HTTP 200 error code, Response: {"status":"error","errorCode":"General Error","errors":["We encountereda backend error. Please try again."]}

When updating the plugin from Settings > Plugin Repository, the plugin code is getting updated but it is not reloaded, which is causing this error. Try to restart the docker services using the command $ docker-compose restart to resolve the issue.

When you see this message: Error URL host invalid, Top Level Domain required, it’s typically because the domain name used for accessing CE does not have TLD. To resolve this, use a hostname with TLD.

Containers not starting after rebooting the server/VM is due to the restart: on-failure:5 tag in the docker-compose.yml file. To resolve this, change tag value as restart: always” in the docker-compose.yml file.

A Bad Gateway error appears due to mongo-data folder permissions not being properly set. Copying the wrong MongoDB folder can cause a permission issue to occur. To resolve this, delete the mongo-data folder, recreate the folder, and copy data from the backup location.

One of the possible reason for getting Bad gateway is the mongo-data directory is removed or the mongo-data directory does not have enough permissions. To resolve the issue:

Bad gateway error appears due to the maintenance password not being setup, which is available in latest version, but not in old docker compose version. To resolve this, do a git pull, and then run the setup script with maintenance password and restart CE.

If you are having any issue while checking for installed a podman-compose version, you can reinstall podman-compose by running below command: sudo pip3 install podman-compose.

CE is designed to leverage dynamically defined DNS due to limitations of docker-compose. If you wish to have CE utilize rotating DNS services, the resolv.conf file must be customized to eliminate references to corporate DNS servers and restored after every code upgrade

To apply a workaround customers need to follow below steps:

Note that you will need to maintain this resolv.conf file after every upgrade as the setup script will overwrite your changes.

Follow the steps below to upgrade and maintain your file:

If a proxy is enabled while migrating to cloud, ”Connection time out” error messages appear in the logs. To avoid this error, you need to remove the proxy from Cloud Exchange> Settings > Proxy, clear the data, and then save it.