Cloud Exchange High Availability - Netskope Technical Documentation

This document describes how High Availability (HA) works in Cloud Exchange. After reviewing the architectural diagram feature list, prerequisites, and sizing guidelines, deploy HA in Cloud Exchange. After the deployment section, there are sections that explain migrating, upgrading, hardening, known limitations, and troubleshooting.

HA Architecture

Features

Active-Active configurations for Cloud Exchange nodes, enhancing system availability and fault tolerance.
- The Cloud Exchange Core container is engineered to function as a dedicated worker capable of handling multiple tasks concurrently. For a medium-sized setup, it can manage 10 tasks, while a large setup allows for 20 tasks. These tasks may include operations such as polling, ingestion, and heartbeat monitoring. In a High Availability (HA) cluster with three nodes, the processing power effectively triples, enhancing the system’s execution capacity.
- Task assignments are coordinated through RabbitMQ, which distributes tasks to the core container’s workers based on their current load and the number of tasks they are actively processing. In the event of a core container restart or a node failure, transformation and ingestion tasks are requeued to ensure no data is lost. Meanwhile, data retrieval tasks are reassigned to another node, typically within five minutes of the failure, subject to the workload of the new node.
- To encapsulate, the core container is designed with both task-level and node-level HA to ensure continuous operation and data integrity.
In a cluster, there will be multiple instances of MongoDB and RabbitMQ. If one node goes down, the other nodes will continue to serve requests, ensuring zero downtime.
The multiple identical nodes of Cloud Exchange will be running concurrently. And all of them are actively processing plugin tasks simultaneously.
For any queries about the Multi-DC deployments refer the knowledge base article for more details. KB article.

To watch a video about how to configure Cloud Exchange HA, click play.

To watch a video about how Cloud Exchange HA Failover works, click play.

UI Dashboard for Cluster Status

Check the current cluster status on the Home page of Cloud Exchange.

The Core service is dependent on the UI services. Because for security reasons, we are not making the core service public. The Core service will be accessible through UI service, and it will be accessed by an internal network. If the UI service is down, the core will show an unknown status on the dashboard.

Important Notes

If you are transitioning from a standalone configuration to a High Availability (HA) setup, you should know your maintenance password, as it is required to migrate the Mongo data into the new setup.
Make sure that every machine can connect to the ports listed below on every machine including the machine itself. The reason for including the current machine is, the API calls will be made to the IP address of the server, and the API call will be made from inside the docker container. So the connectivity of the port using IP address must be allowed. The firewall policies for these listed ports and all the machines should be configured to ensure seamless connection between the machines.
- 4369 (A peer discovery service used by RabbitMQ nodes and CLI tools)
- 8000 (Used by CE Management Server)
- 5671 (Used by AMQP 0-9-1 and AMQP 1.0 clients without TLS)
- 15671 (HTTP API clients, management UI and rabbitmqadmin without TLS)
- 25672 (Used for internode and CLI tools communication)
- 35672 (Used for CLI tools communication)
- 27017 (The default port for mongod and mongos instances)
- Selected UI port (Default 443 for HTTPS and 80 for HTTP) (To access the UI and internode healthcheck)
- 24007-24029 (GlusterFS ports used for inter node communication, health checks, selfheal demon, brick communication)
If you are using Cloud Exchange as a VM, expand Deploy HA for Cloud Exchange as a VM and complete Prepare Each VM before you set up the nodes. It covers the unique hostnames and the firewall that the VM needs.

Sample command to open 443 port using a firewall:

sudo firewall-cmd --permanent --add-port=443/tcp

These ports will be used for the clustering of the MongoDB and RabbitMQ services. Also the UI ports are required to perform a health check from all the machines.

Deploy HA in Cloud Exchange

The steps depend on how you installed Cloud Exchange. Select the section that matches your deployment, and follow only those steps. Then continue with Add Secondary Nodes into HA Cluster, which is required for both deployments.

Cloud Exchange as a VM: You installed Cloud Exchange from a VM image. The Cloud Exchange files are already on the VM, so you do not clone the GitHub repository. For installation details, refer to Install Cloud Exchange on a Virtual Machine or Install Cloud Exchange on a Cloud Platform. For deployment details, expand the Deploy HA for Cloud Exchange as a VM section below.
Containerized deployment: You installed Cloud Exchange on your own Linux host by cloning the GitHub repository. For installation details, refer to Install Cloud Exchange on Ubuntu or Install Netskope Cloud Exchange on a Red Hat Enterprise 9.5 Linux Host. For deployment details, expand the Deploy HA for a Containerized Deployment section below.

Deploy HA for Cloud Exchange as a VM

The VM already has the Cloud Exchange files in the /opt/cloudexchange/cloudexchange directory, so you do not clone the GitHub repository. Complete Prepare Each VM on every VM in the cluster.

Prepare Each VM

Make sure that every VM has a different hostname. To change the hostname of a VM, run this command:
```
sudo hostnamectl set-hostname <new_hostname>
```
Enable the firewall. firewalld is installed on the VM but disabled by default. Run these commands, which also restart the Docker service:
```
sudo systemctl enable firewalld
sudo systemctl start firewalld
sudo systemctl restart docker
```
Open the ports listed in Important Notes on every VM.

Set Up a Primary Node

Go to the Cloud Exchange directory:
```
cd /opt/cloudexchange/cloudexchange
```
(optional) Copy and edit the cloudexchange.config file. Edit the variables as per your requirements, and save the file:
```
cp cloudexchange.config.example cloudexchange.config
vi cloudexchange.config
```
The maintenance and JWT Secret passwords are used for database authentication. You need these passwords when you restore a backup.

Non-ASCII characters, such as accented letters, other scripts, and emoji, are not supported in the maintenance and JWT Secret passwords. Some processes might not work, and the system might fail, if you use them.

Stop and disable the Cloud Exchange service:

sudo systemctl stop cloud-exchange && sudo systemctl disable cloud-exchange

Execute setup:
```
sudo ./setup
```
Execute start to bring up the standalone instance:
```
sudo ./start
```
After the node is up, go to Settings > General > Node Configuration.
Click Enable HA, and confirm the choice. Observe the streamed logs. While enabling HA, the node will restart once.
After the node is up, validate the service status for RabbitMQ, MongoDB, Core, and UI.

Set Up a Secondary Node

Copy the JWT secret key from the primary node. If you don’t recall the JWT secret value of the primary node, generate and apply a new JWT token by following the steps in Cloud Exchange Troubleshooting.
Go to the Cloud Exchange directory on the secondary node:
```
cd /opt/cloudexchange/cloudexchange
```
Copy and edit the cloudexchange.config file, and update the JWT secret key from the primary node:
```
cp cloudexchange.config.example cloudexchange.config
vi cloudexchange.config
```

Stop and disable the Cloud Exchange service:

sudo systemctl stop cloud-exchange && sudo systemctl disable cloud-exchange

Copy the existing CA key from /opt/cloudexchange/cloudexchange/data/ssl_certs/mongodb_rabbitmq_certs/tls_cert_ca.key of the primary node.
Paste it at the same path on the secondary node. The file name must be exactly tls_cert_ca.key.
Execute setup on the secondary node. After setup is complete on a new node, the new certificates for that node will be generated and will be signed by an existing CA, and this will enable the Management server communications between the new node and the HA Cluster.
```
sudo ./setup
```

After you set up the primary node and the secondary nodes, continue with Add Secondary Nodes into HA Cluster. This step is required.

Deploy HA for a Containerized Deployment

Use this track when you installed Cloud Exchange on your own Linux host. You clone the Cloud Exchange GitHub repository on every machine in the cluster.

Set Up a Primary Node

Clone the netskopeoss/ta_cloud_exchange GitHub repository on all the machines where Cloud Exchange will be deployed, and go to the repository directory:
```
mkdir netskope
cd netskope
git clone https://github.com/netskopeoss/ta_cloud_exchange
cd ta_cloud_exchange
```
If you already have the repository cloned, remove any local changes and pull the latest version:
```
git reset --hard
git pull
```
Check out the version that you want to deploy. For example, to check out version 6.1.0:
```
git checkout v6.1.0
```
(optional) Copy and edit the cloudexchange.config file. Edit the variables as per your requirements, and save the file:
```
cp cloudexchange.config.example cloudexchange.config
vi cloudexchange.config
```
Important

If you already created the cloudexchange.config file on this machine, do not run the copy command again. It replaces your file with the example file and removes your settings.

Important

Non-ASCII characters, such as accented letters, other scripts, and emoji, are not supported in the maintenance and JWT Secret passwords. Some processes might not work, and the system might fail, if you use them.

The maintenance and JWT Secret passwords are used for database authentication. You need these passwords when you restore a backup.

Stop and disable the Cloud Exchange service:

sudo systemctl stop cloud-exchange && sudo systemctl disable cloud-exchange

Execute setup:
```
sudo ./setup
```
Execute start to bring up the standalone instance:
```
sudo ./start
```
After the node is up, go to Settings > General > Node Configuration.
Click Enable HA, and confirm the choice. Observe the streamed logs. While enabling HA, the node will restart once.
After the node is up, validate the service status for RabbitMQ, MongoDB, Core, and UI.

Set Up a Secondary Node

Copy the JWT secret key from the primary node. If you don’t recall the JWT secret value of the primary node, generate and apply a new JWT token by following the steps in Cloud Exchange Troubleshooting.
Clone the netskopeoss/ta_cloud_exchange public GitHub repository on the secondary node, and go to the repository directory:
```
mkdir netskope
cd netskope
git clone https://github.com/netskopeoss/ta_cloud_exchange
cd ta_cloud_exchange
```
Check out the version that you want to deploy. For example, to check out version 6.1.0:
```
git checkout v6.1.0
```
In the folder where the repository is cloned, copy and edit the cloudexchange.config file, and update the JWT secret key from the primary node:
```
cp cloudexchange.config.example cloudexchange.config
vi cloudexchange.config
```
Important

If you already created the cloudexchange.config file on this machine, do not run the copy command again. It replaces your file with the example file and removes your settings.

Stop and disable the Cloud Exchange service:

sudo systemctl stop cloud-exchange && sudo systemctl disable cloud-exchange

Copy the existing CA key from ta_cloud_exchange/data/ssl_certs/mongodb_rabbitmq_certs/tls_cert_ca.key of the primary node.
Paste it at the same path on the secondary node. The file name must be exactly tls_cert_ca.key.
Execute setup on the secondary node. After setup is complete on a new node, the new certificates for that node will be generated and will be signed by an existing CA, and this will enable the Management server communications between the new node and the HA Cluster.
```
sudo ./setup
```

After you set up the primary node and the secondary nodes, continue with Add Secondary Nodes into HA Cluster. This step is required.

Both deployment types continue below with Add Secondary Nodes into HA Cluster. This step is required, and you complete it after you set up the nodes in either section above.

Add Secondary Nodes into HA Cluster

After the setup script is executed successfully in all nodes, log in to the primary node and go to Settings > General > Node Configurations.
Click Add New Node, and enter the FQDN/IP address of the new node.
Click the + icon beside the new node to proceed adding it as a new node.
Observe the streamed logs. While adding a new node, the primary node will also restart.

HA Node Management

To add a new secondary node to a cluster, go to Settings > General > Node Configurations.
Click Add and provide the secondary node IP address, and then click the add (+) icon.

Use the Stop icon besides the node to stop a node.

User can also view the historical node for the node setup process using the Logs icon.

Use the Delete icon to remove a secondary node from the cluster.

Similarly, use the Restart icon to restart any node from the Cloud Exchange UI.

In the event you want to migrate back to standalone mode, watch this video to learn how.

Hardening Guidelines

In order to establish the necessary connectivity between Docker services across various machines and facilitate the integration of nodes into the cluster, we have exposed the ports listed below from the Docker services. It is imperative that these ports remain accessible from all machines where the Cloud Exchange HA deployment is intended. To enhance security measures, it is also advisable to restrict access to these ports from other IP addresses.
These ports will be used for the clustering of the MongoDB and RabbitMQ services. Also the UI ports are required to perform a health check from all the machines.
- 4369 (A peer discovery service used by RabbitMQ nodes and CLI tools)
- 8000 (Used by CE Management Server)
- 5671 (Used by AMQP 0-9-1 and AMQP 1.0 clients without TLS)
- 15671 (HTTP API clients, management UI and rabbitmqadmin without TLS)
- 25672 (Used for inter-node and CLI tools communication)
- 35672 (Used for CLI tools communication)
- 27017 (The default port for mongodb and mongos instances)
- Selected UI port (Default 443 for HTTPS and 80 for HTTP) (To access the UI and internode healthcheck)
GlusterFS installation and setup will be driven through the Cloud Exchange UI and management server, which will act as shared storage and will also have HA capabilities. All the VMs involved in the HA cluster should have connectivity to GlusterFS repo during HA configuration.

Cluster Node Count Requirements

As a part of MongoDb replication and RabbitMQ mirroring requirements in the event of a failure, it is crucial to ensure that the HA cluster remains operational with the majority of the Cloud Exchange nodes ACTIVE/ONLINE. An HA cluster with an odd number of Cloud Exchange nodes has a higher chance of remaining operational than a cluster with an even number of Cloud Exchange nodes.

Total Number of CE Nodes in an HA cluster.	Minimum Number of ACTIVE/ONLINE CE Nodes Required in an HA Cluster at any Given Time for Successful Operation.
3	2
4	3
5	3
6	4
7	4
8	5
9	5

Limitations

When setting up HA on RHEL machines, make sure the systems can reach the GlusterFS repositories. On Ubuntu 24.04, GlusterFS is installed directly from Ubuntu’s own repositories.
Running multiple redundant instances of the Cloud Exchange requires additional hardware and computing resources.
In some cases, HA setups may introduce increased latency due to the need to replicate data or coordinate between active instances.
Whenever there are modifications to IP addresses or node-related configurations, it will be necessary to execute both the setup and start scripts on all machines within the system to ensure that the changes are properly propagated and synchronized across the cluster. It is recommended to add all the required machines at once and use a fixed IP addresses for the machines.
It is crucial to maintain the majority of nodes operational at all times, as the failure to do so would result in cluster failure. For example,in a 3-node cluster, at least 2 nodes must be operational at any given time; similarly, in a 5-node cluster, at least 3 nodes should be up and running. The pattern continues accordingly for larger clusters. This is because MongoDB will not initiate the Primary election process under such circumstances. Additionally, we have implemented a pause_minority option for RabbitMQ to mitigate the challenges associated with network partition-induced inconsistencies. This should recover once the node comes back up and connects to the cluster again. Refer to Cluster Node Count Requirements to see the minimum required operational nodes in a cluster.
Ref: https://www.mongodb.com/docs/v5.0/core/replica-set-elections/#network-partition | https://www.rabbitmq.com/partitions.html#options
In rare cases, when a cluster is facing back-to-back node failures, there are chances of data loss in the RabbitMQ because the queue will become leader and mirror very frequently, and in such cases, other nodes may lose the data which is not synchronized yet.
Ref: https://www.rabbitmq.com/ha.html#behaviour
To use SSO, we will be able to add only one node’s IP address as a redirect URL. Optionally, we can configure the load balancer against all the IP addresses, and use IP address of the load balancer, to configure the SSO. And it will redirect the request to any available node.

Troubleshooting HA in Cloud Exchange

The ./stop script in HA will remove the MongoDB and RabbitMQ node from the cluster. And stop the services in that particular machine. It is crucial to have the services up while running the ./stop script. Otherwise the script might fail as the script won’t be able to create the Mongo client to remove the node.
During migration, if you have any issues with node renaming, check the prerequisite and restore the backup from 1st step. Then retry the migration from 2nd step.
```
sudo rm -rf <path-to-ta_cloud_exchange>/data/mongo-data/data/* 
sudo cp -R <temp-mongo-path>/* 
<path-to-ta_cloud_exchange>/data/mongo-data/data/
```
Make sure docker/podman services are running. If not, run the services using this command
```
sudo systemctl restart docker
```
If the UI shows a down status for any container for a long period of time, check the status of the container and perform a reboot if required. Use podman wherever applicable.
- Check the container status. If all the services are running move to the next step. Otherwise, run the start script again to recover the container.
```
docker ps
```
- Check for the MongoDB cluster status. Execute these commands to run the command inside the MongoDB container.
```
docker compose exec -- mongodb-primary bash 
mongosh -u root -p $MONGO_INITDB_ROOT_PASSWORD 
rs.status()
```
  If the status shows anything other than PRIMARY and SECONDARY for a long period of time, run this command to restart the container on a specific machine.
```
docker compose restart mongodb-primary
```
- Check the RabbitMQ cluster status. Execute these commands to run the command inside the RabbitMQ container.
```
docker compose exec -- rabbitmq-stats bash 
rabbitmqctl cluster_status
```
  Check the running node section in the output and figure out the connected and running nodes. Restart the RabbitMQ container if required.
```
docker compose restart rabbitmq-stats
```
If any node experiences a worker lost error, like if the worker is not connected with RabbitMQ, restart the specific core container to recover the workers. This issue is likely to occur when the RabbitMQ cluster is down for a long period of time.
```
docker compose restart core
```
GlusterFS installation errors while starting Cloud Exchange:
- While running the start script if you see the error failed to install glusterfs, you can ignore this error if you wish to use Cloud Exchange as a standalone instance and do not need to enable an HA cluster.
- If you wish to use the HA clustering, make sure the environment has connectivity to the relevant GlusterFS repo and ensure that the system requirements are met before proceedings.

HA Architecture

Features

Active-Active configurations for Cloud Exchange nodes, enhancing system availability and fault tolerance.
- The Cloud Exchange Core container is engineered to function as a dedicated worker capable of handling multiple tasks concurrently. For a medium-sized setup, it can manage 10 tasks, while a large setup allows for 20 tasks. These tasks may include operations such as polling, ingestion, and heartbeat monitoring. In a High Availability (HA) cluster with three nodes, the processing power effectively triples, enhancing the system’s execution capacity.
- Task assignments are coordinated through RabbitMQ, which distributes tasks to the core container’s workers based on their current load and the number of tasks they are actively processing. In the event of a core container restart or a node failure, transformation and ingestion tasks are requeued to ensure no data is lost. Meanwhile, data retrieval tasks are reassigned to another node, typically within five minutes of the failure, subject to the workload of the new node.
- To encapsulate, the core container is designed with both task-level and node-level HA to ensure continuous operation and data integrity.
In a cluster, there will be multiple instances of MongoDB and RabbitMQ. If one node goes down, the other nodes will continue to serve requests, ensuring zero downtime.
The multiple identical nodes of Cloud Exchange will be running concurrently. And all of them are actively processing plugin tasks simultaneously.
For any queries about the Multi-DC deployments refer the knowledge base article for more details. KB article.

To watch a video about how to configure Cloud Exchange HA, click play.

To watch a video about how Cloud Exchange HA Failover works, click play.

UI Dashboard for Cluster Status

Check the current cluster status on the Home page of Cloud Exchange.

Important Notes

If you are transitioning from a standalone configuration to a High Availability (HA) setup, you should know your maintenance password, as it is required to migrate the Mongo data into the new setup.
Make sure that every machine can connect to the ports listed below on every machine including the machine itself. The reason for including the current machine is, the API calls will be made to the IP address of the server, and the API call will be made from inside the docker container. So the connectivity of the port using IP address must be allowed. The firewall policies for these listed ports and all the machines should be configured to ensure seamless connection between the machines.
- 4369 (A peer discovery service used by RabbitMQ nodes and CLI tools)
- 8000 (Used by CE Management Server)
- 5671 (Used by AMQP 0-9-1 and AMQP 1.0 clients without TLS)
- 15671 (HTTP API clients, management UI and rabbitmqadmin without TLS)
- 25672 (Used for internode and CLI tools communication)
- 35672 (Used for CLI tools communication)
- 27017 (The default port for mongod and mongos instances)
- Selected UI port (Default 443 for HTTPS and 80 for HTTP) (To access the UI and internode healthcheck)
- 24007-24029 (GlusterFS ports used for inter node communication, health checks, selfheal demon, brick communication)
If you are using Cloud Exchange as a VM, expand Deploy HA for Cloud Exchange as a VM and complete Prepare Each VM before you set up the nodes. It covers the unique hostnames and the firewall that the VM needs.

Sample command to open 443 port using a firewall:

sudo firewall-cmd --permanent --add-port=443/tcp

These ports will be used for the clustering of the MongoDB and RabbitMQ services. Also the UI ports are required to perform a health check from all the machines.

Deploy HA in Cloud Exchange

Cloud Exchange as a VM: You installed Cloud Exchange from a VM image. The Cloud Exchange files are already on the VM, so you do not clone the GitHub repository. For installation details, refer to Install Cloud Exchange on a Virtual Machine or Install Cloud Exchange on a Cloud Platform. For deployment details, expand the Deploy HA for Cloud Exchange as a VM section below.
Containerized deployment: You installed Cloud Exchange on your own Linux host by cloning the GitHub repository. For installation details, refer to Install Cloud Exchange on Ubuntu or Install Netskope Cloud Exchange on a Red Hat Enterprise 9.5 Linux Host. For deployment details, expand the Deploy HA for a Containerized Deployment section below.

Deploy HA for Cloud Exchange as a VM

The VM already has the Cloud Exchange files in the /opt/cloudexchange/cloudexchange directory, so you do not clone the GitHub repository. Complete Prepare Each VM on every VM in the cluster.

Prepare Each VM

Make sure that every VM has a different hostname. To change the hostname of a VM, run this command:
```
sudo hostnamectl set-hostname <new_hostname>
```
Enable the firewall. firewalld is installed on the VM but disabled by default. Run these commands, which also restart the Docker service:
```
sudo systemctl enable firewalld
sudo systemctl start firewalld
sudo systemctl restart docker
```
Open the ports listed in Important Notes on every VM.

Set Up a Primary Node

Go to the Cloud Exchange directory:
```
cd /opt/cloudexchange/cloudexchange
```
(optional) Copy and edit the cloudexchange.config file. Edit the variables as per your requirements, and save the file:
```
cp cloudexchange.config.example cloudexchange.config
vi cloudexchange.config
```
The maintenance and JWT Secret passwords are used for database authentication. You need these passwords when you restore a backup.

Non-ASCII characters, such as accented letters, other scripts, and emoji, are not supported in the maintenance and JWT Secret passwords. Some processes might not work, and the system might fail, if you use them.

Stop and disable the Cloud Exchange service:

sudo systemctl stop cloud-exchange && sudo systemctl disable cloud-exchange

Execute setup:
```
sudo ./setup
```
Execute start to bring up the standalone instance:
```
sudo ./start
```
After the node is up, go to Settings > General > Node Configuration.
Click Enable HA, and confirm the choice. Observe the streamed logs. While enabling HA, the node will restart once.
After the node is up, validate the service status for RabbitMQ, MongoDB, Core, and UI.

Set Up a Secondary Node

Copy the JWT secret key from the primary node. If you don’t recall the JWT secret value of the primary node, generate and apply a new JWT token by following the steps in Cloud Exchange Troubleshooting.
Go to the Cloud Exchange directory on the secondary node:
```
cd /opt/cloudexchange/cloudexchange
```
Copy and edit the cloudexchange.config file, and update the JWT secret key from the primary node:
```
cp cloudexchange.config.example cloudexchange.config
vi cloudexchange.config
```

Stop and disable the Cloud Exchange service:

sudo systemctl stop cloud-exchange && sudo systemctl disable cloud-exchange

Copy the existing CA key from /opt/cloudexchange/cloudexchange/data/ssl_certs/mongodb_rabbitmq_certs/tls_cert_ca.key of the primary node.
Paste it at the same path on the secondary node. The file name must be exactly tls_cert_ca.key.
Execute setup on the secondary node. After setup is complete on a new node, the new certificates for that node will be generated and will be signed by an existing CA, and this will enable the Management server communications between the new node and the HA Cluster.
```
sudo ./setup
```

After you set up the primary node and the secondary nodes, continue with Add Secondary Nodes into HA Cluster. This step is required.

Deploy HA for a Containerized Deployment

Use this track when you installed Cloud Exchange on your own Linux host. You clone the Cloud Exchange GitHub repository on every machine in the cluster.

Set Up a Primary Node

Clone the netskopeoss/ta_cloud_exchange GitHub repository on all the machines where Cloud Exchange will be deployed, and go to the repository directory:
```
mkdir netskope
cd netskope
git clone https://github.com/netskopeoss/ta_cloud_exchange
cd ta_cloud_exchange
```
If you already have the repository cloned, remove any local changes and pull the latest version:
```
git reset --hard
git pull
```
Check out the version that you want to deploy. For example, to check out version 6.1.0:
```
git checkout v6.1.0
```
(optional) Copy and edit the cloudexchange.config file. Edit the variables as per your requirements, and save the file:
```
cp cloudexchange.config.example cloudexchange.config
vi cloudexchange.config
```
Important

If you already created the cloudexchange.config file on this machine, do not run the copy command again. It replaces your file with the example file and removes your settings.

Important

Non-ASCII characters, such as accented letters, other scripts, and emoji, are not supported in the maintenance and JWT Secret passwords. Some processes might not work, and the system might fail, if you use them.

The maintenance and JWT Secret passwords are used for database authentication. You need these passwords when you restore a backup.

Stop and disable the Cloud Exchange service:

sudo systemctl stop cloud-exchange && sudo systemctl disable cloud-exchange

Execute setup:
```
sudo ./setup
```
Execute start to bring up the standalone instance:
```
sudo ./start
```
After the node is up, go to Settings > General > Node Configuration.
Click Enable HA, and confirm the choice. Observe the streamed logs. While enabling HA, the node will restart once.
After the node is up, validate the service status for RabbitMQ, MongoDB, Core, and UI.

Set Up a Secondary Node

Copy the JWT secret key from the primary node. If you don’t recall the JWT secret value of the primary node, generate and apply a new JWT token by following the steps in Cloud Exchange Troubleshooting.
Clone the netskopeoss/ta_cloud_exchange public GitHub repository on the secondary node, and go to the repository directory:
```
mkdir netskope
cd netskope
git clone https://github.com/netskopeoss/ta_cloud_exchange
cd ta_cloud_exchange
```
Check out the version that you want to deploy. For example, to check out version 6.1.0:
```
git checkout v6.1.0
```
In the folder where the repository is cloned, copy and edit the cloudexchange.config file, and update the JWT secret key from the primary node:
```
cp cloudexchange.config.example cloudexchange.config
vi cloudexchange.config
```
Important

If you already created the cloudexchange.config file on this machine, do not run the copy command again. It replaces your file with the example file and removes your settings.

Stop and disable the Cloud Exchange service:

sudo systemctl stop cloud-exchange && sudo systemctl disable cloud-exchange

Copy the existing CA key from ta_cloud_exchange/data/ssl_certs/mongodb_rabbitmq_certs/tls_cert_ca.key of the primary node.
Paste it at the same path on the secondary node. The file name must be exactly tls_cert_ca.key.
Execute setup on the secondary node. After setup is complete on a new node, the new certificates for that node will be generated and will be signed by an existing CA, and this will enable the Management server communications between the new node and the HA Cluster.
```
sudo ./setup
```

After you set up the primary node and the secondary nodes, continue with Add Secondary Nodes into HA Cluster. This step is required.

Both deployment types continue below with Add Secondary Nodes into HA Cluster. This step is required, and you complete it after you set up the nodes in either section above.

Add Secondary Nodes into HA Cluster

After the setup script is executed successfully in all nodes, log in to the primary node and go to Settings > General > Node Configurations.
Click Add New Node, and enter the FQDN/IP address of the new node.
Click the + icon beside the new node to proceed adding it as a new node.
Observe the streamed logs. While adding a new node, the primary node will also restart.

HA Node Management

To add a new secondary node to a cluster, go to Settings > General > Node Configurations.
Click Add and provide the secondary node IP address, and then click the add (+) icon.

Use the Stop icon besides the node to stop a node.

User can also view the historical node for the node setup process using the Logs icon.

Use the Delete icon to remove a secondary node from the cluster.

Similarly, use the Restart icon to restart any node from the Cloud Exchange UI.

In the event you want to migrate back to standalone mode, watch this video to learn how.

Hardening Guidelines

In order to establish the necessary connectivity between Docker services across various machines and facilitate the integration of nodes into the cluster, we have exposed the ports listed below from the Docker services. It is imperative that these ports remain accessible from all machines where the Cloud Exchange HA deployment is intended. To enhance security measures, it is also advisable to restrict access to these ports from other IP addresses.
These ports will be used for the clustering of the MongoDB and RabbitMQ services. Also the UI ports are required to perform a health check from all the machines.
- 4369 (A peer discovery service used by RabbitMQ nodes and CLI tools)
- 8000 (Used by CE Management Server)
- 5671 (Used by AMQP 0-9-1 and AMQP 1.0 clients without TLS)
- 15671 (HTTP API clients, management UI and rabbitmqadmin without TLS)
- 25672 (Used for inter-node and CLI tools communication)
- 35672 (Used for CLI tools communication)
- 27017 (The default port for mongodb and mongos instances)
- Selected UI port (Default 443 for HTTPS and 80 for HTTP) (To access the UI and internode healthcheck)
GlusterFS installation and setup will be driven through the Cloud Exchange UI and management server, which will act as shared storage and will also have HA capabilities. All the VMs involved in the HA cluster should have connectivity to GlusterFS repo during HA configuration.

Cluster Node Count Requirements

Total Number of CE Nodes in an HA cluster.	Minimum Number of ACTIVE/ONLINE CE Nodes Required in an HA Cluster at any Given Time for Successful Operation.
3	2
4	3
5	3
6	4
7	4
8	5
9	5

Limitations

When setting up HA on RHEL machines, make sure the systems can reach the GlusterFS repositories. On Ubuntu 24.04, GlusterFS is installed directly from Ubuntu’s own repositories.
Running multiple redundant instances of the Cloud Exchange requires additional hardware and computing resources.
In some cases, HA setups may introduce increased latency due to the need to replicate data or coordinate between active instances.
Whenever there are modifications to IP addresses or node-related configurations, it will be necessary to execute both the setup and start scripts on all machines within the system to ensure that the changes are properly propagated and synchronized across the cluster. It is recommended to add all the required machines at once and use a fixed IP addresses for the machines.
It is crucial to maintain the majority of nodes operational at all times, as the failure to do so would result in cluster failure. For example,in a 3-node cluster, at least 2 nodes must be operational at any given time; similarly, in a 5-node cluster, at least 3 nodes should be up and running. The pattern continues accordingly for larger clusters. This is because MongoDB will not initiate the Primary election process under such circumstances. Additionally, we have implemented a pause_minority option for RabbitMQ to mitigate the challenges associated with network partition-induced inconsistencies. This should recover once the node comes back up and connects to the cluster again. Refer to Cluster Node Count Requirements to see the minimum required operational nodes in a cluster.
Ref: https://www.mongodb.com/docs/v5.0/core/replica-set-elections/#network-partition | https://www.rabbitmq.com/partitions.html#options
In rare cases, when a cluster is facing back-to-back node failures, there are chances of data loss in the RabbitMQ because the queue will become leader and mirror very frequently, and in such cases, other nodes may lose the data which is not synchronized yet.
Ref: https://www.rabbitmq.com/ha.html#behaviour
To use SSO, we will be able to add only one node’s IP address as a redirect URL. Optionally, we can configure the load balancer against all the IP addresses, and use IP address of the load balancer, to configure the SSO. And it will redirect the request to any available node.

Troubleshooting HA in Cloud Exchange

The ./stop script in HA will remove the MongoDB and RabbitMQ node from the cluster. And stop the services in that particular machine. It is crucial to have the services up while running the ./stop script. Otherwise the script might fail as the script won’t be able to create the Mongo client to remove the node.
During migration, if you have any issues with node renaming, check the prerequisite and restore the backup from 1st step. Then retry the migration from 2nd step.
```
sudo rm -rf <path-to-ta_cloud_exchange>/data/mongo-data/data/* 
sudo cp -R <temp-mongo-path>/* 
<path-to-ta_cloud_exchange>/data/mongo-data/data/
```
Make sure docker/podman services are running. If not, run the services using this command
```
sudo systemctl restart docker
```
If the UI shows a down status for any container for a long period of time, check the status of the container and perform a reboot if required. Use podman wherever applicable.
- Check the container status. If all the services are running move to the next step. Otherwise, run the start script again to recover the container.
```
docker ps
```
- Check for the MongoDB cluster status. Execute these commands to run the command inside the MongoDB container.
```
docker compose exec -- mongodb-primary bash 
mongosh -u root -p $MONGO_INITDB_ROOT_PASSWORD 
rs.status()
```
  If the status shows anything other than PRIMARY and SECONDARY for a long period of time, run this command to restart the container on a specific machine.
```
docker compose restart mongodb-primary
```
- Check the RabbitMQ cluster status. Execute these commands to run the command inside the RabbitMQ container.
```
docker compose exec -- rabbitmq-stats bash 
rabbitmqctl cluster_status
```
  Check the running node section in the output and figure out the connected and running nodes. Restart the RabbitMQ container if required.
```
docker compose restart rabbitmq-stats
```
If any node experiences a worker lost error, like if the worker is not connected with RabbitMQ, restart the specific core container to recover the workers. This issue is likely to occur when the RabbitMQ cluster is down for a long period of time.
```
docker compose restart core
```
GlusterFS installation errors while starting Cloud Exchange:
- While running the start script if you see the error failed to install glusterfs, you can ignore this error if you wish to use Cloud Exchange as a standalone instance and do not need to enable an HA cluster.
- If you wish to use the HA clustering, make sure the environment has connectivity to the relevant GlusterFS repo and ensure that the system requirements are met before proceedings.