Migrate Cloud Exchange - Netskope Technical Documentation

Cloud Exchange supports these migration options:

Current CE Version	Migration Path
v5.1.1	v6.1.0
v5.1.2	v6.1.0
v6.0.0	v6.1.0
v6.0.1	v6.1.0

Before proceeding with the migration, identify your Cloud Exchange deployment type to ensure you follow the appropriate migration path. See How to Identify your Cloud Exchange Deployment Type.

Notes

Customers using Cloud Exchange version 3.x or 4.x who wants to migrate to the latest version, refer to this Cloud Exchange KB article.
Customers using Cloud Exchange version 5.0.1 in a containerized deployment and planning to migrate to the latest version, refer to this Cloud Exchange KB article .
Customers using Cloud Exchange version 5.0.1 in a VM deployment and planning to migrate to the latest version, refer to this Cloud Exchange KB article .

Containerized Deployment

Important

Contact your SE/AM if you have questions regarding Cloud Exchange installation, deployment, configuration, and migration.

Prerequisites

Before initiating migration, ensure your instance meets the system requirements for Cloud Exchange.
Disable all Source Plugins in all modules, then wait until in-queue tasks finished before proceeding with the migration. Use these steps to identify in-queue tasks mentioned below.
1. After disabling the source plugins in all modules, wait for 20-30 minutes.
2. Apply filter on logs:
  1. Go to the Logging section.
  2. Click Filter Query.
  3. In the Filters input, enter the following: message Like "Ingested " || message Like "Stored " || message Like "task(s)" || message Like "Completed storing"
  4. Click Load.
  5. Apply the filter to view relevant logs.
3. Monitor Logs: Once the logging filter is applied, monitor for new logs. When no new logs appear under this filter, you can proceed to the next step.
4. After migrating to the latest CE version by following steps mentioned below as per your current deployment and CE version, please enable all previously disabled source plugins.
Ensure you have the maintenance password readily available, as it will be needed to complete certain steps in this process.

Important

If the maintenance password is lost, the data could not be retained.

Notes

RabbiMQ data migration is not supported due to the change of queue type from Classic to Quorum queue.

Standalone Deployment

To v6.1.0 from v5.1.1, v5.1.2, v6.0.0, v6.0.1 using script (Standalone)

This script is used to migrate to Latest Cloud Exchange Standalone. This script needs to be executed in the destination (New) machine.

Prerequisites for this script

When migrating from a containerized machine, you must have the ssh credentials (root password or pem file based credentials) of the old machine where CE is installed.
In order to run the migrate_ce script, you must have the latest Cloud Exchange repository cloned in new machine.
Use this command to clone the latest Cloud Exchange repository:
```
git clone https://github.com/netskopeoss/ta_cloud_exchange
```

Steps

In the New machine, go the directory where latest Cloud Exchange is cloned and run below command to copy and edit the cloudexchange config file.
```
cp cloudexchange.config.example cloudexchange.config
vi cloudexchange.config
```
Update the cloudexchange configuration file with value for required fields. Also, Make sure to configure optional fields like proxy configuration, ui port etc if used/customised.
```
MAINTENANCE_PASSWORD=<old cloud exchange maintenance password>
JWT_SECRECT=<old cloud exchange JWT secret> (Optional, Setup script will generate new JWT secret if not provided)
```
In the New machine, Run below command:
```
sudo ./migrate_ce
```
In the next step, It will ask for the old machine’s username, machine IP, authentication method and the password of the user in case of user based authentication or file path of PEM file in case of PEM file based authentication.
After successful connection with the old machine, it will run these commands automatically in order to migrate:
1. It will execute Stop Script in old machine.
2. It will zip the existing data in the old machine and transfer it on the new machine.
3. After successful transfer to the new machine, it will unzip the file in the current installation package folder on the new machine.
4. Now it will run the setup script and ask for the necessary inputs in order to up the Latest Cloud Exchange.
5. In the end, it will run the Start script.
To check the status of the container run this command:
sudo docker ps. In case of podman, use sudo podman ps
Now open the UI of newly migrated Latest Cloud Exchange in the web browser. Log in with the CE UI credentials to check the configurations and data and ensure everything is working properly.

HA Deployment

To v6.1.0 from v.5.1.1, v5.1.2, v6.0.0, v6.0.1 Standalone Deployment

Before proceeding, ensure that all prerequisites have been met. Verifying these requirements in advance is essential to avoid potential issues during the process.
In order to transfer the data from the older standalone to the latest container based HA, it is necessary to copy the data from the old standalone and transfer it to the latest HA primary node.
To transition copied data into the new HA configuration, refer to the HA deployment guide for instructions on adding the necessary HA parameters and initializing the cluster. This process facilitates the migration of MongoDB data into the replica set. Additionally, it involves the importation of RabbitMQ messages into the new HA machine, with subsequent integration of other nodes into the cluster.
Go to ta_cloud_exchange directory into the current standalone machine.
```
cd ta_cloud_exchange
```
Stop the containers into the current standalone machine.
```
sudo ./stop
```
Go to data directory in current standalone machine.
```
cd data
```
Create a zip file for Mongo data after going to the data directory.
```
sudo zip -r ce_backup.zip mongo-data/ repos/ plugins/
```
Add custom plugins to the backup zip. This step is applicable only if you’re using custom plugins.
```
sudo zip -r ce_backup.zip custom_plugins
```
Copy the backup to the new instance using this scp command.
```
sudo scp ce_backup.zip <username>@<ip-of-vm>:<ta_cloud_directory>/data
```
On Primary Node Only(the node you intend to designate as primary)
Go to the new node of the HA machine which contains the latest version of Cloud Exchange.
```
cd <ta_cloud_directory>/data
```

Extract the data into zipped folders.

sudo unzip ce_backup.zip "mongo-data/*" -d .

sudo mkdir /opt/shared/data/ -p

sudo unzip ce_backup.zip "plugins/*" "repos/*" "custom_plugins/*" -d /opt/shared/data/

Go back to the original working directory.
```
cd ..
```

Copy and edit the cloudexchange config file using below commands:

cp cloudexchange.config.example cloudexchange.config
vi cloudexchange.config

Update the cloudexchange configuration file with value for required fields. Also, Make sure to configure optional fields like proxy configuration, ui port if used/customised.

MAINTENANCE_PASSWORD=<old cloud exchange maintenance password>
JWT_SECRECT=<old cloud exchange JWT secret> (Optional, Setup script will generate new JWT secret if not provided)

Run the setup script in the primary node first using this command.
```
sudo python3 ./setup
```
Execute the start script using the below command.
```
sudo ./start
```
Next, open the UI of the primary node in your web browser. Log in using your existing Cloud Exchange credentials. You can access the UI using the system’s IP address: https://<ip>:<port>
Navigate to Settings > General Settings > Node Configurations and enable the Enable HA toggle. A Confirm Action pop-up window will appear. Enter the Cloud Exchange IP address or FQDN only if you need to update it; otherwise, leave it unchanged and click the Enable HA button.

Important

During the HA enablement process, Cloud Exchange will be temporarily unavailable and may restart multiple times.

On Secondary Nodes Only
Navigate to the newly cloned ta_cloud_exchange directory using the following command:
```
cd <ta_cloud_exchange-dir>
```

Copy the existing CA key from primary node located at:

<ta_cloud_exchange-dir>/data/ssl_certs/mongodb_rabbitmq_certs/tls_cert_ca.key

Create new CA key on secondary nodes using below command:

vi /data/ssl_certs/mongodb_rabbitmq_certs/tls_cert_ca.key

Paste the CA key copied from primary node during step no.20 and save the file. This will enable the Management server communications between new node and HA Cluster.

Copy and edit the cloudexchange config file.

cp cloudexchange.config.example cloudexchange.config
vi cloudexchange.config

Update the cloudexchange configuration file with values for
```
JWT_SECRET=<JWT_SECRET value provided in primary node>
MAINTENANCE_PASSWORD=<old maintenance password>
```
Notes

For secondary nodes, the JWT secret and CA key must be the same as the primary node. Using different JWT secrets or CA keys will cause authentication errors when adding the secondary node to the HA cluster from a primary node UI.

If you don’t recall the JWT Secret value of the primary node, generate and apply new JWT token by following the steps here.
Execute the setup script using the below command.
```
sudo python3 ./setup
```
Add the secondary nodes one by one through the CloudExchange UI of the primary node in the Settings > General Settings > Node Configurations section.

Notes

For secondary nodes, there is no need to manually run the start script—it is automatically executed when adding the secondary nodes through the CloudExchange UI of the primary node during Step 26.

To v6.1.0 from v.5.1.1, v5.1.2, v6.0.0, v6.0.1 HA Deployment

Before proceeding, ensure that all prerequisites have been met. Verifying these requirements in advance is essential to avoid potential issues during the process.
In order to transfer the data from the current container based HA to the latest container based HA, it is necessary to make a copy of data in the current HA primary node and transfer it to the latest containerized HA primary node.
To ensure proper container shutdown, stop both Secondary nodes before stopping the Primary node using this command.
```
sudo ./stop
```
Copy the data from the primary node. Create a zip file for Mongo data after going to the data directory.
```
cd <ta_cloud_exchange_directory_path>/data 
sudo zip -r ce_backup.zip mongo-data/
```
Go to shared drive to copy plugins and repos folder. For v6.0.0 or above, shared drive path will be /opt/shared/data/.
```
cd <shared_drive> 
sudo zip -r <ce_backup_zip_location>/ce_backup.zip repos/ plugins/
```
This step is applicable only if you’re using custom plugins. Add custom plugins to backup zip.
```
sudo zip -r <ce_backup_zip_location>/ce_backup.zip custom_plugins/
```
In order for the migration to be successful, you need to transfer data from the current machine primary node to the primary node new machine, which has the latest version of Cloud Exchange.
```
sudo scp ce_backup.zip <username>@<ip-of-vm>:<ta_cloud_exchange_directory_path>/data
```
On Primary Node Only
Go to the primary node of the new HA machine which contains the latest version of Cloud Exchange.
```
cd <ta_cloud_exchange_directory_path>/data
```

Extract the data into zipped folders and then navigate back to the original working directory.

sudo unzip ce_backup.zip "mongo-data/*" -d . 
sudo mkdir /opt/shared/data/ -p 
sudo unzip ce_backup.zip "plugins/*" "repos/*" "custom_plugins/*" -d /opt/shared/data/
cd ..

Edit the cloudexchange config file.

cp cloudexchange.config.example cloudexchange.config
vi cloudexchange.config

Update the cloudexchange configuration file with values for required fields. Also, make sure to configure optional fields like proxy configuration, ui port etc if used/customised.

HA_ENABLED=True
HA_CURRENT_NODE=<current node ip>
HA_PRIMARY_NODE_IP=<current node ip>
HA_IP_LIST=<current node ip>
MAINTENANCE_PASSWORD=<old maintenance password>
JWT_SECRET=<old cloud exchange JWT secret> (Optional, Setup script will generate new JWT secret if not provided)

Execute the setup script using the below command.
```
sudo python3 ./setup
```
To migrate the Mongo data, use this command one time only.
```
sudo ./restore_ha_backup
```
Execute the start script using the below command.
```
sudo ./start
```
Run this command after the UI is accessible. If you are using Containerized CE both standalone and HA, use podman-compose instead of docker compose for RHEL base OS.

On Secondary Nodes Only
Navigate to the newly cloned ta_cloud_exchange directory using the following command:
```
cd <ta_cloud_exchange_directory_path>
```

Copy the existing CA key from primary node located at:

<ta_cloud_exchange_directory_path>/data/ssl_certs/mongodb_rabbitmq_certs/tls_cert_ca.key

Create new CA key on secondary nodes using below command:
```
vi /data/ssl_certs/mongodb_rabbitmq_certs/tls_cert_ca.key
```
Paste the CA key copied from primary node and save the file. This will enable the Management server communications between new node and HA Cluster

Copy and edit the cloudexchange config file.

cp cloudexchange.config.example cloudexchange.config
vi cloudexchange.config

Update the cloudexchange configuration file with values for
```
JWT_SECRET=<JWT_SECRET value provided in primary node>
MAINTENANCE_PASSWORD=<old maintenance password>
```
Notes

For secondary nodes, the JWT secret and CA key must be the same as the primary node. Using different JWT secrets or CA keys will cause authentication errors when adding the secondary node to the HA cluster from a primary node UI.

If you don’t recall the JWT Secret value of the primary node, generate and apply new JWT token by following the steps here.
Execute the setup script using the below command.
```
sudo python3 ./setup
```
Add the secondary nodes one by one through the CloudExchange UI of the primary node under the Settings > General Settings > ‘Node Configurations’ section.

Notes

For secondary nodes, there is no need to manually run the start script—it is automatically executed when adding the secondary nodes through the CloudExchange UI of the primary node during Step 22.

Cloud Exchange as a VM Deployment

Prerequisites

Before initiating a migration, ensure your instance meets the system requirements for Cloud Exchange.
For Cloud Exchange as a VM, connectivity to following URL is required:
https://cloud-exchange-store.s3.us-east-1.amazonaws.com
Disable all Source Plugins in all modules, then wait until in-queue tasks finished before proceeding with the migration. Use these steps to identify in-queue tasks mentioned below.
1. After disabling the source plugins in all modules, wait for 20-30 minutes.
2. Apply filter on logs:
  1. Go to the Logging section.
  2. Click Filter Query.
  3. In the Filters input, enter the following: message Like "Ingested " || message Like "Stored " || message Like "task(s)" || message Like "Completed storing"
  4. Click Load.
  5. Apply the filter to view relevant logs.
3. Monitor Logs: Once the logging filter is applied, monitor for new logs. When no new logs appear under this filter, you can proceed to the next step.
4. After migrating to the latest CE version by following steps mentioned below as per your current deployment and CE version, please enable all previously disabled source plugins.
Ensure you have the maintenance password readily available, as it will be needed to complete certain steps in this process.

Important

If the maintenance password is lost, the data could not be retained.

Notes

RabbiMQ data migration is not supported due to the change of queue type from Classic to Quorum queue.

Customers running the CE as a VM (OVA or Hyper-V) on versions earlier than 5.1.1 will require root credentials.
Username: root
Password: M5#w6V+.T^8gv?%,

Azure and AWS CE as VM customers can upgrade to Latest Version of Cloud Exchange by themselves by following these steps.

Standalone Deployment

To v6.1.0 from v5.1.1, v5.1.2, v6.0.0, v6.0.1 Standalone Deployment using script

This script is used to migrate to Latest Version of Cloud Exchange Standalone. This script needs to be executed in the destination (New) machine.

Prerequisites for this script

When migrating from CE as a VM machine, the script will work with the password of the cteadmin user.

Steps

In the New CE as a VM machine, go to the /opt/cloudexchange/cloudexchange folder and run below command to copy and edit the cloudexchange config file.
```
cp cloudexchange.config.example cloudexchange.config
vi cloudexchange.config
```
Update the cloudexchange configuration file with value for required fields. Also, make sure to configure optional fields like proxy configuration, ui port etc if used/customised.
```
MAINTENANCE_PASSWORD=<old cloud exchange maintenance password>
JWT_SECRECT=<old cloud exchange JWT secret> (Optional, Setup script will generate new JWT secret if not provided)
```
In the new CE as a VM machine, go the directory where latest Cloud Exchange is deployed and run this command:
```
sudo ./migrate_ce
```
In the next step, It will ask for the old machine’s username, machine IP and the password of the old machine or file path of PEM file in case of PEM file.
After successful connection with the old machine, it will run these scripts automatically in order to migrate:
1. It will run Stop Script.
2. It will zip the existing data in the old machine and transfer it on the new machine.
3. After successful transfer to the new machine, it will unzip the file in the desired folder on the new machine.
4. Now it will run the setup script and ask for the necessary inputs in order to up the Latest Version of Cloud Exchange.
5. In the end, it will run the Start script.
6. To check the status of the container run this command:
  sudo docker ps. In case of podman, use sudo podman ps)
7. Now open the UI of newly migrated Latest Version of Cloud Exchange in the web browser. Log in with the old credentials of CE to check the configurations and data and ensure everything is working properly.

HA Deployment

To v6.1.0 From v.5.1.1, v5.1.2, v6.0.0, v6.0.1 Standalone Deployment

Before proceeding, ensure that all prerequisites have been met. Verifying these requirements in advance is essential to avoid potential issues during the process.
In order to transfer the data from the older standalone to the latest CE as VM based HA, it is necessary to copy the data from the old standalone and transfer it to the latest HA primary node.
To transition copied data into the new HA configuration, refer to the HA deployment guide for instructions on adding the necessary HA parameters and initializing the cluster. This process facilitates the migration of MongoDB data into the replica set. Additionally, it involves the importation of RabbitMQ messages into the new HA machine, with subsequent integration of other nodes into the cluster. Make sure that /opt/shared/data directory is available for CE to store shared resources between cluster nodes.
Go to CE installation package directory. (For CE as a VM, go to /opt/cloudexchange/cloudexchange/)
Stop the CE as VM into the current standalone machine.
```
sudo ./stop
```
Go to data directory in current standalone machine.
```
cd data
```
Create a zip file for Mongo data after going to the data directory.
```
sudo zip -r ce_backup.zip mongo-data/ repos/ plugins/
```
Add custom plugins to the backup zip. This step is applicable only if you’re using custom plugins.
```
sudo zip -r ce_backup.zip custom_plugins
```
Copy the backup to the new cloud based instance using this scp command.
```
sudo scp -i <public_key> ce_backup.zip <username>@<public_ip>:/opt/cloudexchange/cloudexchange/data
```
On Primary Node Only(the node you intend to designate as primary)
Go to the new node of the HA machine which contains the latest version of Cloud Exchange.
```
cd /opt/cloudexchange/cloudexchange/data
```

Extract the data into zipped folders.

sudo unzip ce_backup.zip "mongo-data/*" -d .

sudo mkdir /opt/shared/data/ -p

sudo unzip ce_backup.zip "plugins/*" "repos/*" "custom_plugins/*" -d /opt/shared/data/

Navigate back to the original working directory.
```
cd ..
```

Copy and edit the cloudexchange config file using below commands:

cp cloudexchange.config.example cloudexchange.config
vi cloudexchange.config

Update the cloudexchange configuration file with value for required fields. Also, Make sure to configure optional fields like proxy configuration, ui port etc if used/customised.
```
MAINTENANCE_PASSWORD=<old cloud exchange maintenance password>
JWT_SECRECT=<old cloud exchange JWT secret> (Optional, Setup script will generate new JWT secret if not provided)
```
Run the setup script in the primary node first using this command.
```
sudo python3 ./setup
```
Execute the start script using the below command.
```
sudo ./start
```
Next, open the UI of the primary node in your web browser. Log in using your existing Cloud Exchange credentials. You can access the UI using the system’s IP address: https://<ip>:<port>

On Secondary Nodes Only
Navigate to the /opt/cloudexchange/cloudexchange directory using the following command:
```
cd /opt/cloudexchange/cloudexchange
```

Copy the existing CA key from primary node located at:

/opt/cloudexchange/cloudexchange/data/ssl_certs/mongodb_rabbitmq_certs/tls_cert_ca.key

Create new CA key on secondary nodes using below command:

vi /data/ssl_certs/mongodb_rabbitmq_certs/tls_cert_ca.key

Paste the CA key copied from primary node during step no.19 and save the file. This will enable the Management server communications between new node and HA Cluster.

Copy and edit the cloudexchange config file.

cp cloudexchange.config.example cloudexchange.config
vi cloudexchange.config

Update the cloudexchange configuration file with values for
```
JWT_SECRET=<JWT_SECRET value provided in primary node>
MAINTENANCE_PASSWORD=<old maintenance password>
```
Notes

For secondary nodes, the JWT secret and CA key must be the same as the primary node. Using different JWT secrets or CA keys will cause authentication errors when adding the secondary node to the HA cluster.

If you don’t recall the JWT Secret value of the primary node, generate and apply new JWT token by following the steps here.
Execute the setup script using the below command.
```
sudo python3 ./setup
```
Go to Settings > General Settings > Node Configurations and enable the Enable HA toggle. A Confirm Action pop-up window will appear. Enter the Cloud Exchange IP address or FQDN only if you need to update it; otherwise, leave it unchanged and click the Enable HA button.

Important

During the HA enablement process, Cloud Exchange will be temporarily unavailable and may restart multiple times.
Add the secondary nodes one by one through the CloudExchange UI of the primary node in the Settings > General Settings > Node Configurations section.

Notes

For secondary nodes, there is no need to manually run the start script. It is automatically executed when adding the secondary nodes through the CloudExchange UI of the primary node during Step 26.

To v6.1.0 from v.5.1.1, v5.1.2, v6.0.0, v6.0.1 HA Deployment

Before proceeding, ensure that all prerequisites have been met. Verifying these requirements in advance is essential to avoid potential issues during the process.
In order to transfer the data from the current CE as VM based HA to the latest CE as VM based HA, it is necessary to make a copy of data in the current HA primary node and transfer it to the latest CE as VM HA primary node.
To ensure proper CE as VM shutdown, stop both Secondary nodes and than stop the Primary node using below command:
```
sudo ./stop
```
Copy the data from the primary node. Create a zip file for Mongo database located inside the data directory.
```
cd /opt/cloudexchange/cloudexchange/data 
sudo zip -r ce_backup.zip mongo-data/
```
Go to shared drive to copy plugins and repos folder. From CE v6.0.0 or above, shared drive location will be /opt/shared/data/
```
cd <shared_drive>
sudo zip -r <ce_backup_zip_location>/ce_backup.zip repos/ plugins/
```
This step is applicable only if you’re using custom plugins. Add custom plugins to backup zip.
```
sudo zip -r <ce_backup_zip_location>/ce_backup.zip custom_plugins/
```
In order for the migration to be successful, you need to transfer data from the current machine primary node to the primary node new machine, which has the latest version of Cloud Exchange.
```
sudo scp -i <public_key> ce_backup.zip <username>@<public_ip>:/opt/cloudexchange/cloudexchange/data
```
On Primary Node Only
Go to the primary node of the new HA machine which contains the latest version of Cloud Exchange.
```
cd /opt/cloudexchange/cloudexchange/data
```

Extract the data into zipped folders and then navigate back to the original working directory.

sudo unzip ce_backup.zip "mongo-data/*" -d . 
sudo mkdir /opt/shared/data/ -p 
sudo unzip ce_backup.zip "plugins/*" "repos/*" "custom_plugins/*" -d /opt/shared/data/
cd ..

Edit the cloudexchange config file.

cp cloudexchange.config.example cloudexchange.config
vi cloudexchange.config

Update the cloudexchange configuration file with values for required fields. Also, make sure to configure optional fields like proxy configuration, ui port etc if used/customised.
```
HA_ENABLED=True
HA_CURRENT_NODE=<current node ip>
HA_PRIMARY_NODE_IP=<current node ip>
HA_IP_LIST=<current node ip>
MAINTENANCE_PASSWORD=<old maintenance password>
```
Execute the setup script using the below command.
```
sudo python3 ./setup
```
To migrate the Mongo data, use this command one time only.
```
sudo ./restore_ha_backup
```
Execute the start script using the below command.
```
sudo ./start
```
Run this command after the Primary node UI will be accessible.

On Secondary Nodes Only
Go to the newly cloned /opt/cloudexchange/cloudexchange directory using the following command:
```
cd /opt/cloudexchange/cloudexchange
```

Copy the existing CA key from primary node located at:

/opt/cloudexchange/cloudexchange/data/ssl_certs/mongodb_rabbitmq_certs/tls_cert_ca.key

Create new CA key on secondary nodes using below command:
```
vi /data/ssl_certs/mongodb_rabbitmq_certs/tls_cert_ca.key
```
Paste the CA key copied from primary node and save the file. This will enable the Management server communications between new node and HA Cluster

Copy and edit the cloudexchange config file.

cp cloudexchange.config.example cloudexchange.config
vi cloudexchange.config

Update the cloudexchange configuration file with values for
```
JWT_SECRET=<JWT_SECRET value provided in primary node>
MAINTENANCE_PASSWORD=<old maintenance password>
```
Notes

For secondary nodes, the JWT secret and CA key must be the same as the primary node. Using different JWT secrets or CA keys will cause authentication errors when adding the secondary node to the HA cluster from a primary node UI.

If you don’t recall the JWT Secret value of the primary node, generate and apply new JWT token by following the steps here.
Execute the setup script using the below command.
```
sudo python3 ./setup
```
Add the secondary nodes one by one through the CloudExchange UI of the primary node in the Settings > General Settings > Node Configurations section.

Notes

For secondary nodes, there is no need to manually run the start script—it is automatically executed when adding the secondary nodes through the CloudExchange UI of the primary node during Step 22.