Migrating URL Lists to Destination Profiles

The Destination Profile feature is replacing the URL Lists feature. This article outlines how to migrate your URL lists to destination profiles.

URL Lists Versus Destination Profiles

Destination profiles use and support different objects and syntax than URL lists. Before you begin migrating your URL lists, review the differences between these two features.

There are two types of URL lists available: enhanced and legacy. Enhanced URL lists support regular expressions (regex), while legacy URL lists do not. To confirm the type of URL lists enabled on your tenant, go to the URL Lists page and click New URL List.

Enhanced URL Lists

If you have enhanced URL lists, consider the following differences between your URL lists and destination profiles:

• Supported Objects
  Destination profiles support port, path, and query for all destination types, including CIDR and ranges. These are not supported in enhanced URL lists.
• Schemes
  Enhanced URL lists support schemes like http://*.domain.com, which the proxy evaluates as port 80. However, destination profiles disallow schemes. To perform this same behavior with a destination profile, you must specify a port (e.g., *.domain.com:80).
• CIDR and Ranges
  Enhanced URL lists define CIDRs and ranges as written (e.g., 163.116.128.0/17 and 163.116.128.80-81). For destination profiles, you must include prefixes to your CIDR and ranges. For example: CIDR:163.116.128.0/17 and RANGE:163.116.128.80-81.
• Case Sensitivity
  For enhanced URL lists, you can set case sensitivity for paths and queries globally with the URL Case Insensitivity Match option. With destination profiles, you can set case sensitivity per profile.
• Longest Prefix Match
  Destination profiles do not support longest prefix match. If your tenant was created before October 2021, then your enhanced URL lists use longest prefix match by default. With longest prefix match, in a scenario where you have *.example.com and www.example.com in different URL lists, the proxy only matches the list with the longer definition for requests to www.example.com. You can disable this behavior with the Custom URL Lookup Longest Match Mode option.

Legacy URL Lists

If you have legacy URL lists, consider the following differences between your URL lists and destination profiles:

• Supported Objects
  For destination profiles, matching based on resolved IPs is supported for IPs, CIDR, and ranges. Ports and regex are also supported. These are not supported in legacy URL lists. 
• PQDN
  Legacy URL lists only match subdomains. For example, *.example.com only matches subdomains like www.example.com and not the top domain example.com. Destination profiles match both the subdomain and top domain.
• Longest Prefix Match
  Destination profiles do not support longest prefix match. Legacy URL lists use longest prefix match by default. With longest prefix match, in a scenario where you have www.example.com and www.example.com/somepath in different URL lists, the proxy only matches the list with the longer definition when a request targets www.example.com/somepath. The www.example.com list doesn’t match. You can disable this behavior with the Custom URL Lookup Longest Match Mode option.

About the Migration

You can complete the migration for URL lists to destination profiles via the UI or APIs, one URL list at a time or in bulk. 

There are two stages to the migration. The first stage is duplicating your URL lists as destination profiles. The second is migrating the newly created destination profiles to your custom categories (i.e., replacing the URL lists with the destination profiles). Migration is complete once all URL lists are duplicated as destination profiles and all of your custom categories reference the new destination profiles instead of the URL lists.

A message at the top of the URL Lists page provides details on the scope of the migration. Click the number in the In Custom Category to view a list of all related categories for the URL list.

Stage 1: Duplicating URL Lists as Destination Profiles

To begin the migration process, you must first duplicate your URL lists as destination profiles. This causes no impact on your environment until the next stage of the migration.

Assisted migration is available from the URL Lists page or via the following REST APIs: POST /api/v2/profiles/destinations/migrate and PATCH /api/v2/profiles/destiantions/{id}/migrate. 

When duplicating your URL lists as destination profiles, note the following:

• You can only duplicate applied URL lists. 
• When duplicating URL lists, the syntax automatically updates to maintain the configured behavior.
• The source URL list is not deleted once it is duplicated.
• During this stage, you can review all duplication changes as well as apply or revert changes. Note that changes are not automatically applied and you must manually apply all changes. 
• You can duplicate a URL list multiple times. When a URL list is duplicated again, you can either update the existing destination profile or create a new one. 
• The latest duplication (i.e., the most recently created destination profile) is recorded and shown in the Netskope UI.
• If you modify a URL list after duplicating it, then the Netskope UI will display a warning.

Stage 1 Workflow Examples

The following examples include duplicating a single URL list, duplicating multiple URL lists, reduplicating a URL list and updating the destination profile, and reduplicating a URL list and creating a new destination profile.

Example: Duplicating a Single URL List

The following is an example workflow for duplicating a single URL list from the URL Lists page for the first time.

1. From the URL Lists page, click Duplicate to Destination Profile for a URL list. 

The Dest Profile Migration column updates for the URL list.

2. Go to the Destination Profiles page and review the content of the new destination profile, i.e., the converted syntax from the original URL list. 

3. Click Apply Changes.

Example: Duplicating Multiple URL Lists

The following is an example workflow for duplicating multiple URL lists in bulk from the URL Lists page for the first time.  

1. From the URL Lists page, select the URL lists and then click Duplicate to Destination Profile. 

2. The Bulk Duplicate to Destination Profiles message appears. Click Duplicate.

The Duplicating URL Lists to Destination Profiles window appears and displays the progress of the duplication.

3. When the process is complete, go to the Destination Profiles page and review the pending destination profiles.

4. Click Apply Changes.

Example: Reduplicating a URL List and Updating the Destination Profile

The following is an example workflow for duplicating a URL list that was already duplicated and then updating the existing destination profile. In this example, the URL list was updated after the first duplication.

1. For the URL list, click Duplicate Again and then To Override the Duplicated Destination Profile. 

2. Go to the Destination Profiles page and review the pending changes to the destination profile before applying them.

Example: Reduplicating a URL List and Creating a New Destination Profile

The following is an example workflow for duplicating a URL list that was already duplicated and then creating a new destination profile. In this example, the URL list was updated after the first duplication. 

1. For the URL list, click Duplicate Again and then To a New Destination Profile. 

The name of the new destination profile appears in the Dest Profile Migration column.

2. Go to the Destination Profiles page to view and apply the new destination profile. The old destination profile will remain unchanged. 

Stage 2: Migrating Destination Profiles to Custom Categories

To complete the migration process, you must update your custom categories with the newly created destination profiles, replacing the URL lists. 

Assisted migration is available from the URL Lists page or via the following REST API: PATCH /api/v2/profiles/customcategories/{id}. 

When migrating your destination profiles to custom categories, note the following:

• You must apply all destination profile changes before migrating to custom categories.
• You can only update applied custom categories.
• During this stage, you can review all migration changes as well as apply or revert changes. Note that changes are not automatically applied and you must manually apply all changes.
• Rolling back changes is possible even after applying changes. To roll back a change, manually edit the custom category and replace the destination profile with the original URL list.
• Netskope recommends that you avoid editing a URL list that’s no longer referenced in a custom category to prevent confusion. Instead, once all custom categories referencing a URL list are migrated, make future changes in the new destination profile. 

Stage 2 Workflow Examples

The following examples include migrating a single destination profile and migrating multiple destination profiles.

Example: Migrating a Single Destination Profile

The following is an example workflow for migrating a single destination profile (i.e., a duplicated URL list) to custom categories from the URL Lists page.

1. From the URL Lists page, click the number button in the In Custom Category column for the duplicated URL list. 

2. The Migrate Categories window appears. Select the custom categories to update and click Migrate. 

3. Go to the Custom Categories page to review the pending changes to the custom categories.

4. Click Apply Changes. 

Once the migration for the URL list is complete (i.e., all custom categories now use the new destination profile), the In Custom Category column on the URL Lists page updates to zero.

Example: Migrating Multiple Destination Profiles

The following is an example workflow for migrating multiple destination profiles (i.e., multiple duplicated URL lists) in bulk to custom categories from the URL Lists page.

1. From the URL Lists page, select the URL lists and then click Migrate Categories. 

2. The Bulk Migration In Category window appears. All custom categories that reference the selected URL lists are automatically selected. Click Migrate. 

The Migrating Categories Referencing From URL Lists To Destination Profiles window appears and displays the progress of the migration. 

3. When the process is complete, go to the Custom Categories page and review the pending changes to the custom categories.

4. Click Apply Changes. 

Completing the Migration

Once all URL lists have been duplicated and the custom categories no longer reference them, the URL Lists page will no longer display the migration message.