Available troubleshooting options for the Cloud TAP stitcher include performance-based and reliability-based metrics for insight into common issues, as well as CLI options for configuring specific Cloud TAP stitcher settings to help you mitigate those issues.
Metrics for Troubleshooting
Cloud TAP metrics for troubleshooting are available in the INFO log and the HTTP interface in JSON format. These metrics allow you to diagnose performance issues and pinpoint root causes (e.g., network bottlenecks, latency, or etc.).
To access the troubleshooting metrics via the HTTP interface, use the following command to locally query the metrics: curl http://localhost:8000
Throughput Metrics
The metrics that provide details for the Cloud TAP stitcher throughput when downloading and processing traffic captures from BLOBs. These metrics are available under the app_stats field:
| Location | Field | Description |
|---|---|---|
| Cloud Storage | key_listing_duration_ms | The total time taken for listing the key BLOBs in a single batch from cloud storage. |
| Cloud Storage | data_listing_duration_ms | The total time taken for listing the data BLOBs in a single batch from cloud storage. |
| Cloud Storage | download_key_throughput_Mbps | The download throughput for the key BLOBs. |
| Cloud Storage | downloadKeyElapsedTimeMs | The total elapsed time for key downloads in a single batch. |
| Cloud Storage | batch_download_data_throughput_Mbp | The download and processing throughput for the data BLOBs. |
| Cloud Storage | batchDownloadDataElapsedTimeMs | The total elapsed time for data downloads and processing in a single batch. |
| Cloud Storage | thread_download_data_throughput_Mbps | The Pure Data for the download throughput. |
| NDR System | export_data_throughput_Mbps | The data processing and export throughput. |
| NDR System | exportElapsedTimeMs | The total elapsed time for data processing and export in a single batch. |
| NDR System | thread_export_data_throughput_Mbps | The Pure Data for the export throughput. |
The following is an example of the throughput metrics:
{
"app_stats": {
"batchDownloadDataElapsedTimeMs": 0,
"batch_blob_size_bytes": 0,
"batch_download_data_throughput_Mbps": 0.0,
"batch_throughput_Mbps": 0.0,
"cached_keys": 100000,
"data_listing_duration_ms": 0,
"downloadKeyElapsedTimeMs": 0,
"download_key_throughput_Mbps": 0.0,
"downloadfail_blobs": 0,
"elapsedTimeMs": 3,
"evicted_keys": 1041713,
"exportElapsedTimeMs": 0,
"export_data_throughput_Mbps": 0.0,
"fetched_keys": 1141757,
"fetchfail_keys": 0,
"key_listing_duration_ms": 0,
"packet_time_nonascending": 0,
"refetch_keyblobs": 1,
"skipped_blobs": 0,
"thread_download_data_throughput_Mbps": 0.0,
"thread_export_data_throughput_Mbps": 0.0
},
"decode_stats": {
"decryptedDataSize": 72169057488,
"decryptedPacketZeroSizeCount": 0,
"empty_packets": 76021398,
"empty_packets_size": 3048044664,
"exceededSegLimitFlowCount": 155,
"passThroughPacketSize": 31922039345,
"preDecryptionSize": 81697605957,
"reorder_packets": 18411,
"retransmitted_packets": 589284,
"retransmitted_packets_size": 241567181,
"syn_packets": 2179167,
"tls_stats": {
"cant_parse_secret": 0,
"client_random_missing": 12,
"no_secret": 3247,
"set_master_secret_error": 0,
"ssl_v3_sessions": 0,
"tls_bad_content_type": 397,
"tls_bad_data": 0,
"tls_bad_mac": 1105,
"tls_bad_pms": 75,
"tls_cipher_failure": 0,
"tls_decrypt_fail": 497865,
"tls_decrypt_success": 85422640,
"tls_no_chts": 0,
"tls_no_cts": 98944,
"tls_no_data": 1285,
"tls_no_decrypt": 11253,
"tls_no_shts": 2025,
"tls_no_sts": 126,
"tls_unknown_fail": 0,
"tls_v11_sessions": 86,
"tls_v12_sessions": 436884,
"tls_v13_sessions": 1196882,
"tls_v1_sessions": 170,
"total_tls_sessions": 1634022,
"unsupported_tls_version": 0,
"zero_client_random_requests": 13959
},
"total_packets": 201862028,
"total_packets_size": 123254066719,
"unknown_finack_packets": 0,
"unknown_packets": 1314865,
"unknown_packets_size": 993890152,
"unknown_rst_packets": 416026,
"unknown_synack_packets": 14
},
"timestamp": 1740988871552,
"tx_stats": {
"geneve_batch_tx_bytes": 0,
"geneve_packets": 0,
"geneve_tx_bytes": 0,
"geneve_tx_gbytes": 0,
"geneve_tx_kbytes": 0,
"geneve_tx_mbytes": 0,
"vxlan_packets": 0,
"vxlan_tx_bytes": 0,
"vxlan_tx_gbytes": 0,
"vxlan_tx_kbytes": 0,
"vxlan_tx_mbytes": 0
}
}
Debugging Metrics
The metrics that provide real-time information for the Cloud TAP stitcher after an issue occurs (e.g., a failed TLS decryption).
Note
The debugging metrics are available exclusively in the HTTP interface. Use the following command to locally query these metrics: curl http://localhost:8000
Slow Downloading Metrics
The metrics that provide details for BLOBs that took an unusually long amount of time to download from cloud storage. These metrics are available under the blobs field.
| Field | Description |
|---|---|
name | The name of the fetched BLOB. |
download_time | The start time for the downloading process in seconds. |
fetch_time_ms | The total elapsed time for fetching the BLOB in milliseconds. |
size | The total size of the fetched BLOB in bytes. |
The following is an example of the slow downloading metrics:
{
"blobs": [
{
"fetch_time_ms": 647,
"name": "data/1721665920/NSKP-HIPPO/ctap-default-75fdb676b4-gxbj2/83",
"process_time_ms": 99,
"size": 33247370,
"total_time_ms": 746
},
{
"fetch_time_ms": 622,
"name": "data/1721665920/NSKP-HIPPO/ctap-default-75fdb676b4-gxbj2/87",
"process_time_ms": 54,
"size": 33450197,
"total_time_ms": 676
}
]
}
Failed Downloading Metrics
The metrics that provide details for the BLOBs that failed to download, including the corresponding error. These metrics are available under the err_blobs field:
| Field | Description |
|---|---|
name | The name of the BLOB that failed to download. |
download_time | The start time of the failed downloading process in seconds. |
failure_reason | The corresponding error code or message for the failed download. |
The following is an example of the failed downloading metrics:
{
"err_blobs": [
{
"download_time": 1740374961228,
"failure_reason": "GetObject failed for object: data/1734025680/NSKP-HIPPO/ctap-default-7797647b9b-fr7zf/44168. Reason: curlCode: 28, Timeout was reached",
"name": "data/1734025680/NSKP-HIPPO/ctap-default-7797647b9b-fr7zf/44168"
},
{
"download_time": 1740374961228,
"failure_reason": "GetObject failed for object: data/1734025740/NSKP-HIPPO/ctap-default-7797647b9b-fr7zf/44169. Reason: curlCode: 28, Timeout was reached",
"name": "data/1734025740/NSKP-HIPPO/ctap-default-7797647b9b-fr7zf/44169"
}
]
}
Failed Decryption Metrics
The metrics that provide details for connections that failed TLS decryption. These metrics are available under the connection_errors field:
| Field | Description |
|---|---|
id | The generated unique ID. |
sport | The source port for the TCP connection. |
dport | The destination port for the TCP connection. |
client_random | The ClientHello value. This serves as the strong key when matching connections to secrets. |
ipv4_src | The source IP for the TCP connection. |
ipv4_dst | The destination IP for the TCP connection. |
sni | The Server Name Indication. |
ssl_error | The TLS-related error code. |
blob | The name of the BLOB. |
The following is an example of the failed decryption metrics:
{
"connection_errors": [
{
"blob": "data/1721665860/NSKP-HIPPO/ctap-default-75fdb676b4-nfbvb/8",
"dport": 80,
"id": "1:10.1.0.19:12737/10.111.127.187:80",
"ipv4_dst": "10.111.127.187",
"ipv4_src": "10.1.0.19",
"sni": "10.111.127.187",
"sport": 12737,
"ssl_error": 2
},
{
"blob": "data/1721665860/NSKP-HIPPO/ctap-default-75fdb676b4-nfbvb/7",
"dport": 80,
"id": "1:10.1.0.19:57153/10.111.127.187:80",
"ipv4_dst": "10.111.127.187",
"ipv4_src": "10.1.0.19",
"sni": "10.111.127.187",
"sport": 57153,
"ssl_error": 3
}
]
}
CLI Options for Troubleshooting
You can use the following CLI options to configure the following Cloud TAP stitcher settings for troubleshooting purposes.
Historical Metric Data Options
The following CLI options allow you to control the amount of historical metric data retained by the Cloud TAP stitcher. By default, the Cloud TAP stitcher retains approximately 1 minute of metric history.
Note
Netskope highly recommends avoiding querying the HTTP interface too frequently, as excessive polling can lead to unnecessary CPU and network usage.
| Parameter | Description |
|---|---|
--metrics-history-size UINT | The maximum number of historical metric entries retained. The default is 100. Each entry represents metrics generated at the end of a batch processing cycle, including BLOB downloads, processing statistics, and data exports. |
--max-blob-records UINT | The maximum number of records for BLOB downloads retained per thread. The default is 2000. |
--max-error-blob-records UINT | The maximum number of records for BLOB errors retained per thread. The default is 2000. |
--max-decrypt-error-count UINT | The maximum number of records for decryption errors retained per thread. The default is 4000. |
Connection & Retry Management Options
The following CLI options allow you to manage connection timeouts, behavior for data transfers, and connection retries between Cloud TAP and cloud storage, especially for on-premises deployments.
| Parameter | Description |
|---|---|
--conn-timeout INT | The connection timeout period for establishing connections to cloud storage in seconds. The default is 1. This is only applicable to AWS and Azure. |
--stall-timeout INT | The maximum amount of time allowed for inactivity during data transfers in seconds. The default is 3. This is only applicable to AWS and GCP. |
--request-timeout INT | The timeout period for cloud storage requests and data transfers in seconds. This is disabled by default and is only applicable to AWS. |
--max-retries INT | The maximum number of retries for failed cloud storage requests. The default is 1. |
--retry-delay INT | The delay period between retries for failed cloud storage requests in seconds. The default is 1. This is only applicable to GCP and Azure. |
--max-retry-delay INT | The maximum amount of time allowed for delays between retries for failed cloud storage requests in seconds. The default is 2. This is only applicable to GCP and Azure. |