Choosing Data Sets - Netskope Knowledge Portal

Each stream type can collect a different set of logs. A data set lets users define the log collections to be sent and select the delivery format.

Each stream type can collect different sets of data. Admins can select from the following data sets: (highlighted in the image above)

Transaction Events
Page Events
Application Events
Network Events
Endpoint Events
Incidents
Alerts

There is a max limit of seven streams per tenant. One stream for Transaction Events and up to six streams for Alerts and Events.

Alerts & Events contain the following logs: Alerts, Incidents, Page Events, Application Events, Network Events, and Endpoint Events.

When admins select Alerts and Events, they are streamed together.

You cannot stream a data collection multiple times, e.g. if you select Transaction Events as your data set, you cannot create a second stream for transaction events.

If you select Transaction Events as your data set, the options for Alerts & Events are grayed out, as shown below.

You can select one or more of the other data sets which make up the Alerts & Events (Page Events, Application Events, Network Events, Endpoint Events, Incidents, and Alerts). Note, Transaction Events and Alerts & Events cannot be combined into a single stream. Transaction Events is grayed out if you select any of the Alerts & Events data sets, as shown below.

Selecting Parser Order (Transaction Events Only)

For Transaction Events streams, admins can configure two additional options when modifying the data set fields (click the pencil icon on the Transaction Events data set). These options apply to Transaction Events streams only; they are not available for Alerts & Events data sets.

Parser Order

Select the Parser Order from the Manage field.

PARSER ORDER	FIELD COVERAGE
Parser Order 1	Supports fields up to and including Transaction Events Format 3 (legacy format).
Parser Order 2	Supports all fields of the Transaction Events Universal (TE Universal) format.

Transaction Events Universal is the recommended, fully customizable format introduced with 197 fields, and it will be extended over time. Legacy Formats 1–4 have a rigid field list. For the complete field reference of each format, see Transaction Events Formats.

If you require fields introduced after Format 3 (e.g., authentication fields such as x-c-authn-user, device fields such as x-c-hostname, or threat protection fields such as x-tp-result), select Parser Order 2 (TE Universal). Parser Order 1 limits the available fields to the Format 3 field list. You must validate with the respective SIEM vendor for your organization if they support field filtering and assorted selection.

Delimiter

In the same modified view, admins can also select the delimiter used in the delivered CSV log files for the Transaction Events stream. This allows the output to align with the parsing requirements of the downstream SIEM or analytics tool.

Field-level Filtering

Admins can perform smart filtering to send only what matters with field-level filtering, reducing noise and cost for SIEM’s. By default the system sends all fields.

1. Select a data set and click the pencil icon.

2. The Field-level Filtering page opens.

Options include:

Filtering the fields – start typing a field name to narrow the list
Preview – you must select at least one radio button at the end of the field to preview
Enable or Disable All – click the carrot to make a selection.
Radio button – toggle on to send data or toggle off to not send data
Drag and drop – move the entire row to a specific position in the list

Click Preview. Once you’re satisfied with your selections, click Save.

For Transaction Events streams, the fields available in the Field-level Filtering page depend on the selected parser order: Parser Order 1 exposes fields up to Format 3, while Parser Order 2 exposes the full TE Universal field list.

Choosing Log Delivery Format Types

Admins can select the format in which they want to receive the log files, currently CSV is the only option (#2 in the image at the top of the article).

Choose the compression type, GZIP or ZSTD (#3 in the image at the top of the article).