Data flow rules and filters

Introduction to DTS

CYBERQUEST's Data Transformation Services is a general multipurpose data intelligence component used in treating and manipulating event data. DTS is a parsing service based on a script (JavaScript) that has a wide range of functions. Its primary role is performing additional transformations on data extracted from collected events. Typical usage involves extracting useful information from multiple fields, depending on several factors, when the source event does not split its useful information in to separate fields.

Data Transformation Services extends and expands native CYBERQUEST capabilities with custom logic to further enrich and complete use cases:

• can help prepare data for future easy access, allowing data to be transformed and loaded from heterogeneous sources;
• can be used for event data correlation;
• adds, modifies, removes, enriches and obfuscates partial or complete event data;
• can remove an event;
• dynamically creates and fulfills default set up lists with data collected from incoming events (for example, a list of Blacklist IPs). Built in lists can be modified/added in Object Manager;
• can extract information from previously defined lists and take specific actions, such as sending alerts;
• uses a built-in JavaScript engine, the administrator being able to engage out-of-the-box or create custom logic for performing various actions;
• can send alerts;
• encrypts and decrypts data;
• can send HTTP/HTTPS requests for external systems/APIs;
• can query data stored in events database;
• and more.

DTS Service stores in memory lists of objects that the user can use to do specific tasks. For example, User logged on from specified IP ADDRESS list can be accessed by using the following script:

if(this.inputEvent.UserName == 'undefined')
{
    this.inputEvent.UserName = LocalStorage.get('UserLists',this.inputEvent.SrcIP); 
}

Generic Lists

if(this.inputEvent.EventType == 'undefined')
{
    this.inputEvent.EventType = LocalStorage.get(‘list_name’,this.inputEvent.PropertyName);
}

Custom lists with dynamic objects can be created, which are shared between all components. The lists can be populated directly from DTS by using the "set" method from LocalStorage object. To achieve this, use the following method:

if(this.inputEvent.EventType == ‘16’) //*16 means Failed Audit event*//
{
    LocalStorage.set('hosts_with_failed_audit',this.inputEvent.Computer, this.inputEvent.Computer);
}

LocalStorage object exposes several methods of interacting with user lists

set() Method

Example:

LocalStorage.set(listName, objectName, objectValue, ttl=900)

Description: This method stores user defined data into internal object lists which can also be edited in the Object Manager.

Parameters:

• listName defines the list on which we operate. This parameter is required.

• objectName defines the object name which is used for setting/retrieving data. This parameter is required.

• objectValue defines the value which can be stored. This parameter is optional. When it is not specified, the system will automatically use a time to live of 900 seconds. When -1 is used, the system will store the date indeterminately.

get() Method

Example:

LocalStorage.get(listName, objectName)

Description: This method retrives user defined data from internal object lists which can also be edited in the Object Manager.

Parameters:

• listName defines the list on which we operate. This parameter is required.

• objectName defines the object name which is used for setting/retrieving data. This parameter is required.

JS parsers (DTS objects)

Parsing or syntactic analysis is the process of analyzing a string of symbols, either in natural language or in computer languages, conforming to rules of a formal grammar. The parser's task is essential to determine if and how the input can be derived from grammar's start symbol.

JS Parser is a Javascript object that uses event logs and intelligently sorts data, making it easier for a user to interpret resulted information. Parsing is done by calling obj.exec with the event as a parameter in JSON Format.

JS parsers can be accessed from Web Interface by navigating to Settings > Rules > DTS Objects. DTS Objects page opens, listing defined objects and here you have options for importing an object or creating a new DTS object from scratch. In Actions menu includes options for edit, delete or export a parser, and you can mark parsers as active or inactive.

To export a parser, press button next to it. The export is saved as a proprietary CQO file. Likewise, to import a parser select in DTS Objects page.

To edit details for a specific object, press button next to it. Edit DTS Objects page opens allowing you to change the Name and Description, the Script or active/inactive the object.

To delete a parser from the list, press button next to it. As a measure of precaution, you will be asked to confirm deletion.

To mark a parser as activate/inactivate, press button.

Creating a new JS parser

In DTS Objects page, select button and configuration page opens allowing you to create the script for a new parser and mark it as active or inactive. When you finished creating the parser, press "Save" button to save changes.

Here is a sample JS parser:

Complete the following fields:

• In Name field, type a name that identifies the newly created DTS Object

• In Description field, insert a description of the object

• Use the field is Active to activate/inactivate the object

• In Script field, add the javascript code

When you finished creating the parser, press "Save" button to save changes or press "Cancel" button to discard changes.

To see how to create a parser, please follow the link: How to create a parser.

Filter rules

CYBERQUEST uses an intelligent event filter mechanism for sending data to JS parsers. You can instruct DTS on how to filter events sent to parsers by creating filter rules.

Filter rules can be managed by navigating to Settings > Rules > Filter Rules. Filter Rules page opens allowing rules to be defined based on operators like "eq", "noteq", "isInList", "isNotInList", "startsWith", "endsWith", "intInterval". There is practically no limit to adding additional fields.

The Actions menu includes options for edit, delete or export a rule, and you can mark rules as active or inactive. Filter Rules page opens, listing defined rules and here you have options for importing an object or creating a new filter rule from scratch.

To export a rule, press button next to it. The export is saved as a proprietary CQO file. Likewise, to import a rule select in Filter Rules page.

To edit details for a specific rule, press button next to it. Edit Filter Rule page opens allowing you to change the Filter Name and Description, as well as add, correct or delete already defined filters.

To delete a rule from the list, press button next to it. As a measure of precaution, you will be asked to confirm deletion.

To mark a rule as activate/inactivate, press button.

Creating a new filter rule

In Filter Rules page, select button and configuration page opens allowing you to create a new rule:

Complete the following fields:

• In Filter Name field, type a name that identifies the newly created filter rule

• In Description field, insert a description of the filter rule

• Press Add field condition button to define one or more condition to filter data

A filter rule can have one or more condition defined:

• In Select Field drop-down list, select the event field you want to filter

• In Operator drop-down list select from one of the available operators eq, noteq, isInList, isNotInList, startsWith, endsWith, intInterval.

• Type in the Value used for comparison

By default, the new rule will be active. Press on switch to inactivate the rule.

At any moment you can press to delete a filter.

When you finished creating the rule, press button to save changes and return to rules list.

Data acquisition rules (DA Rules)

CYBERQUEST includes a decisional engine based on rules that allows to combine filter rules with JS parsers for the purpose of establishing a granular data collection flow. As such, for a given flow of events, you will be able to assign one or more parsers to specific filtering rules and this way instruct the platform on what events will be collected.

Data acquisition rules can be accessed from Web Interface by navigating to Settings > Rules > DA Rules. DA Rules page opens, listing defined rules, and here you have options for importing and object or creating a new DA rule from scratch. The Actions menu includes options for edit, delete or export a rule, and you can mark as active or inactive.

To export a rule, press button. The export is saved as a proprietary CQO file. Likewise, to import a rule select in DA Rules page.

To edit details for a specific rule, press button. Edit DA Rule page opens allowing you to change the Name and Description, as well as add, correct or delete already defined filters.

To delete a rule from the list, press button next to it. As a measure of precaution, you will be asked to confirm deletion.

To mark a rule as activate/inactivate, press button.

To change the order in which rules are applied, simply drag and drop. Please note the last rule in list applies last and therefore may supersede other uphill rules.

Creating a new DA rule

In DA Rules page, select button and configuration page opens allowing you to create a new rule:

Complete the following fields:

• In Name field, type a name that identifies the newly created data acquisition rule. This name will appear in DA Rules list

• In Description field, insert an explanatory rule description

• AND filter rules allow the usage of AND operators in order to get information on multiple layers

• OR filter rules allow the usage of OR operators in order to get information on multiple layers

• Data Storages allows you to select the long-term data storage to use

• In DTS Objects drop-down list, select the DTS objects for which this rule will apply

• In Order field, set a usage priority for the newly created data acquisition rule

• Send data to short term storage allows you to send data to elasticsearch database

• Send to data correlation allows you to send data to data correllation service

• Active allows you to mark a DA Rule as activate/inactivate

When you finished creating the DA Rule, press "Save" button to save changes or press "Cancel" button to discard changes.