Skip to content

Tag based Parsing

Internal CQ DataFlow

Data is collected from various sources using CYBERQUEST's Collecting Agent (WMI, ODBC or file-level gathering etc.), or receiving data flows (syslog, NetFlow etc.). Data is organized in queues, sent to a Data Acquisition Service (DAS), which applies acquisition rules and then sends raw data to a Data Transformation Service (DTS). DTS is responsible for parsing data and generate real-time alerts.

DataFlow

Data Server - is responsible for receiving events and pre-processing them. After this process, the events are sent to the Data Acquisition service using the internal queue service (RabbitMQ).

Windows Agent - send events from Windows stations to the Data Acquisition service using the internal queuing service (RabbitMQ).

RabbitMQ - is responsible for the internal queue management of the application.

Data Acquisition - it processes events using API parsers. After the events have been processed, they are sent to: online database (Elasticsearch), Data Storage service and Data Correlation service.

Data Storage - this service encrypts and compresses events in the archive. The Data Storage service allows the CYBERQUEST application to maintain huge archive repositories. In projects where data needs to be kept for very long periods of time and with immediate access requirements.

Data Correlation - generate alerts based on real time and correlate events for alerting.

Elastisearch - is the online database.

Automatic parsing

Automatic parsing is performed by the Data Server service together with the Data Acquisition service.

The Data Server service receives events, pre-processes them and sends them to the Data Acquisition service for processing.

Events that have been pre-processed by the Data Server are given a tag and Data Acquisition processes them according to the tag given by the Data Server.

If the events have not been pre-processed by the Data Server, they will not be processed by the Data Acquisition service and the following must be added data source for that event type. Data sources can be added from the web application in the Settings -> Management -> Data Source Manager page.

Tag Alias

Tag alias is a function that allows parsing events using a parser other than the original one given by the data server.

In order to use Tag Alias you have to identify the Initial Tag in the Browser Module. Initial Tag could have: EventID 100000, another initial tag which doesn't exist in Data Acquisition or without data on STRINGFIELDS.

To access this function you have to open Tag alias page following the steps bellow:

a) Navigate to Settings Settings by expanding in the top-right corner of the interface, then click Management > Tag Alias:

TagAlias

b) Tag Alias window will open:

TagAlias

To add a new Tag Alias that you found in Browser Module, press AddTagAlias button and the window will open:

TagAliasPage

Complete the following:

  • Initial Tag: the Tag which you found in Browser Module that is not processed in CYBERQUEST.
  • Final Tag: you can choose from the predefined list the data sources tag. The events will be processed with the used Final Tag.

Edit EditButton button on the right corner allows you to edit selected group.

To delete a group from the list, press Delete Button button next to it. As a measure of precaution, you will be asked to confirm deletion.