Tag based Parsing
Internal CQ DataFlow
Data is collected from various sources using CYBERQUEST's Collecting Agent (WMI, ODBC or file-level gathering etc.), or receiving data flows (syslog, NetFlow etc.). Data is organized in queues, sent to a Data Acquisition Service (DAS), which applies acquisition rules and then sends raw data to a Data Transformation Service (DTS). DTS is responsible for parsing data and generate real-time alerts.
Data Server - is responsible for receiving events and pre-processing them. After this process, the events are sent to the Data Acquisition service using the internal queue service (RabbitMQ).
Windows Agent - send events from Windows stations to the Data Acquisition service using the internal queuing service (RabbitMQ).
RabbitMQ - is responsible for the internal queue management of the application.
Data Acquisition - it processes events using API parsers. After the events have been processed, they are sent to: online database (Online DataStorage), Data Storage service and Data Correlation service.
Data Storage - this service encrypts and compresses events in the archive. The Data Storage service allows the CYBERQUEST application to maintain huge archive repositories. In projects where data needs to be kept for very long periods of time and with immediate access requirements.
Data Correlation - generate alerts based on real time and correlate events for alerting.
Elastisearch - is the online database.
Automatic parsing is performed by the Data Server service together with the Data Acquisition service.
The Data Server service receives events, pre-processes them and sends them to the Data Acquisition service for processing.
Events that have been pre-processed by the Data Server are given a tag and Data Acquisition processes them according to the tag given by the Data Server.
If the events have not been pre-processed by the Data Server, they will not be processed by the Data Acquisition service and the following must be added data source for that event type. Data sources can be added from the web application in the Settings -> Management -> Data Source Manager page.
Tag alias is a function that allows parsing events using a parser other than the original one given by the data server.
In order to use Tag Alias you have to identify the Initial Tag in the Browser Module. Initial Tag could have: EventID 100000, another initial tag which doesn't exist in Data Acquisition or without data on STRINGFIELDS.
To access this function you have to open Tag alias page following the steps bellow:
a) Navigate to Settings in the left side of the interface where are the quick access buttons, then click Management > Tag Alias:
b) Tag Alias window will open:
To add a new Tag Alias that you found in Browser Module, press button and the window will open:
Complete the following:
- Initial Tag: the Tag which you found in Browser Module that is not processed in CYBERQUEST.
- Final Tag: you can choose from the predefined list the data sources tag. The events will be processed with the used Final Tag.
Edit button on the right corner allows you to edit selected group.
To delete a group from the list, press button next to it. As a measure of precaution, you will be asked to confirm deletion.
Enhance parsing is translating events and adding additional information to what Windows already provides. Windows Events Enricher adds the following information on the Forensics field:
- Who - describes who created the event
- What - the Category of the event
- Where - describes the location where the event occurred
- Why - describes why the event was created
For example, go to Browser Module and search the EventID: 4625 and you will find the category field which is named _FORENSICS. There you will see 4 forencsics fields of Windows Enricher:
- Who: "UserName/Account Name"
- What: "Category"
- Where: "Computer"
- Why: ""