Routing data in QRadar
There are two options for routing data in QRadar:
Online:
Forwarding takes place during the QRadar event pipeline as part of ECS-EC (event correlation service – event collection) process.
It can be described as real-time streaming of data as it is in the event pipeline, the Event Forwarding process that lives in ECS-EC routes the data live to the destination address. This is a best effort forwarding as if the destination is not listening or the connection is offline then QRadar sends the data stream and expects the other side to be present to receive the communication.
In online mode, potential lost of data, when the forwarding destination is unreachable, is possible because streaming data off appliance to a Syslog system, another SIEM, etc. is provided as it is, without any protection. If the remote destination has no active listener for the Syslog stream or the network is down, then the data will not be received. In this case is, you need to remember that the remote system would not receive the data, however, a copy of what is written to Ariel and it would still be available, unless Drop was also selected. You can send data to forwarding destination as well as to Event Processor for storage and processing again As data is going through the ECS pipeline on the appliance, send allows you to pick and choose data out of the event pipeline and forward it to a unique destination. The data you select is copied, meaning that the existing data continues through the QRadar event pipeline and a copy of the data matching the routing rule is forwarded off appliance.
When you select DROP, the event pipeline drops the matching data at the “Event Forwarding / Routing” stage, which is the last step in ECS-EC. Dropped data will never see the CRE, because the data is dropped before we enter ECS-EP where the first process is the CRE (Custom Rules Engine).
As data is going through the event pipeline, data matching to the routing rule filter is forwarded to the off-site destination (in theory to another SIEM or Big Data system for correlation). The data continues through the ECS pipeline, but we do not want to generate an offense based off of the data that we just sent to the Off-site destination, so the ECS pipeline flags that results to skip rule correlation in QRadar.
Offline:
QRadar delivers data around 1 minute behind and it will keep track of where it was, in case if the link/destination goes down. It can pick up where it left off to make sure all data makes it through. This is sent through a different process that is actually reading the raw ariel data files off disk as they become available. Datanodes also will be sending data, if offline forwarding is in use.
This data is not streamed, but this is event forwarding done via storage. When you select Offline, then a process called the “Offline Forwarder” is responsible for working with Ariel reader to retrieve events from the disk after it is written to storage and decide what data should be forwarded.
This process checks for a connection to the destination at startup, looks to see where the last successful data was sent (using bookmarks), and forwards the results to the remote destination. This data is always at least 1 minute behind as Ariel writes data in 1 minute intervals to disk. As data is forwarded, a flat file is created as a “bookmark” in /store/offline forwarder for either events or flow data to show the timestamp and next file that needs to be read based off of the last successful file that sent to the destination. If for some reason the bookmark files are missing, Ariel Reader will create a new starting point 2 minutes behind the current timestamp to use as a start time.
Because offline mode exists after data is written to disk and only occurs after the event pipeline is complete then Drop and Bypass correlation are not available in Offline Mode. The drop function occurs during “Event Forwarding/Routing” and CRE bypass are part of the ECS pipeline and you are already beyond the pipeline, when you talking about offline forwarding.