Performance degradation in QRadar on ecs-ec

Performance degradation occurs in QRadar on two main services ecs-ec and ecs-ep. Depends on service, which is affected (sometimes it can be on both at the same time), you need to understand different cause and different solution. In this article we will discuss only a problem on ecs-ec. EC stands for events collector and you can assume the problem is caused by wrong matching events with designed Log Source. Be aware that this is far different problem from Performance degradation on ecs-ep.

When performance degradation on ecs-ec happens in QRadar appliance, the one of the possible reasons is the wrong detection of an event source. This ends up with the not correct assignment to Log Source. For example, in this case, we know that events sending Windows workstation. We created Log Source for our Sysmon events as described in the documentation:

Despite that, we see that QRadar recognized events as TippingPoint IPS device from Trend Micro company but not as Sysmon.

What is more, this auto-discovery process repeats each time we cut existing TippingPoint Log Source and number of events sent to storage is growing causing Performance Degradation. This happens because events, which sending Windows do not match TippingPoint pattern. As a result, the system expects more resources than if the same events would parse Windows Security Event Log Source.

Traffic Analysis

The event collector (ecs-ec) includes the following components: Protocol, Throttle, Parsing, Log source traffic analysis and auto-discovery, Coalescing and Event Forwarding.

Events received by QRadar for auto-discovery when device addresses do not yet have to match a configuration. The Traffic Analysis component performs this detection. Suitable DSMs testing each event to see if it can recognize that type of event. Also, Traffic Analysis keeps a statistic for each device type/IP combination of successful vs. unsuccessful recognition.

The configuration file has own recognition thresholds and it compares gathered statistics. QRadar creates automatically a sensor device configuration and assigns the events to that device type when a specific pair device type/IP meet an expected threshold. QRadar correctly routes events through to the newly created device within a few seconds after its creation.

Please note that each host (console or MH) (prior 7.3.1) has its own copy of TrafficAnalysisConfig.xml to be used for ECS. Each host has own version of the configuration file, separately tuned. QRadar does not copy over this file during deploying changes. The file is in the following place:

/opt/qradar/conf/TrafficAnalysisConfig.xml

Within five seconds of any changes made to the configuration file, ecs detects changes and automatically reloads them or immediately after restarting ecs.

When existing log source can’t recognize the data, that goes through “Event Parsing”, the QRadar sends it to “Traffic Analysis” for further auto detection. Please note, that auto-detection process doesn’t include definitions for ALL log sources available in the system. You can check the configuration file TrafficAnalysisConfig.xml for the list of Log Sources. When QRadar detects events from a new source, sends them to configservices for further analysis. The “Traffic Analysis” filter/engine will only ignore events originating from a source for which a log source already exists and which is disabled. You can’t drop a log source, because QRadar will just auto-detect them again and recreate log source.

Problem resolution of Performance degradation

In order to resolve this issue, you can run a script called

/opt/qradar/bin/tatoggle.pl

and exclude problematic Log Source from Traffic Analysis by disabling it.

This resolved issue of performance degradation on ecs-ec.