DSM Editor (part two)

This is the second part of the article about DSM Editor. Please find the link here to the first part of this article. As mentioned there, DSM Editor can create a new Log Source, based on repeating information in any kind of log.

Sample Log

Suppose, that you are dealing with logs collected from the Facility Centre. Logs contain Time and date, IP address of the reader and a room number. There are also details regarding accessed doors and swipe card used as an electronic key, Room Temperature and Room Humidity. Below a small  sample of these logs:

22 May 2019 19:00:00 SmartReader 10.0.0.2 22/may/2019:19:00:00 room A32 Reader:door open   roomTemp:19 roomHumidity:45%
22 May 2019 19:00:01 SmartReader 10.0.0.3 22/may/2019:19:00:01 room B15 Reader:door closed roomTemp:20 roomHumidity:45%
22 May 2019 19:00:02 SmartReader 10.0.0.5 22/may/2019:19:00:02 room H23 Reader:card swipe  roomTemp:18 roomHumidity:40%
22 May 2019 19:00:03 SmartReader 10.0.0.5 22/may/2019:19:00:03 room H23 Reader:card error  roomTemp:18 roomHumidity:40%
22 May 2019 19:00:04 SmartReader 10.0.0.5 22/may/2019:19:00:04 room H23 Reader:card swipe  roomTemp:18 roomHumidity:41%
22 May 2019 19:00:05 SmartReader 10.0.0.5 22/may/2019:19:00:05 room H23 Reader:door open   roomTemp:18 roomHumidity:42%
22 May 2019 19:00:06 SmartReader 10.0.0.2 22/may/2019:19:00:06 room A32 Reader:door closed roomTemp:19 roomHumidity:44%
22 May 2019 19:00:07 SmartReader 10.0.0.3 22/may/2019:19:00:07 room B15 Reader:card swipe  roomTemp:20 roomHumidity:45%
22 May 2019 19:00:08 SmartReader 10.0.0.3 22/may/2019:19:00:08 room B15 Reader:door open   roomTemp:20 roomHumidity:45%
22 May 2019 19:00:09 SmartReader 10.0.0.2 22/may/2019:19:00:09 room A32 Reader:door closed roomTemp:19 roomHumidity:44%

After you start to receive these logs in QRadar, apparently you will see them assigned to SIM Generic Log DSM. The first field from each event will be the Start Time. This is the timestamp for when the Event Collector received the raw event.

By default, each event found in QRadar has three timestamps. Apart from Start Time QRadar is using also the Storage Time. This is the timestamp for when the Event Processor stored the normalized event in its database. If you would notice any difference between Start Time and Storage Time, it may indicate some issue in your system. It can be caused for example that the system is not storing events fast enough or problems with hard drives. There is also a Log Source Time. The timestamp that the log source recorded in the raw event. If Log Source Time is much different than Start and Storage Time, it was probably recorded in a different timezone. Otherwise, you need investigate isn’t the wrong time set on the appliance from which the specific event is coming from.

Creating Log Source

After filtering the sample of events similar to the one above, run DSM Editor and create a New Log Source type.

You can call it for this example Smart Reader.

Adding a log source of the newly created type to QRadar let us collect the events. For this example, as Log Source Identifier you can use a phrase SmartReader (which could be an example of the computer hostname).

You should disable the coalescing option in this case. This option will only display the first 3 events within 10 seconds when the events appear with the same QID, Source IP, Destination IP, Destination port, and Username. The fourth event will have coalesced with all other events in the same appearance until the end of this 10-second interval. Instead of the number of events, you would see just one event with the displayed number which indicates how many events combined into one. This process is for saving the storage space, and it was quite important in the past but not now. Obviously, during configuring a new log source this is mostly not useful because every event is important.

After adding a new Log Source as usual you need to Deploy Changes. When it has done, you will see on Log Activity your new Log Source with these events assigned to it but all the events still will have only a name Unknown in the column Event Name. Our task now will be to teach QRadar how to name each event based on data found in the log.

Mapping of Event Properties

In order to do it, we are going to use DSM Editor again. A property called Event ID is the minimum to create a proper mapping of Event. You can find it from the side pane among existing properties. Because this property is already present in the system, so we need to override it. With Override system behaviour selected, QRadar SIEM creates mentioned in the previous part a Log Source Extension (LSX). Adding LSX extension on top of Log Source let the system to extract information for the specific property value from a log using a regular expression (Regex) and replace it the existing one according to our needs.

After selecting the property you need to add the expression. In this particular case, we search for the phrase reader preceded by the Regular Expression \s for space: {\sReader:}. Two words following after semicolon (take look on the sample log now), we would like to use as Event ID, so we construct the following expression {\sReader:(\w\s\w)} where \w stands for a word in Regex world. Surrounding that part in brackets means, that we are interested only in this part of the extracted sentence and we are going to ignore word Reader while using only brackets part. To show this interest in DSM Editor we use $1 sign in Format String field. This is sometimes called capturing group and please note that we can have more than one capturing group in our expression (in brackets) or even mix the order of capturing groups.

When you have Event ID extracted from the content of a log, you can create a new Event Mapping by clicking the blue plus sign in DSM Editor. Once the window opens, type Event ID (say, door open), then you can add Category name and finally create a new QID where you create own name for this event (it can be the same like Event ID). Since now each time QRadar extract as Event ID door open value then it will display the name which you have chosen.

Creating Custom Event Properties

With a similar approach, you can extract your own Custom Event Properties (CEP) from the content of the log. Any CEP can be displayed in the Log Activity after adding its column in Log Activity, but all possible CEPs for a specific event we can find when Event Properties opens.

In summary, using DSM Editor is an extremely easy task, so watch this move from Jose Bravo to see how simple it is.