Devo Foundations - Exercise 4.5 - Analyzing IDS data

  • 30 January 2023
  • 1 reply
  • 24 views
Devo Foundations - Exercise 4.5 - Analyzing IDS data
Userlevel 4
Badge

This time we are going to focus on the Suricata Intrusion Detection System (IDS) that Arcadia uses. It stores its logs into the ids.suricata.fast data table.

  • Open this data table.
  • Set the time range to the last 3 days. There is no built-in preset interval for that, so use the calendar to build it yourself.
  • Locate the signature field, and hover over its header. We are looking for signatures containing the term “TROJAN”.
  • We can expect more than one single type of trojan occurrence. So we need to use an operation than filters by including anything with that term.
  • Use the operation contains, represented by ->
    • The first argument is the field signature.
    • For the second argument, click on the ✏️ icon and add “TROJAN”.
      • Don’t enter the quote symbols.
      • This contains (->) operation is case sensitive, so write it in uppercase.
  • Now, where are those connections associated with trojan activity going to? Where are their destinations? To address that, we can check the dest_ip field.
  • Let’s make use of the handy ispublic operation and filter private IPs out. Click on the Create field button and type “is public” to find that operation. Select dest_ip as the argument.
  • The filtering phase is good enough, so let’s try to enrich our data to make more sense of these trojans.
  • We are going to get the location associated to those destination IPs. We could extract the city, but let’s try a different approach.
  • Click on the Create field button and create two fields using the following operations:
    • Get the latitude with MaxMind2 latitude (mm2latitude).
    • Get the latitude with MaxMind2 longitude (mm2longitude).
  • To the next phase, grouping. Let’s group every minute by host, and the new fields for latitude and longitude.
  • Then, to the aggregation phase. Click on the Aggregate button and do a simple count operation.
  • Now go to Additional tools / Charts / Maps and select Google heatmap.
  • Add latitude and longitude in their specific placeholders and the count field for Size/Color.
  • What areas of activity can you spot? Which countries?

Spoiler alert: Solution

This the equivalent LINQ query for these operations. Note that the name of enriched fields does not have to coincide with yours.

 

from ids.suricata.fast
  where `signature` -> "TROJAN"
  where ispublic(dest_ip)
  select mm2latitude(dest_ip) as latitude
  select mm2longitude(dest_ip) as longitude
  group every 1m by host, latitude, longitude
  select count() as count

 


 

Still scrolling? Then you might want to read this extra task:

Back in the filtering process, when you scrolled through the list of operations looking for the contains LINQ operation, you might have noticed other operations with similar descriptions besides the sensitivity to case.

As you can see, we have contains (or has or ->) and also contains tokens (or toktains).

 

Token

A token in this context is just a set of characters separated by symbols. For example, this fake email address, “mike.oldfield@devo.com” would contain 4 tokenized items:

  • mike
  • oldfield
  • devo
  • com

Learn more

 

  • After this interlude, go back to the query and try to filter the signature field by using tokens, looking for “TROJAN”.
  • Now, imagine that you were not aware of the field where “TROJAN” is appearing and figuring it out might be time-consuming. You can use toktains over the raw field. As in:
    from ids.suricata.fast
      where toktains(raw,"trojan")
    • This “raw” field contains the message unaltered, as is ingested, with all the fields condensed into one single field.

      Using toktains over this field is especially suited when you don’t know in which field the information you are looking for is

  • Use the tool Stat count in Additional tools and compare the total number of events of a) the unfiltered table, b) the table filtered with contains over signature, c) the table filtered with toktains over signature, d) the table filtered with toktains over raw.

1 reply

Userlevel 2

Hi Alex,

I couldn’t find the tool “Stat Count” to explore the last part of the exercise; I’m not sure I’m looking in the right place:

 

 

Thanks in advance and have a great weekend ahead!

 

Reply