Devo Foundations - Exercise 4.5 - Analyzing IDS data

This time we are going to focus on the Suricata Intrusion Detection System (IDS) that Arcadia uses. It stores its logs into the ids.suricata.fast data table.

• Open this data table.
• Set the time range to the last 3 days. There is no built-in preset interval for that, so use the calendar to build it yourself.

  

• Locate the signature field, and hover over its header. We are looking for signatures containing the term “TROJAN”.
• We can expect more than one single type of trojan occurrence. So we need to use an operation than filters by including anything with that term.
• Use the operation contains, represented by ->
  • The first argument is the field signature.
  • For the second argument, click on the ✏️ icon and add “TROJAN”.
    • Don’t enter the quote symbols.
    • This contains (->) operation is case sensitive, so write it in uppercase.
• Now, where are those connections associated with trojan activity going to? Where are their destinations? To address that, we can check the dest_ip field.
• Let’s make use of the handy ispublic operation and filter private IPs out. Click on the Create field button and type “is public” to find that operation. Select dest_ip as the argument.
• The filtering phase is good enough, so let’s try to enrich our data to make more sense of these trojans.
• We are going to get the location associated to those destination IPs. We could extract the city, but let’s try a different approach.
• Click on the Create field button and create two fields using the following operations:
  • Get the latitude with MaxMind2 latitude (mm2latitude).
  • Get the latitude with MaxMind2 longitude (mm2longitude).
• To the next phase, grouping. Let’s group every minute by host, and the new fields for latitude and longitude.
• Then, to the aggregation phase. Click on the Aggregate button and do a simple count operation.
• Now go to Additional tools / Charts / Maps and select Google heatmap.

  

• Add latitude and longitude in their specific placeholders and the count field for Size/Color.
• What areas of activity can you spot? Which countries?

Spoiler alert: Solution

Still scrolling? Then you might want to read this extra task:

Back in the filtering process, when you scrolled through the list of operations looking for the contains LINQ operation, you might have noticed other operations with similar descriptions besides the sensitivity to case.

As you can see, we have contains (or has or ->) and also contains tokens (or toktains).

• After this interlude, go back to the query and try to filter the signature field by using tokens, looking for “TROJAN”.
• Now, imagine that you were not aware of the field where “TROJAN” is appearing and figuring it out might be time-consuming. You can use toktains over the raw field. As in:
  from ids.suricata.fast

    where toktains(raw,"trojan")
  • This “raw” field contains the message unaltered, as is ingested, with all the fields condensed into one single field.

    

    Using toktains over this field is especially suited when you don’t know in which field the information you are looking for is. 

• Use the tool Stat count in Additional tools and compare the total number of events of a) the unfiltered table, b) the table filtered with contains over signature, c) the table filtered with toktains over signature, d) the table filtered with toktains over raw.