Getting Started with Intrusion Detection System (IDS) Bro: Troubleshooting

The last three articles I’ve published step you through how to setup the Bro intrusion detection system (IDS) on Red Hat 7. You’ve read through installing prerequisites, compiling and installing Bro,  and configuring it for the first time. But despite anyone’s best efforts, there is likely to be some hiccups along the way. Your Bro workers might drop packets, hit a Bro bug, or perhaps a worker crashes. This post will examine some tools to help you diagnose common issues and unfold some potential causes and solutions with Bro.

Tools

HTop

HTop is a good CLI tool to monitor CPU across all cores on your system. It helps to see how the load is distributed across all the cores. If you are usually evenly distributed, such as in the above screenshot, that is a pretty good sign.

PF_Ring

To see if the PF_Ring driver loaded properly you can run:

dmesg | grep -i pf_ring

PF_Ring can give you some good statistics as found in:

cat /proc/net/pf_ring/info

This prints some basic settings about the card such as the number of slots. It also tells you various statistics and settings currently on the interface including number of packets received, number of packets dropped, capture direction, and so forth. When broctl netstats runs from the manager, it grabs the dropped packet statistics from this file.

cat /proc/net/pf_ring/<interface name>

To check other NIC settings, you can run:

ethtool -k <interface>

Often people disable most NIC settings that ethtool allows you to so that more processing power goes toward not dropping packets. Checksums can also interfere with Bro’s checksums and cause CUP spikes and dropped packets.

Dropped Packets

By far the most effort you put into Bro is by minimizing packet drops. The source of packets drops can be many things, some of which are explored on this page.

CPU Spikes or High CPU

Sometimes you see a single core or even multiple cores spike in processing. These can be hard to pin down.

Some other possibilities for CPU spikes are:

• Cron jobs
• Unsupported packet types (ie many NIC vendors don’t support MPLS traffic)
• Abnormal traffic patterns, such as oscillating between 0 to maximum throughput
• Confusing traffic, such as large dropped or out-of-order packets
• Mismatch checksums from the NIC

Some of the above issues could also lead to memory leaks.

Other Issues

In some cases the throughput of the PCI bus is not large enough. If you have a PCIe 2.0 slot that can only do 4Gbps, don’t expect to be able to handle traffic with microbursts higher than that.

Misconfigured NIC settings can also be a big issue. Whether it is checksums, or small rings, this is where you can really optimize the NICs.

Linux default settings are not optimized for high throughput. You will want to expand you operating system buffers as outlined in http://dak1n1.com/blog/7-performance-tuning-intel-10gbe/.

Troubleshooting Tips

The best approach is to capture the packets you are sending to Bro and see if replaying those packets can reproduce the issue. This usually means having a second device that can receive the same traffic, and that can also write to disk fast enough.  If you can replicate the issue consistently, then you can start eliminating variables to track down the source.

You can also try disabling the various protocol analyzers and scripts in Bro.  Disabling scripts alone isn’t enough. It’s best to disable both. Just place the following snippet into your local.bro file to disable the analyzers.

event bro_init() {
Analyzer::disable_analyzer(Analyzer::ANALYZER_AYIYA);
Analyzer::disable_analyzer(Analyzer::ANALYZER_BACKDOOR);
Analyzer::disable_analyzer(Analyzer::ANALYZER_BITTORRENT);
Analyzer::disable_analyzer(Analyzer::ANALYZER_BITTORRENTTRACKER);
Analyzer::disable_analyzer(Analyzer::ANALYZER_CONNSIZE);
Analyzer::disable_analyzer(Analyzer::ANALYZER_DCE_RPC);
Analyzer::disable_analyzer(Analyzer::ANALYZER_DHCP);
Analyzer::disable_analyzer(Analyzer::ANALYZER_DNP3_TCP);
Analyzer::disable_analyzer(Analyzer::ANALYZER_DNP3_UDP);
Analyzer::disable_analyzer(Analyzer::ANALYZER_CONTENTS_DNS);
Analyzer::disable_analyzer(Analyzer::ANALYZER_DNS);
Analyzer::disable_analyzer(Analyzer::ANALYZER_FTP_DATA);
Analyzer::disable_analyzer(Analyzer::ANALYZER_IRC_DATA);
Analyzer::disable_analyzer(Analyzer::ANALYZER_FINGER);
Analyzer::disable_analyzer(Analyzer::ANALYZER_FTP);
Analyzer::disable_analyzer(Analyzer::ANALYZER_FTP_ADAT);
Analyzer::disable_analyzer(Analyzer::ANALYZER_GNUTELLA);
Analyzer::disable_analyzer(Analyzer::ANALYZER_GSSAPI);
Analyzer::disable_analyzer(Analyzer::ANALYZER_GTPV1);
Analyzer::disable_analyzer(Analyzer::ANALYZER_HTTP);
Analyzer::disable_analyzer(Analyzer::ANALYZER_ICMP);
Analyzer::disable_analyzer(Analyzer::ANALYZER_IDENT);
Analyzer::disable_analyzer(Analyzer::ANALYZER_IMAP);
Analyzer::disable_analyzer(Analyzer::ANALYZER_INTERCONN);
Analyzer::disable_analyzer(Analyzer::ANALYZER_IRC);
Analyzer::disable_analyzer(Analyzer::ANALYZER_KRB);
Analyzer::disable_analyzer(Analyzer::ANALYZER_KRB_TCP);
Analyzer::disable_analyzer(Analyzer::ANALYZER_CONTENTS_RLOGIN);
Analyzer::disable_analyzer(Analyzer::ANALYZER_CONTENTS_RSH);
Analyzer::disable_analyzer(Analyzer::ANALYZER_LOGIN);
Analyzer::disable_analyzer(Analyzer::ANALYZER_NVT);
Analyzer::disable_analyzer(Analyzer::ANALYZER_RLOGIN);
Analyzer::disable_analyzer(Analyzer::ANALYZER_RSH);
Analyzer::disable_analyzer(Analyzer::ANALYZER_TELNET);
Analyzer::disable_analyzer(Analyzer::ANALYZER_MODBUS);
Analyzer::disable_analyzer(Analyzer::ANALYZER_MYSQL);
Analyzer::disable_analyzer(Analyzer::ANALYZER_CONTENTS_NCP);
Analyzer::disable_analyzer(Analyzer::ANALYZER_NCP);
Analyzer::disable_analyzer(Analyzer::ANALYZER_CONTENTS_NETBIOSSSN);
Analyzer::disable_analyzer(Analyzer::ANALYZER_NETBIOSSSN);
Analyzer::disable_analyzer(Analyzer::ANALYZER_NTLM);
Analyzer::disable_analyzer(Analyzer::ANALYZER_NTP);
Analyzer::disable_analyzer(Analyzer::ANALYZER_PIA_TCP);
Analyzer::disable_analyzer(Analyzer::ANALYZER_PIA_UDP);
Analyzer::disable_analyzer(Analyzer::ANALYZER_POP3);
Analyzer::disable_analyzer(Analyzer::ANALYZER_RADIUS);
Analyzer::disable_analyzer(Analyzer::ANALYZER_RDP);
Analyzer::disable_analyzer(Analyzer::ANALYZER_RFB);
Analyzer::disable_analyzer(Analyzer::ANALYZER_CONTENTS_NFS);
Analyzer::disable_analyzer(Analyzer::ANALYZER_CONTENTS_RPC);
Analyzer::disable_analyzer(Analyzer::ANALYZER_NFS);
Analyzer::disable_analyzer(Analyzer::ANALYZER_PORTMAPPER);
Analyzer::disable_analyzer(Analyzer::ANALYZER_SIP);
Analyzer::disable_analyzer(Analyzer::ANALYZER_CONTENTS_SMB);
Analyzer::disable_analyzer(Analyzer::ANALYZER_SMB);
Analyzer::disable_analyzer(Analyzer::ANALYZER_SMTP);
Analyzer::disable_analyzer(Analyzer::ANALYZER_SNMP);
Analyzer::disable_analyzer(Analyzer::ANALYZER_SOCKS);
Analyzer::disable_analyzer(Analyzer::ANALYZER_SSH);
Analyzer::disable_analyzer(Analyzer::ANALYZER_DTLS);
Analyzer::disable_analyzer(Analyzer::ANALYZER_SSL);
Analyzer::disable_analyzer(Analyzer::ANALYZER_STEPPINGSTONE);
Analyzer::disable_analyzer(Analyzer::ANALYZER_SYSLOG);
Analyzer::disable_analyzer(Analyzer::ANALYZER_CONTENTLINE);
Analyzer::disable_analyzer(Analyzer::ANALYZER_CONTENTS);
Analyzer::disable_analyzer(Analyzer::ANALYZER_TCP);
Analyzer::disable_analyzer(Analyzer::ANALYZER_TCPSTATS);
Analyzer::disable_analyzer(Analyzer::ANALYZER_TEREDO);
Analyzer::disable_analyzer(Analyzer::ANALYZER_UDP);
Analyzer::disable_analyzer(Analyzer::ANALYZER_XMPP);
Analyzer::disable_analyzer(Analyzer::ANALYZER_ZIP);
}

You can also try disabling checksums, thus saving some CPU cycles. The following line can also be placed in local.bro.

redef ignore_checksums = T;

The Bro Packet Loss logs found in the capture_loss.log file can be a good resource. Bro estimates packet loss based on what it expects to have happened during TCP sessions. If you’re NIC isn’t reporting packet loss but this log is, it’s likely packets are being dropped upstream from Bro.

Conclusion of Series

This concludes the four-part series to setup Bro on Red Hat Linux. Check out part one, two, or three to get started with your own instance of Bro.