Hello Again everyone,
Today I’d like to talk about network logs packet analysis and how truly expensive it can be. First off, I want to make it clear that I am not in any way recommending that you don’t log packets. Logging packets is an essential part of network security and I would argue that it is pretty much impossibe to ensure that your network is secure without doing some form of packet logging (at least in the short term). This is really more of a warning form the (now) wiser person based on a bad experience that I’ve just recently had.
As you are probably all well aware I am a bit of a pfSense nut (I feel perfectly safe admitting it, this is a “safe zone”). As such my pfSense box has tons of packages installed everything from Suricatta to squid. My pfSense box has around a 150GB (or there about) spinning metal hard drive. Over the past year or so I’ve noticed that the hard drive usage has gone from a few single-digit percentage points to around 98% usage. Incredibly high amount of growth just in as little as a year.
At first, I thought maybe this had something to do with the Squid proxy. Since it caches everything. Realistically it could fill up even a few tarabyte drive given enough time and the proper (or more accurately, the improper) settings. A few months ago I cleared the cache, reduced the cache size and went on with it (yes, I know that I should have seen from the drive usage that this didn’t really fix the problem but it’s worth remembering that I had my fingers in my ears completely oblivious to the real problem).
Fast forward a few mouths and the same thing happens and then I start to really wonder what the issue is. I once again login to the router, clear out the squid cache and that does almost nothing. I reclaimed only a few gigs of disk space. Confused, I fire up the pfSense command prompt (which is really just a FreeBSD command line) and run “du -h /” to try to find out what is eating all of my storage. Low and behold the storage is being taken by Suricatta’s log files. Yes that’s right, the Suricatta logs where over 100 GB in size! At this point I realize that it’s been keeping the logs THE ENTIRE TIME Suricatta’s been installed. At this point, I realize that we have a problem. I rush to check the Suricatta log settings in the WebGUI. And find out that by default there is no limit to log sizes. It will happily fill your hard drive with logs from the Surricatta package. I quickly changed this to something far more reasonable, applied the settings, and waited a bit to see if the machine has started deleting the logs. For some reason, the logs where still there. Flustered, I run “rm -r /var/log/suricatta*” and press “enter”.
My hard drive usage dropped down to around 50%. Mission accomplished! (although I do remember the drive being less full then that before).
I figure “Whatever, the drive is fine now”. It should be fine. I checked the Suricatta settings just to make sure I didn’t just wipe out my settings. Good news: my settings are still there, bad news: Suricatta isn’t running on any of my interfaces. I figure alright, no big deal. I try to start Suricatta. It starts up but then blocks all my connections to the Internet. Even switching pages in the WebGUI takes a long time (Although I did notice that the RAM usage shot up to +90%). Somethings up again. My only option at this point was to reboot the router. I do this it starts up fine.
I log into the web interface and find out that the disk usage is now 4%, Suricatta is working fine, my Internet connection is back, the RAM usage is back to it’s normal 50%. And all is well. I don’t know why the RAM usage spiked before but after I restarted all was well.
The moral of the story: make sure Suricatta has limited log file sizes. Set it right after you install the package, before you forget, and it gets out of hand. The second moral of the story: sometimes a good old-fashioned restart does wonders.