Is It Better To Leave Some Logs Behind?

Log management has emerged in the past few years as a must-do discipline in IT for complying with regulatory standards, and protecting the integrity of critical IT assets. However, with millions of logs being spit out on a daily basis by firewalls, routers, servers, workstations, applications and other sources across a network, enterprises are deluged with log data and there is no stemming the tide. In fact, the tide is just beginning to come in. With always-on high-speed internet connectivity and an increasing number of servers and devices that an IT department has to manage, the task of collecting, storing and making sense of all this data is no mean feat.

Adding to the confusion are non-specific regulatory requirements relating to logging and archiving that are entirely vague on what an IT department must do, coupled with the increasing pressure for data privacy. It is not surprising then that for many companies the default plan to keep the auditors happy is to simply collect and retain everything from every source. However, collecting and retaining every single log ever generated is often unnecessary from both a regulatory and forensic standpoint, and the retention of the data can often represent a security or liability risk itself.

This confusion in the log management space is further compounded by vocal proponents amongst the vendor community of the “collect everything” approach as necessary for being compliant and secure. My experience is that the world is not a black and white place but a myriad of grays. If you dig a little deeper you might find a reason for the extreme position. It turns out that some vendors really sell capacity for storing logs, others have license fees tied to log volume, yet others have no ability to enforce central configuration of filters across a large installation.

OK, putting aside cynicism, are they actually right? Is this one of those rare cases where the broad statement is simply the correct statement (“don’t smoke” immediately springs to mind)? Let’s explore this in some more detail.

Firstly, many systems generate large amounts of log data that is often completely worthless from a compliance standpoint or even for a generic forensic inquiry. For instance, with system auditing levels on high for Common Criteria certified Operating systems such as Windows or Solaris BSM or the new LuAS in the 2.6 Linux kernel, logs are generated at a highly granular level to record every kernel action. If applied indiscriminately to any or all machines and all users/files, then routine actions such as listing directory contents generate a flood of events.

While this is necessary for C-2 certification, it is rarely useful in the common situation. In cases where such audit levels are needed, they must be applied with care but beyond that, it is necessary to consider if these logs even need to be gathered and retained.

One such fairly modestly-sized utility company concerned with compliance and driven by uncertainty decided to err on the side of caution and collect everything. This resulted in the generation of a whopping 1.5 million events every 5 minutes that quickly overwhelmed the security team. Realizing that most of the data was nothing but worthless noise, they decided to adjust their logging policy and collect only relevant data. Just by cutting out the noise, they were able to scale back to a much more manageable 150,000 events every 5 minutes. A number of other users also have said enough already, and have elected to put in place a company policy on what gets generated, what gets collected and for how long log data gets stored. I like to think of this as maturation of the market.

To Continue Reading: Click Here
------------------------------------------
Source: Computer Technology Review
By: Steve Lafferty