Zen 2.0

Intelligent Enterprise’s Cindi Howson did a great review of several BI tools in the market. The review series took a close look at features and functions critical in the business intelligence (BI) product evaluation. Cindi used 6 different criteria to evaluate these products.

Most of the criteria Cindi used can translate directly to log analysis.

According to the Insider Abuse Study performed by U.S. Secret Service and Carnegie Mellon University Software Engineering Institute’s CERT® Coordination Center (CERT/CC),

In 74% of the cases, after detection, the insiders’ identities were obtained using system logs.

Microsoft Windows maintains atleast three event logs:

- Security Log – Tracks events such as logon, logoff, change to access rights, and system startup and shutdown.
- Application Log – Records events logged by applications, such as the failure of MS SQL to access a database.
- System Log – Records events logged by the operating system or its components, such as the failure of a service to start at bootup.

In Windows 2000 or later, there might also be other event logs, including

- Directory Service – Records events logged by Active Directory and its related services.
- DNS Server – Records DNS queries, responses, and other DNS activities.
- File Replication Service – Records file replication activities on the system.

Every Windows event must be identified as one of five event types:

- Information – An informational event that is generally related to a successful action.
- Success Audit – An event related to the successful execution of an action.
- Failure Audit – An event related to the failed execution of an action.
- Warning – A warning. Details for warnings are often useful in preventing future system problems.
- Error – An error, such as the failure of a service to start.

I posted a question to the loganalysis mailing list a while back asking what everyone uses for their collection solution. The summary I gathered is as follows:

1. Snare (or similar agents) sending to syslog…the kewl thing is that Intersect Alliance has provided a couple scripts that will allow you to install Snare onto remote machines in your domain…

2. DumpEvt…this is actually a pretty good way as you can dump logs from local and remote machines and format it the way you want it to…only thing is this is not real-time, so if you are looking for real-time, this is not the solution; however, in my case, I was not looking for real-time.

3. Win32::EventLog…works pretty good, since you can write your own script to do stuff, you can be flexible in what you want to keep or discard…not real-time…but you can easily write something that does what DumpEvt does with the added benefit of dumping the events to syslog…and still be agentless

4. Win32::OLE using WMI…pro’ly the most flexible solution, it can monitor for new log entries…so it can be a real-time solution..plus you have the flexibility of deciding what you want to do with the log once you receive it…send to syslog, discard, etc…however, i have read that this is pretty resource intensive…have not tested this approach to its limit tho…

I will post some sample code for the various collection methods in the future.

BAM, or Business Activity Monitoring, is an emerging technology (can you call it emerging if it’s already 2-3 years old?) defines the capability of monitoring and reporting, in real-time, all business events. For example, deploying a BAM solution can help a bank monitor, in real-time, the transactions that are going through the system and reporting on any anomalies.

BI, or Business Intelligence, on the other hand, mainly focuses on the historical analysis of data and reporting findings for a much longer period of time.

On the technology timeline, BI came way before BAM.

In the SIM world, however, real-time solutions came first. In fact, real-time event filtering and correlation was the major reason the SIM space is created. IT groups were overwhelmed by the amount of security events, including IDS alerts, that came in. The first SIM vendors created solutions to help reduce the alerts administrators have to look at.

Next came the historical analysis. However, due to the sheer volume of data generated by the infrastructure, SIM vendors were having a difficult time handling the retention of the data for a long period of time. Without the data being available, it’s difficult to perform historical analysis. New SIM players started appearing that provided solutions specifically to handle large volumes for long term retention.

Just like BAM having a tactical focus and BI having a strategic focus, the same can be said for the SIM space. Real-time SIM solutions are more focused on the tactical issues as these solutions tell IT what’s happening right now, and IT reacts based on the limited information in hand. Historical log analysis can have more of a strategic focus. IT groups can run complex analytical algorithms over a long period of time. There’s more time for IT groups to gather information, investigate the root cause, then take action.

As we can see, there are some obvious correlation between BI and SIM. In fact, so similar that some of the BI vendors such as SAS are getting into the log analysis business.

Can the SIM vendors learn some lessons from the BI world? My bet is that many of the BI functionalities and solutions can be translated into the SIM world. We will just have to wait and see.

Most of the logs generated in a corporate infrastructure are not reviewed. They are either archived and never looked at, or worse, never even retrieved and archived. Much of the logs generated by devices and applications evaporate into the ether and not missed.

There are many reasons why most of these logs and events are ignored.

First of all, the volume of logs generated by the infrastructure is ENORMOUS. Log volumes can easily go from megabytes to tens of gigabytes. Sifting through the logs and trying to analyze them is such a huge task that no sane human being would put herself through that torture.

Secondly, most of the time the administrators are not sure what to look for. Are they suppose to look for security problems? If so, what kind of security problems? What is considered to be a security breach? Are they suppose to use the data to gather operational intelligence? If so, what should they be looking for?

Thirdly, building a logging infrastructure is not a simple task. It requires a lot of planning. For example, what logs to capture, how to capture them, where to store them, how long to store them, all these questions have to be answered. With tight IT budgets and few IT administrators, many organizations are pushing the problem off until they have more resources (and budget).

Last but not least, some organizations don’t see the value in reviewing logs. They believe that they can get their information via other means. For example, they believe that security intelligence can come from their IDS boxes; they believe that IT intelligence can come from their ESM products; and they believe that business intelligence can come from their BI applications.

All of the reasons are valid and I am definitely not trying to debunk them. However, more and more organizations are seeing the true value of log analysis and are jumping on board. Better technology and lower TCO will also persuade some of the doubters to test the water.

These days, any large corporate infrastructure can generate tens of thousands of events/logs per second:
- A single PIX firewall in a moderately busy environment, with DEBUG level logging turned on, can generate one to two thousands logs per second.
- A single high traffic web server will handle hundreds of connections per second.
- A medium size corporation has several hundred, if not thousands, of desktop computers (mostly running Windows). With auditing turned on, these desktops can generate thousands of events per second, depending on the level of auditing.
- A Primary Domain Controller in a large corporate environment can easily general one to two thousand events per second.
- With the corporate network under attacks (scans and virii), IDS boxes are generating tens of events per second.

Why do we care?

Logs have intrinsic value. They tell us what’s happening in the operational environment. They tell us whether a device or application is having problems. They tell us how the device or application is performing, too busy or not busy enough. They tell us whether our marketing campaign was successful (jump in firewall/web traffic). They tell us if malicious users are trying to attack our infrastructure. They tell us whether there’s virii causing havoc on the corporate network. They tell us what produts/pages the users have visited.

In short, logs provide operational intelligence. There are 3 types of operational intelligence:

1. Security intelligence. By reviewing and analyzing logs, we can determine both external and internal threats.
2. IT operational intelligence. Are the web servers running at maximum capacity? How can we allocate hardware resources more efficiently?
3. Business intelligence. Which of our products attract more visitors? How many visitors make purchases the first time they visit the site?

Another way to look at operational intelligence is strategic vs. tactical:
- Strategic intelligence allows us to better target our customers;
- Tactical intelligence allows us to improve our operational efficiency.

At the end of the day, most organizations care about three things: increase revenue, reduce cost and mitigate risks. It’s not a stretch to say that log analysis can provide benefits for all three. Business intelligence through log analysis can help increase revenue; IT operational intelligence can help reduce cost by improving operational efficiency; and security intelligence can help mitigate risks.

Logs can provide us huge amount of information, the question is how we can extract these information out and provide value to the company. We will discuss these in more details later.

Over the past year I have attempted to create a blog several times but have failed. The main problem being that there was a lack of focus on the topic. I wasn’t sure what I wanted to write about.

Recently, I have been doing some research on the log management/SIM/SEM space and have found very little in terms of people and users sharing their experience.

I have found

- articles and research reports (Forrester/Gartner) that talk about this space
- mailing lists (loganalysis@lists.shmoo.com) that discuss this topic
- various open source tools that perform log analysis on certain log types
- companies formed to attack this problem

However, I have not found any blogs that are focused on this topic. I figure this is my opportunity to start one and hopefully be able to keep it up to date.

A bit about my background.

I developed a log management/analysis platform for Cable & Wireless (acquired by Savvis Communications) a couple of years ago, and I am currently a Senior Product Manager with a SIM vendor.

So I do have some experience in this field. I would like to discuss all log related matters on this blog. I am hoping to be able to invite guests to post to this blog as well.