Zen 2.0

These days, any large corporate infrastructure can generate tens of thousands of events/logs per second:
- A single PIX firewall in a moderately busy environment, with DEBUG level logging turned on, can generate one to two thousands logs per second.
- A single high traffic web server will handle hundreds of connections per second.
- A medium size corporation has several hundred, if not thousands, of desktop computers (mostly running Windows). With auditing turned on, these desktops can generate thousands of events per second, depending on the level of auditing.
- A Primary Domain Controller in a large corporate environment can easily general one to two thousand events per second.
- With the corporate network under attacks (scans and virii), IDS boxes are generating tens of events per second.

Why do we care?

Logs have intrinsic value. They tell us what’s happening in the operational environment. They tell us whether a device or application is having problems. They tell us how the device or application is performing, too busy or not busy enough. They tell us whether our marketing campaign was successful (jump in firewall/web traffic). They tell us if malicious users are trying to attack our infrastructure. They tell us whether there’s virii causing havoc on the corporate network. They tell us what produts/pages the users have visited.

In short, logs provide operational intelligence. There are 3 types of operational intelligence:

1. Security intelligence. By reviewing and analyzing logs, we can determine both external and internal threats.
2. IT operational intelligence. Are the web servers running at maximum capacity? How can we allocate hardware resources more efficiently?
3. Business intelligence. Which of our products attract more visitors? How many visitors make purchases the first time they visit the site?

Another way to look at operational intelligence is strategic vs. tactical:
- Strategic intelligence allows us to better target our customers;
- Tactical intelligence allows us to improve our operational efficiency.

At the end of the day, most organizations care about three things: increase revenue, reduce cost and mitigate risks. It’s not a stretch to say that log analysis can provide benefits for all three. Business intelligence through log analysis can help increase revenue; IT operational intelligence can help reduce cost by improving operational efficiency; and security intelligence can help mitigate risks.

Logs can provide us huge amount of information, the question is how we can extract these information out and provide value to the company. We will discuss these in more details later.