Zen 2.0

An eWeek article explains:

Beginning next week, companies that have public float, or publicly owned shares, exceeding $75 million and that have fiscal years ending on or after Nov. 15 must comply with internal control reporting and disclosure requirements per Section 404 of the Sarbanes-Oxley Act of 2002. Companies with less than $75 million in public float have until July 15 to comply with Section 404.

As my previous blog has mentioned, you need to make sure that

• records are logged in reasonable details, accurate and reflect the transactions
• assurance that transactions are being recorded
• assurance that prevention or timely detection of unauthorized acquisition, use of disposition of the assets that could have a material effect on the financial statements

Log management is a critical component in accomplishing this task.

Do you have an infrastructure and solution in place to review your logs?

p.s. Thanks to joatBlog for the article link.

Yankee Group had an estimate of the SIM/SEM market in 2003 and showed that the SEM market is $180 million this year and $270 million next year.

To put that in perspective, the US information security industry is $8.7 billion.

The Business Intelligence market is on a growth path that should result in a $7.8 billion market by 2005.

The worldwide software revenue increased 5.1% to $178 bln in 2003 and is projected to rise to $189 bln in 2004.

And according to itfacts.biz,

The Semiconductor Industry Association predicted semiconductor sales to rise 28.5% to a record $213.8 bln in 2004 but remain essentially flat in 2005, as the excess inventory problems many chip companies are now reporting make their impact felt. The forecast projects sales will grow by 6.3% to $227.2 bln in 2006 and by 14.2% to $259.4 bln in 2007.

Comparatively, the SEM/SIM market is peanuts!!

It seems like in most real-world cases, log analysis is triggered by some stimuli, e.g. an alert (IDS, SIM, human) or a log report (text or graphical format) showing something interesting. Most sysadmins are probably too busy to consciously go and review logs unless something happens.

It also seems like most of the time, the process of investigation is in reverse chronological (temporal) order, as in, you go backwards in order to find out what happened (when did the attack occur, where did it come from, how did it occur). Sometimes you will go forward in time to find out if other similar incidents has happened or if the attacker used the host for other attacks.

There’s also a spatial component to the process of correlation as well. We generally look at logs from multiple hosts/devices across the network in order to figure out the root cause.

Almost all of the SIM vendors are implementing their systems based on forward temporal correlation, this is probably useful in detecting some of the more well known scenarios. If the scenario is not known, most likely the attack will be missed.

For most investigations and root cause analysis, it seems like reverse spatial & temporal analysis is more useful. Also, it might be less processor and memory intensive.

I call this S-TRACE, Spatial & Temporal Reverse Analytics & Correlation Engine.

The question is can something like that built to perform the reverse correlation automatically (or if it’s useful at all). There probably has to be some rules to tell the engine what to look at next (in reverse).

I would love to hear your thoughts on this.

Computer World has an interesting article on Playing catch-up on analytic technology.

A while back on the loganalysis mailing list there was a long thread of discussion on the “most popular reports”.

Adrian Grigorof from EventID was nice enough to compile the list of reports from the discussion thread.

Any good log analysis software should be able to provide two different views: microscopic and telescopic.

Under a microscope, the user should be able to see all the nitty-gritty details of an event or incident. An event under a microscope should show details of the fields that makes up that event. For example, if you are looking at a network connection of a firewall event under the microscope, the view should give you the source host, destination host, source port, destination port, and any other information that came with the connection.

If you are looking at an incident under a microscope, the view should show you all the events that made up the incident. The events can come from different devices, such as firewall, IDS, routers, switches, or applications such as web/application servers, databases, or operating systems such as Windows or Linux. From that view, you can examine each event under a microscopic view as well.

Under a telescope, the user should be provided a high level view of the infrastructure. It may be that the highest level view is a world map of your infrastructure. From there, you can drill down to each site, then each machine, each application, each incident, each event.

Another type of telescopic view may be a graph, e.g. a line graph showing the connection count of a device over a day/week/month period. From this telescopic view, if one sees something abnormal, such as a spike in connection count, one can select that time period and drill down to find out what makes up the spike. For example, the following graph is a graph of connections of a device over a year.

Yet another type of telescopic view may be a attack pattern graph showing all the alerts you have received from the various IDS sensors. You can then select a specific attack to drill down to view all the events that made up the attack. The following example shows a list of hosts attacking a single one. The number shows the number of attacks and the color shows the standard deviation.

The ability to transition from a telescopic view to a microscopic view is extremely important to any log analysis software. Imagine being able to select a portion of the “Firewall Connection Graph” and drill down to the events or click on the “Attack Pattern Graph” and bring up the attacks from a specific host.

As you are evaluating various tools and products for your environment,

• Ask the vendor to see if they provide that capability
• Use to the software to see how easy it is to drill down
• Compare the products and tools to see if they give you the same results

Think Microscope == Details and Telescope == Trends, Graphs, Charts, Summaries.

Side Note: I borrowed the terms Microscope and Telescope from Guy Kawasaki’s new book. I started reading The Art of the Start couple of nights ago and found it to be one of the best practical books for entrepreneurs. You can find favorable reviews of the book almost everywhere. Just Google It.

Unfortunately I can’t remember where I found this quote, but it’s the best definition of Operational Intelligence I have seen:

Operational intelligence should be focused on patterns of activity, trends, and indications of future intentions.

If you know the original of the definition, please let me know.

Aside from the technical or operational mistakes mentioned in this article, there are also business mistakes that organizations can make in their implementation of the log analysis infrastructure/product.

Below are five common mistakes that are commonly seen in organizations.

1. Lack of clear understanding of the values

Return on Investment (ROI) is usually a metric organizations use to justify any capital investments. Most SIM vendors have one, and they usually use it to help the buyers in the organizations to justify the investment to the upper management. Most ROI are centered around four different areas: revenue, cost, asset and risk.

Unfortunately, most of the metrics around SIM products are “fuzzy” metrics, i.e., was the increase in revenue or reduction in cost really caused by the implementation of the SIM products? There might be many factors, but the implementation of the SIM product most likely helped. Was the risk really reduced due to the implementation? Maybe, maybe not.

Revenue ROI is probably the easiest to understand. How much more money did we make after the implementation?

Cost ROI can have many factors, including labor, productivity, COGS, depreciation, training cost, waste and many others.

Asset ROI can include inventory, DPO, DSO, property plant and equipment, and others.

Risk ROI can include lawsuits, regulations and fines, and other catastrophic events.

Calculating the exact value of a SIM product is almost impossible. This makes measuring and evaluating the success of a SIM solution very difficult.

When calculating the ROI, be sure to have a clear understanding of the TCO (Total Cost of Ownership, and how many TLAs can we throw out?). Some solutions will require professional services while others don’t. Some solutions will require you to get the hardware while others come as an appliance. Without a concrete TCO, the ROI figures are meaningless.

Understanding the various ROI metrics, knowing that these are fuzzy metrics, and clearly defining what the organization expects from a successful implementation will help shape the clear understanding of the values.

2. Lack of clear understanding of the users

Who are the main users of the solution you are implementing? Are they business users or technical users? Are they power users or casual users?

Business users are usually users from marketing, finance, and other non-technical groups in the organization. They are usually managers, directors and upper management. These users want to see summaries, dash boards, charts, graphs, and other reports that gives them a high level view and the ability to spot trends.

Technical users are usually users from IT, security and other operations groups. These are the users who want to see all the nitty-gritty details. They want to drill down and diagnose/troubleshoot problems. They perform the root cause analysis when the **** hits the fan. Some of the operational users will also want to see dash boards so that they can see things at a glance.

Power users are the people who wants to create their own reports. They are used to sophisticated interfaces and want the flexbility over (over-)simplicity. There users generally use the system 3 or 4 times a week or even every day and they can be anywhere in the organization, including marketing and finance.

Casual users generally use the system once a week or every other week. They may not even log into the system and prefer to receive the reports via email. These users wants to run pre-built reports and they have a specific set of reports they want to see. They don’t have the knowledge or time to build their own reports.

Knowing who your users are directly affect the solution you choose. If your users are mostly business users, you may want to choose a solution that’s easy to use and provide the necessary views for these users. If most of your users are technical users, you will want a solution that provides the users the flexibilities that they need. Giving a technical users’ interface to the business users will intimidate them and turn them off.

3. Lack of clear understanding of the requirements

Simply stating that “we need to analyze our firewall and web server logs” is not a requirement. Having a SIM vendor come in to pitch, then draft the requirements based on the conversations is also the wrong way to go. These are just lazy ways of trying to get out of identifying the real requirements. There’s no other way to detail out the requirements other than talking to the potential users.

Are your requirements based on regulations? security attacks? operational problems? If so, which regulation? What type of security attacks? internal or external? DoS attacks? What are the operational problems? Performance or capacity optimization? Trend analysis?

Are your requirements centered around log retention? real-time alerts? long term analysis? If so, how long is the retention requirement? How “real time” do the alerts need to be? How far back do you need to perform analysis?

Do you have requirements on which logs need to be kept? OS logs? Network logs? Application logs? What type of reports the users need to see? High level or detailed reports?

Does your organization have requirements on the platforms? Windows? Linux? Java? No Java? Integration with corporate (authentication/authorization/web/database/storage) infrastructure?

What’s your requirement on performance? scalability? extensibility? manageability? usability? security? How much logs are you receiving (per second, retention period)? What’s your network architecture (VERY distributed or centralized)? How many people are going to maintain this solution (1 admin with MANY OTHER responsibilities or you have spare resources)?

What are the criticality and/or priority of your requirements?

Before evaluating any solutions, all these questions should be answered in details. Knowing your requirements will help you find the best product to fit your needs. Try not to choose a product or create unreasonable requirements because of brand loyalty (we only work with vendor X) or FOE (Friends of Executives).

4. Lack of clear understanding of the options

With all the talks these days about security information or event management and log analysis, it’s easy for organizations to jump on the bandwagon and decide that they must have a SIM product in order to meet the requirements.

There are at least 3 options out there: commercial, open source and home-grown solutions.

Before you jump to a commercial solution, you should evaluate your resources and skills within your own organization and see if you can/should go open source.

There are many factors when considering the commercial, open source or home-grown options. They include

• Resource. Do you have the head count that can support the option you choose? Commercial solutions may relieve your engineers from having to write and maintain code. When there’s a problem, you can throw it over to the vendor and demand a resolution. Open source solutions is a middle ground between commercial and home-grown solutions. But when there’s a problem, you have to either wait for a patch from the maintainers of the software or roll your own patch. Home-grown solutions will probably require the most from your engineers. You will have to write and maintain and enhance your solution.
• Skill set. Do your engineers have the necessary skills to maintain an open source solution or develop a home-grown solution? This really goes back to your requirements. If your requirement is mainly retention and not analysis, you may be able to do that in house or get a open source solution. But if you are looking for deep correlation analysis, sophisticated graphing and charting, and pre-built log parsers and reports, you may want to check out a commercial solution.
• Time. Even if you have the resources and skills within your organization, can you spare the time? Building a solution that meets your requirements is not a small undertaking. It will require dedicated resources and A LOT of time, time your organization probably can’t spare. Can you spend hours or days in diagnosing your home-grown solution?
• Budget. What is your budget for implementing this solution? Commercial solutions are not cheap. They can range anywhere from $25k to $750k, depending on your requirements. Your budget may vary based on the executive sponsor, the scale (departmental or corporate-wide), the urgency (SOX compliant by 2005), the risk (your organization lost 200k credit card #’s to a hacker last month), and other factors. If you have a small or no budget, you may need to go the open source or home-grown route (but know that these routes will have high costs as well).

Knowing your requirements will allow you to find the right option to meet your requirements.

5. Lack of clear understanding of the products

Many of the log analysis solutions out there are strong in some areas and weak in others. For example, some of the solutions provide better real-time analysis, whereas others are better at historical analysis. Some solutions are better at integrating the MANY log sources (network, system, application) whereas others provide better reports. Some solutions come in an appliance form factor, others are packaged software. Some solutions are targeted towards business users whereas others are targeted towards power users. Some solutions have much better visualizations while others have none.

Many vendors will tell you that they are strong in all areas. That’s a LIE! If they tell you that, run away as fast as you can (or kick them out as fast as you can).

As the vendors about your requirements. Ask them to rate how well their solution meets your requirement. Ask to see a demo. Ask for an evaluation (on site or hosted by the vendor). Ask them how their solution compare to their competitors. Ask them to share their product roadmap (when you are seriously considering the product.) Ask about the support structure.

Do not ever buy a solution without actually using it. There’s no way to tell how usable the product is or whether it meets your requirements by looking at power point slides.

Try to be open minded. Most of the solutions you will see are still fairly new to the market. They will have holes and problems. They will not meet all your requirements 100%. This is why it is necessary to prioritize your requirements. It maybe that you need both real-time and historical analysis, but one of those may not be as critical as the other. In that case, if a vendor doesn’t meet the less critical requirement but has a strong roadmap, it might be fine.

—–

These are common mistakes organizations make when they are considering a log analysis solution. Spending some time up front will help you avoid these and allow you to implement a solution that meets your real requirements.

Many log analysis vendors have spent a lot of time trying to make their graphs and reports look flashy and colorful, does that really help you in understanding your logs better? Sure, they demo well. But some vendors are so obsessed about 3D graphs and other flashy aspects of the GUI that they miss the real needs of the users.

Here are a few interesting articles that log analysis vendors should read in order to learn a few things from the BI vendors:

• The Information Cannot Speak for Itself
• Common Mistakes in Data Presentation
• Data Presentation: Tapping the Power of Visual Perception
• Eenie, Meenie, Minie, Moe: Selecting the Right Graph for Your Message
• Elegance Through Simplicity

This series of data presentation articles from Intelligent Enterprise gives the readers some great tips. Definitely worth reading.

RDEP, or Remote Data Exchange Protocol, is a proprietary application-level communications protocol created by Cisco for their Secure IDS version 4 product. (Version 3 of the Cisco Secure IDS uses the Postoffice protocol, which is not covered here.)

RDEP is mainly a request/response protocol utilizing the HTTP/1.1 protocol. RDEP can run over both encrypted (TLS/SSL) or unencrypted connections. The messages are exchanged in the Intrusion Detection Interaction and Operations Messages (IDIOM) format, an XML specification developed by Cisco.

There are three types of request/response messages defined by RDEP:

• Event Messages - IDS alarms (detected incidents), IDS server status or errors
• IP Log - IP log data (in pcap format) from the servers
• transaction Messages - Configuration and control of the IDS servers

Events

There are two possible ways of retrieving events, by query or by subscription. The client can query the server for a set of events with a specification, the server will then return all the events in a single IDIOM document. If the client wants to keep a live feed, it can subscribe to the server. With this connection, the client can retrieve events by sending subscription-get messages to the server. The first get will retrieve all current events that matches the get criteria, subsequent gets will retrieve events that are new since the last get.

IP Logs

The client has the ability to ask the server to create an IP log in the libpcap format. Once the IP log is available, the IP Log request message are used to retrieve the logs.

Transactions

Transactions are used to configure and control the IDS server. There’s not a lot of details in what commands can initiate, so if you have more information, I would love to hear from you.

Authentication

The RDEP protocol uses a modified HTTP BASIC method for authenticating to the server. The client sends the server an HTTP Authorization header like the web browser. Once authenticated, the server sends back a cookie which will be used for future requests.

Net::RDEP

Joe Minieri from Open Service has written a very useful Perl module, Net::RDEP. It can be used to retrieve events from the IDS server using either query or subscription. Thanks Joe.