Zen 2.0

Anton Chuvakin has posted some comments regarding my “Searching for Root Cause” article.

Anton Chuvakin is a great guy. Very smart and definitely knows a lot about log analysis. I have the highest respect for him.

However, I think he misunderstood the article. In his comments, he said that “the article claims that you have to search logs in order to discover the real issue.”

This is definitely somewhat of an overstatement. My article does not claim that the only way to troubleshoot issues and determine root cause is through searching. Searching, however, is and will always be one of the ways admins use to troubleshoot issues. No amount of intelligence or reporting or whatever will replace drilling down into the details of the logs to determine root causes.

Many of tools today will help float the issues and problems to the top so admins will notice the problem faster. Then the admins will need to tools to drill down and find out what exactly are the cause of the problems. Search is one of those tools. Others may include further drill down on the reports.

Full-text indexed search is a much faster way to search. You can obviously insert all the logs into MySQL or some database and utilize the database to do the indexing. However, that can only carry you so far as the database insertion will be slowed down dramatically and can only handle a small number of messages per second.

The only real method to do it is utilize existing full-text indexing technologies to index log data. A great book on this topic is Managing Gigabytes.

Anton is correct in that the search technology can also be extended to determine and highlight the root cause. This is definitely true and possible to implement. I believe we will see tools, open source or commercial, with this type of features in the near future.

Love to hear more thoughts from everyone on this topic.

This is funny. Someone named “Fish” translated an article I wrote a while back on What to do before an IOS disaster strikes into Chinese.

Fish, assuming he did the translation, did a pretty good job of translating the whole article. Can’t say how legal it is since he most likely does not have permission from Computerworld to do so.

Regardless, I do appreciate the effort for the translation and spreading the word out.

In a previous column, I outlined the five steps in the problem management process: detection, identification, determination, resolution and reflection [article]. I explained how new technologies will be required to help IT administrators determine the root causes of IT problems.

But how do IT administrators determine them today?

I have written an article on how search technology can help in finding root cause.

To respond to an article I discussed in a previous post, I wrote this article on Steps for preserving the integrity of log data, which is published by Computerworld.

This article describes the importance of perserving unaltered log data for court admissibility, enabling trust and accelerating investigation and troubleshooting.

Scott Gordon, VP of Marketing for SenSage (I used to work there), has written an interesting article on how most of the SIM products are not ready for compliance prime time.

Most of the points are valid, however, Scott seems to have forgotten to mention that archival of unaltered raw logs is a crucial requirement for compliance.

Scott did mention that “companies simply gathered all raw event data and stored it” will not meet compliance. However, without the archival of raw logs, these companies also won’t meet compliance.

Compliance is a combination of alerting, reporting and archiving. All three processes have to be in place in order to meet compliance.

Log Article on Secure Convergence Journal.

Today’s enterprise networks are at risk — threatened by privacy breaches, information leakage, security attacks, policy violations and network downtime. Incidents are increasingly associated with hard dollar losses that go beyond the damage to a company’s reputation. About 95 percent of these financial losses are attributable to intentional or unintentional actions by insiders. Security issues — such as worms and viruses, internal or external fraud and policy violations — result in an average of 22 hours of downtime per year. Human error, system failures and natural disasters account for an additional 87 hours per year of downtime, the cost of which can be up to $6.5 million per hour. More disturbingly, the financial losses from IP theft are rising; already totaling an average of $1.3 million per company each year.

My article, IT Needs Help Finding Root Causes , has been published in the 8/15/05 print edition of Computerworld.

I am quoted in this article on the Cisco vulnerability disclosure incident.

My article, What to do before an IOS disaster strikes, has been published on Computerworld.

The following list of links are related to the Black Hat event that happened last week.

SecurityFocus: Cisco, ISS file suit against rogue researcher

North American Network Operators Group

BoingBoing: Michael Lynn’s controversial Cisco security presentation

Wired: Whistleblower Faces FBI Probe

Forbes: Cisco, Security Researcher Settle Dispute

ABC News: Cisco Seeks to Quiet Software Flaw Talk

Associated Press: AP
Cisco, Security Researcher Settle Dispute

Schneier on Security: Cisco Harasses Security Researcher

Tom’s Networking: Owning IOS at Black Hat 2005

News.com: Hackers rally behind Cisco flaw finder

BoingBoing: Mike Lynn presentation mirrors and legal fund

Good Morning Silicon Valley: Cisco patches security researcher vulnerability

BoingBoing: Security researcher quits job and blows whistle on Cisco’s fatal flaws

ZDNet: Cisco reaches deal with flaw researcher

WSJ Online: Cisco Tries to Squelch Claim About a Flaw In Its Internet Routers

TechTarget: Security researcher causes furor by releasing flaw in Cisco Systems IOS

Computerworld: Furor over Cisco IOS router exploit erupts at Black Hat

Wired: Cisco Security Hole a Whopper

My article, Insecurity through obscurity, has been posted on Computerworld.

Opinion: Software developers should heed the writings of a 19th century cryptographer, who can teach them a thing or two about designing security into their products, says columnist Jian Zhen.

Report: IT shops lax about logging

If a new report from the SANS Institute is any indication, enterprises are jeopardizing security by taking a sloppy approach to log keeping. As a result, the report recommends some companies abandon home-grown logging systems in favor of commercial tools or simply outsource the task.

My article on Computerworld: Know Your Options

The SANS Institute Analyst White Paper:

The Log Management Industry - An Untapped Market

The SANS Institute uncovers and defines the growing importance of proper log management, including real-world issues facing the IT community around best practices of log management, traditional approaches vs. new high-performance commercial solutions and the capturing of all log data messages including informational-levels.

SECURITY tools tapped for compliance

My article on Five tips for building log management infrastructures is now posted on Computerworld.