EMC and Network Intelligence: What it Means.

In the last few months, Novell bought e-Security and IBM got GuardedNet through its acquisition of Micromuse. Cisco grabbed Protego about and year ago and rumor has it that Oracle is about to buy either NetForensics or Intellitactics. It’s likely that HP, McAfee, and BMC are looking at other leaders like LogLogics as well as network behavior specialists like Mazu and Q1.

Building The New EMC, One Acquisition At A Time

Interesting comments from Dennis Hoffman, vice president of information security at EMC

Network Intelligence plays in three areas of the security industry, he said. The first is log management, a space where the leader is another company, LogLogic, San Jose, Calif., Hoffman said.

The second is event management, or the real-time processing of data for security purposes. “ArcSight is the leader,” Hoffman said. “There are lots of others in this space, too. Names you’ve never heard of.”

The third is security information management, which includes the reporting and forensic analysis of where security problems occur. Network Intelligence is the leader here, Hoffman said.

On a side note, here’s an article about ArcSight.

Ray Lane buys dinner – Who buys ArcSight?

One of ArcSight’s board members told me the company is hitting close to the $75m revenue number, that’s getting close to the magic $80m to $100m level that could initiate an IPO–except that the IPO market is in the doldrums.

[Update: Announcement was made today on the NI acquisition by EMC. Interesting how the NI story is hidden inside a much bigger story. Does that indicate what's to come? That NI is going to just be a small piece of the EMC security story? The PR doesn't even mention the price. However, Reuters' piece mentions the $175m figure.]

• Eric Fitzgerald’s Windows Security Logging and Other Esoterica – Always a great blog to get all kinds of good info on Windows events. Eric’s a Program Manager for Windows Core Security. He’s pretty active on the loganalysis list as well and always gives out great tips.
• Windows & Active Directory Auditing
• Top 5 Security Settings to Audit
• Windows Security Logging and Other Esoterica : What is up with Audit Collection Services?
• Software Update Services White Paper – Lots of info here that talks about the different event IDs related to SUS
• Randy Franklin Smith’s Security Log Encyclopedia – The one and only resource you need if you want a good categorization of windows events
• Microsoft Events and Errors Message Center – Find detailed message explanations, recommended user actions, and links to additional support and resources.
• Automating Windows Patch Management: Part II – Some more event IDs here…
  

There are plenty of links out there. Can you add to this list?

I think the general consensus is that log signing ALONE is not going to be enough, and that signing just the filtered log is also not enough.

Very interesting read. Should definitely check it out.

I agree with Marcus… log signing [alone] is not going to make or break
a court case — it [alone] might almost be asking for trouble.

As I pointed out later in my earlier response, the big deal is to get
all possible logs, even if they don’t appear relevant to the particular
matter — so you can show the trace, other anomalies (or lack of other
anomalies).

A quick bit of news, LogLogic today open sourced (GPL) Project Lasso, a centralized Windows event collector. The original code base came from SNARE but now due to the different nature of the collection mechanism, there’s about 20-25% of the SNARE code left in it. Most of the common code are around message expansion. In fact, the Lasso messages will appear to the users exactly the same as SNARE. So if you already have a parser that can parse SNARE messages, you can parse Lasso messages as well.

Lasso is a LogLogic-sponsored and community-supported collector that can
- perform multi-threaded remote event collection of multiple Windows machines
- reliable transportation using TCP syslog (syslog-NG compatible)
- data buffering when network connection is down
- support for custom application event logs

We are trying to get this on sourceforge, but those guys are a bit slow in setting up new projects. So for now, you can download the binary and source from http://loglogic.com/logforge/.

I would love to hear your thoughts and comments. Don’t feel obligated to love it, you can bash it as well if we did something stupid. We are always looking to make it better.

It talks about a distributed aggregation and filtering method using Google’s Sawzall interpreted language. Very interesting paper, the concept of applying distributed computing resources to do work in parallel is not new. LogLogic have implemented this concept to achieve massive parallelism and performance on log analysis for quite sometime now.

The interesting part of the paper relates to its new language, Sawzall. It’s a new language designed specifically for simplicity and parallelism.

First I don’t understand why they couldn’t have created Sawzall as a library for one of the existing languages such as Perl or Python. After some discussion with a Googler, I am somewhat convinced that there might be good reason for a new language. The main reason being parallelism. Most of the languages aren’t designed to program and execute in parallel from the ground up.

However, I have to nitpick the performance example they gave in the paper. The benchmark test cases are all CPU-bound cases. However, earlier in the paper, the authors talked about the applications for this language being mostly IO-bound. It would seem to make sense if they gave some examples that are IO-bound and still be able to show the performance advantage of Sawzall.

Another question I have is how much Sawzall relies on GFS. I am assuming that the parallel execution of Sawzall depends on many of the GFS features, but I have no basis for that.

.

Will let you know how it reads.

A vulnerability has been reported in Cisco Security Monitoring, Analysis and Response System (CS-MARS), which can be exploited by malicious, local users to gain escalated privileges.

The vulnerability is caused due to CS-MARS appliances being shipped with a default password for an undocumented administrative root account that is intended for debugging purposes. This can be exploited by malicious users to gain root privileges using the undocumented “expert” command. The password for the account reportedly cannot be changed.

Successful exploitation requires logon to the administration command line interface with e.g. the “pnadmin” account.

The vulnerability has been reported in versions prior to 4.1.3.

Please upgrade to 4.1.3 asap.

As reported by Secunia and Cisco.

I’ve always wondered what is the best way to incrementally upload logs from files that gets updated all the time. For example, application A writes to a log file. It continues to write to that file until the log file gest rotated, either through some external mechanism or the application itself.

There are several options here, obviously.

Batch Retrieval

First, the simplest thing to do is wait until the log file is closed and rotated, then upload the file to a central log server. Or the central log server comes and collect the log file using SCP/SFTP/HTTP/HTTPS/FTP. The problem with this approach is that the file may only get rotated every hour, day or week. This is not a feasible solution if there’s real-time analysis requirements. For example, you wouldn’t want to find some malicious sudo commands were executed a week later.

Tail + Logger

The second approach is to tail the file and convert it to syslog using something like logger. This approach works somewhat. However, there are also several issues. One is that the tail command may exit for whatever reason. When that happens, you will stop sending logs. The obvious thing to do is to wrap tail in some script that will catch the exit and restart it. However, you may lose some logs during the process (probably unlikely unless lots of logs are being written.) In addition, converting to UDP syslog always has that slight chance of UDP packets being lost on a busy network.

Another problem with converting to syslog is that it won’t work with log files that have headers. For example, W3C formatted files have a header that tells the any log parser what fields are included in the file. Without that, it would be pretty difficult to parse the logs.

Be sure to use the -F option with tail in case files get rotated or modified by hand by some user.

Continuous Curl

The third approach I thought of is to use curl to upload the files to the central server periodically. However, I don’t want to upload the whole file every time, otherwise I will get a ton of duplicate data. So I wrote a small wrapper in perl, ccurl (continuous curl), to remember the last position uploaded, and upload only the new logs next time. Basically the script does the following:

1. When supplied a file, it will look for the last uploaded position. If never uploaded before, 0; otherwise the last uploaded position.
2. Run curl to upload the file starting at the last uploaded position. (my curl command uploads to a LogLogic appliance, but you can change it to upload to anything that accepts HTTP uploads.)
3. Update the position file with the latest uploaded position
4. Script exits

The idea is that someone will put this in a cron job and upload every few minutes.

However, there are two huge problems with this script as quoted in the script.

# if $size < $pos, that means the file has been rotated
# however, there are two problems here
# 1. $size could have increased so fast that the next time
# the file is looked at, that it has increased passed
# $pos. this means we will miss all the logs before pos
# 2. if the file is rotated, that means there's a possibility
# that we have lost some logs from the previous file,
# like from $pos to the end of the file

$size is the size of the actual log file, $pos is the file position of the last upload.

I am relying simply on the file name, which is totaly not fail proof. I added simple logic in there to detect when a file might have been rotated.

I think I will change the script a bit later to have it use inode numbers to detect whether the file I am looking at is the same as before. This should work if the file is being APPENDED to ONLY. If someone decides to open it for writing, then the inode number will change. And that would totally screw me up.

[Disclaimer: this script is by no means production quality. Use/test at your own risk.]

Tail + Curl

The last idea I have is to do a combination of #2 and #3. Basically I will write a script to wrap around tail -F, read the data for a while, upload the data to the central server using curl, and repeat.

This may turn out to be a better way than the first three. It gives me TCP and the wrapper can be maded to work with log files with headers.

Hum…stay tuned…I’ll upload my script here when I get around to it. Or someone else may have done it already and can point me to the right direction.

Anton Chuvakin started the thread by asking other than parsing the individual messages (that could potentially have thousands of different formats), what other methods can be used in analyzing logs?

Some suggestions out of this discussion are listed here.

Clustering

Anton listed this as an option using tools such as slct. Another effort that I am aware of that’s using this approach is Securimine for Snort (SFS) from Securimine.

Securimine is founded by Ophir Rachman, who also founded Entercept Security Technologies (later on acquired by McAfee).

Brute-force Parsing

This method basically tries to guess some of the data structures inside a log message, such as IP address, hostname, username, password, action, etc etc.

Being able to correctly guess what data is a message without first knowing the message format is a tough problem. It relies on the parser knowing the exact structure of some of the data.

However, it can still be used to assist in parsing unknown messages. You can also apply some simple logics to classify the messages. Such as, if you see keywords such as from or to and IP addresses, that may be a firewall message.

Obviously this is not a fool-proof way, but given the alternative (not doing anything with the message at all!), it is a viable solution.

(One may ask the question of, is it better to not do anything so the users won’t be misled? or is it better to attempt in guessing and possibly give the wrong information? what do you think?)

Bayes/Markov/Expert Systems/Neural Nets/Genetic Algorithms

Several of the statisitical type of analysis were mentioned here.

1. Expert system – a collection of empirical data and decision algorithms compiled by developers
2. Hidden Markov models – since they are used in natural language and speech processing they might be applicable to log entries (they are after all some type of “natural speech”).
3. Neural nets – Once built, the neural net would be trained by experienced teachers (log analysis gurus).
4. Genetic algorithms – The trick would be to 1. define the right requirements (for example, determine the least number of message types without discarding significant data) and 2. define the genetic codes for the solution organisms. Maybe GAs are a bit far fetched but I wouldn’t exclude them.
5. Bayes – Bayesian classifiers have been extremely popular and successful in spam filtering. The success of baysian in spam filtering is partly due to the simplicity of classifying emails into ham and spam. In the log world, it is much tougher to tell from good to bad. Also, lots of not-bad messages may also indicate something bad. So it is tough to say how one can apply this type of technology to log analysis.

Obviously I am no mathematician nor do I claim to understand the nitty-gritty details of statistical analysis, so I can’t comment much on the technical merit of these methods. But love to hear from anyone who have more knowledge.

Indexing

One of the newer methods of analyzing logs is indexing and providing Google like search capabilities for all logs. This is something LogLogic and Splunk are doing.

The basic idea is that instead of parsing the messages by understanding every single format, use the full-text indexing approaches to break the messages into tokens, then allow users to use boolean search expressions to search the logs.

This method is great when it comes to troubleshooting and forensic analysis. If complemented with the understanding of the log formats, it can be as powerful as other methods.

I wrote an article on Searching for Root Cause a while back on the benefit of using Google-like indexed search on logs.

Tokenizing

This is the way most log analyzers are using today. This method generally require writing regular expressions or similar methods to parse the individual pieces of information out of the log messages.

Rainer Gerhards has a great summary in his paper On the Nature of Syslog Data.

Various standards

About Windows Event Log

IBM’s Common Base Event XML format – This is a VERY complicated XML based format that tries to cover everything. I see two huge problem with this type of format. First, it hugely expands the storage requirement given that raw log storage is required. Second, it could make parsing that much slower given the size of a single log (multiple KBs instead of hundres of bytes). It’s been morphed into the OASIS standard WSDM Management Using Web
Services v1.0 (WSDM-MUWS) .

RFC 3881 – Security Audit and Access Accountability Message XML Data Definitions for Healthcare Applications

WELF

W3C

IDMEF – Intrusion Detection Message Exchange Format

IDIOM – Intrusion Detection Interaction and Operations Messages (Cisco message format)

在商业竞争异常激烈的今天，日志存储和数据处理方面的需求将不断增长，而且企业常常是将其作为基础设施来规划，日志管理已成为世界上IT管理的新趋势。因此，LogLogic公司率先提出“日志生命周期智能化管理”概念，对日志数据进行采集、汇聚、存储、归档、分析和报警、实时监控，这种有效的管理方法可以优化网络运行效果, 提高网络可用性和安全性；提供可靠的网络信息审计；简化为适应行业法规要求所需的工作，从而大大降低运营成本；还可以协助IT决策和风险管理。

He’s absolutely right. He’s also written previously an article on Don’t Ignore Lowly Log Analysis.

Anton Chuvakin is a great guy. Very smart and definitely knows a lot about log analysis. I have the highest respect for him.

However, I think he misunderstood the article. In his comments, he said that “the article claims that you have to search logs in order to discover the real issue.”

This is definitely somewhat of an overstatement. My article does not claim that the only way to troubleshoot issues and determine root cause is through searching. Searching, however, is and will always be one of the ways admins use to troubleshoot issues. No amount of intelligence or reporting or whatever will replace drilling down into the details of the logs to determine root causes.

Many of tools today will help float the issues and problems to the top so admins will notice the problem faster. Then the admins will need to tools to drill down and find out what exactly are the cause of the problems. Search is one of those tools. Others may include further drill down on the reports.

Full-text indexed search is a much faster way to search. You can obviously insert all the logs into MySQL or some database and utilize the database to do the indexing. However, that can only carry you so far as the database insertion will be slowed down dramatically and can only handle a small number of messages per second.

The only real method to do it is utilize existing full-text indexing technologies to index log data. A great book on this topic is Managing Gigabytes.

Anton is correct in that the search technology can also be extended to determine and highlight the root cause. This is definitely true and possible to implement. I believe we will see tools, open source or commercial, with this type of features in the near future.

Love to hear more thoughts from everyone on this topic.

But how do IT administrators determine them today?

I have written an article on how search technology can help in finding root cause.

This article describes the importance of perserving unaltered log data for court admissibility, enabling trust and accelerating investigation and troubleshooting.

Most of the points are valid, however, Scott seems to have forgotten to mention that archival of unaltered raw logs is a crucial requirement for compliance.

Scott did mention that “companies simply gathered all raw event data and stored it” will not meet compliance. However, without the archival of raw logs, these companies also won’t meet compliance.

Compliance is a combination of alerting, reporting and archiving. All three processes have to be in place in order to meet compliance.

The new web site looks awesome!! Hate to say this but it’s much better than the old one.

The blog is also a great resource for information on LogLogic as well as tips and hints on how to manage the enterprise log data.

Check’em out..