Zen 2.0

Going through one of my sleepless nights again. So I figure I post a question here and see if I get any response.

I’ve always wondered what is the best way to incrementally upload logs from files that gets updated all the time. For example, application A writes to a log file. It continues to write to that file until the log file gest rotated, either through some external mechanism or the application itself.

There are several options here, obviously.

Batch Retrieval

First, the simplest thing to do is wait until the log file is closed and rotated, then upload the file to a central log server. Or the central log server comes and collect the log file using SCP/SFTP/HTTP/HTTPS/FTP. The problem with this approach is that the file may only get rotated every hour, day or week. This is not a feasible solution if there’s real-time analysis requirements. For example, you wouldn’t want to find some malicious sudo commands were executed a week later.

Tail + Logger

The second approach is to tail the file and convert it to syslog using something like logger. This approach works somewhat. However, there are also several issues. One is that the tail command may exit for whatever reason. When that happens, you will stop sending logs. The obvious thing to do is to wrap tail in some script that will catch the exit and restart it. However, you may lose some logs during the process (probably unlikely unless lots of logs are being written.) In addition, converting to UDP syslog always has that slight chance of UDP packets being lost on a busy network.

Another problem with converting to syslog is that it won’t work with log files that have headers. For example, W3C formatted files have a header that tells the any log parser what fields are included in the file. Without that, it would be pretty difficult to parse the logs.

Be sure to use the -F option with tail in case files get rotated or modified by hand by some user.

Continuous Curl

The third approach I thought of is to use curl to upload the files to the central server periodically. However, I don’t want to upload the whole file every time, otherwise I will get a ton of duplicate data. So I wrote a small wrapper in perl, ccurl (continuous curl), to remember the last position uploaded, and upload only the new logs next time. Basically the script does the following:

1. When supplied a file, it will look for the last uploaded position. If never uploaded before, 0; otherwise the last uploaded position.
2. Run curl to upload the file starting at the last uploaded position. (my curl command uploads to a LogLogic appliance, but you can change it to upload to anything that accepts HTTP uploads.)
3. Update the position file with the latest uploaded position
4. Script exits

The idea is that someone will put this in a cron job and upload every few minutes.

However, there are two huge problems with this script as quoted in the script.

# if $size < $pos, that means the file has been rotated
# however, there are two problems here
# 1. $size could have increased so fast that the next time
# the file is looked at, that it has increased passed
# $pos. this means we will miss all the logs before pos
# 2. if the file is rotated, that means there's a possibility
# that we have lost some logs from the previous file,
# like from $pos to the end of the file

$size is the size of the actual log file, $pos is the file position of the last upload.

I am relying simply on the file name, which is totaly not fail proof. I added simple logic in there to detect when a file might have been rotated.

I think I will change the script a bit later to have it use inode numbers to detect whether the file I am looking at is the same as before. This should work if the file is being APPENDED to ONLY. If someone decides to open it for writing, then the inode number will change. And that would totally screw me up.

[Disclaimer: this script is by no means production quality. Use/test at your own risk.]

Tail + Curl

The last idea I have is to do a combination of #2 and #3. Basically I will write a script to wrap around tail -F, read the data for a while, upload the data to the central server using curl, and repeat.

This may turn out to be a better way than the first three. It gives me TCP and the wrapper can be maded to work with log files with headers.

Hum…stay tuned…I’ll upload my script here when I get around to it. Or someone else may have done it already and can point me to the right direction.

OpenSIMS provides a way for tying together the open source tools used for security management into a common infrastructure. It leads toward the development of new SIMS technologies that allow different networks using risk metrics to collaborate on attacker profiling and remediation. Learn More >>

Don’t know a whole lot about this, but if you do, love to hear from you.

LoGS is a rule-based log analysis engine which attempts to address some of the short-comings of other freely available tools.

LoGS can be obtained at:
http://www.hpc.unm.edu/~download/LoGS/

Oak seems like an intersting syslog monitoring tool. But swatch still seems to be more advanced.

Oak is a program that can be used to monitor syslogs from a collection of servers and notify operators when problem conditions arise. In addition to providing immediate notification of critical problems oak will also batch less critical problems into summary messages that can be sent less often and via any medium. For example you may wish to have oak page you on critical events while sending a summary of less important messages to your terminal once an hour. In addition you could send a daily email message summarizing all events.

OsAudit version 0.1 is available for download.

OsAudit is a complete system for log gathering,
monitoring and analysis. It has two different running
modes: server and client.

For more information, go to:

http://www.ossec.net/osaudit/
http://osaudit.sourceforge.net
http://sourceforge.net/projects/osaudit/

For comments, suggetions or questions:
daniel.cid @ (at) gmail.com

Automated and Secure Log File Analysis Service - SLAC.

An Intelligent Log File Analysis System to keep you informed about your Web Servers safety and your Checkpoint FW-1. No software required!

Interesting idea…I am just not sure that administrators will feel safe enough to send their corporate logs to an external service like this.

Obviously there’s precedence, Counterpane is a monitoring and log analysis service.

What do you think?

The following is what I posted to the loganalysis mailing list. The original question was regarding how to retrieve Web server logs (Apache for Windows) and Application specific logs (written in text format).

You can accomplish this in a couple of ways.

One, you can write a batch script on Windows box and use AT scheduler to upload them periodically to your unix server, using either ftp or curl to upload.

Two, you can setup a sshd server on your Windows box, using Cygwin or some stripped down version of Cygwin. E.g. http://www.certaintysolutions.com/tech-advice/ssh_on_nt.html.

Note that the solution on that link is pretty old, but follow the same instructions using the latest cygwin binaries can get you a ssh2 package.

Once sshd is setup, you can setup rsa key authentication and from your unix box, scp or sftp the files from the windows box.

Three, setup ftp on the WIndows box, then use curl/wget/ncftp on the unix box to grab files off the Windows box. Similarly, you cansetup a web server that has the log dir accessible. Then use curl/wget from the unix box to grab files via HTTP.

Four, share the log dir, then use Samba to mount the shared dir and copy files that way.

All of the options have security concerns, so be sure to think hard before picking a solution.

There are also concerns about log rotation and what not that you will need to consider as well.

I think one of the most frequently asked questions in log management is how to get the Apache logs to the log management server.

Here are a couple workarounds.

• https://lists.balabit.hu/pipermail/syslog-ng/2001-February/001208.html
• http://www.precision-guesswork.com/sage-guide/apache.html

The first option is probably what most people are looking for.

Other options include transferring of the Apache logs after it has been rotated. We will discuss this in more details later.

Eventlog to Syslog Utility from Purdue University.

The Eventlog to Syslog utility is a program that runs on Microsoft Windows NT, 2000, or 2003 server, monitoring eventlog messages. When a new message appears in the eventlog, it is read, formatted, and forwarded to a UNIX syslog server. Depending on the facility and priority of the message and the configuration of the syslog server, the message will be logged to a message file or displayed on the console.

GO BOILERMAKERS!!

Here’s a list of open source log analysis tools that I know of.

• Lire by LogReport.org
• Simple Event Correlator (SEC)
• Apache Chainsaw
• Sisyphus toolkit
• Open Source Security Information Management
• Logsurfer
• LMon
• Automated Incident Reporting
• Crusoe Correlated Intrusion Detection System
• Monitoring, Intrusion Detection, [and] Administration System
• SGUIL

Please let me know if you know of others that are not on this list. Thanks.

                Welcome to the sisyphus toolkit!
                 Version 0.9beta (Nov 5, 2004)

This is a snapshot of some tools created by a project with the
following charter:
  With the specific goal of increasing supercomputer RAS (reliability,
  availability, and serviceability), we intend to produce a
  machine-learning analysis system which enables content-novice
  analysts to efficiently understand evolving trends, identify
  anomalies, and investigate cause-effect hypotheses in large
  multiple-source event log sets.

Currently it provides two independant tools (teirify and slctify)
which address the first two items above by automatically generating
regular expressions of messages in your logfiles, categorized by
increasing anomaly: common, deviant, and anomalous.  Common are those
types which occur at least k times (k is an input argument), deviant
are messages which appear fewer than k times but are similar in
content to common messages, and anomalous are messages which are
completely anomalous in content and occurence.  A simple GUI is
included for efficient review of results.  This provides an efficient
means to define "normal", and thus provides a basis to detect
"abnormal".  See pdfs in doc/ieee_cluster04 for more details.

Posted to the log analysis mailing list by Jon Stearley.
http://www.cs.sandia.gov/sisyphus/

Anders Nordby released his new tool, LMon.

LMon is a package for near real-time monitoring of logs, sending e-mail
alerts upon known (rule hits) or unknown data (rule misses).

Features:

- Buffer multiple rule hits within a given interval, cap at a given maximum
number of lines, wait for a given interval before sending next alert.

- Auto-discovery of log rotation.

- Simplicity. LMon can run from the command line without configuration, or
be controlled from a central configuration file with multiple instances
monitoring different log files/sending alerts to different people. It is very
much intended to be simple (Keep It Simple, Stupid).

Rainer Gerhards announced the initial beta release of the rsyslog package, an alternate syslogd implementation.

Rsyslog has been forked from the sysklogd package. It currently shares
its base design but includes many important enhancements. Most
importantly it supports

- fully configurable output formats, including
* high precision timestamps with year
* access to each of the message parts as well as substrings thereof
(includes access to facility and priority)
* access to the raw message received
- direct logging to MySQL database servers
- compatibility to stock linux syslogd

Rsyslog is GPL’ed software.

InformIT has a detailed article on Sguil, Why Sguil Is the Best Option for Network Security Monitoring Data. According to the website,

Sguil (pronounced sgweel) is built by network security analysts for network security analysts. Sguil’s main component is an intuitive GUI that provides realtime events from snort/barnyard. It also includes other components which facilitate the practice of Network Security Monitoring and event driven analysis of IDS alerts.

A nice and detailed article by Mark Burnett on Microsoft’s LogParser. According to Microsoft:

Log Parser 2.0 is a powerful, versatile tool that you can use to extract information from files of almost any format by using Structured Query Language (SQL)-like queries.

More information on this tool can be found on Microsoft’s site.