Open Source Log Analysis Tools

| Posted in General Techologies

Here’s a list of open source log analysis tools that I know of.

Please let me know if you know of others that are not on this list. Thanks.

December 31st, 2004 | Jian Zhen | No Comments

sisyphus toolkit

| Posted in General Techologies
                Welcome to the sisyphus toolkit!
                 Version 0.9beta (Nov 5, 2004)

This is a snapshot of some tools created by a project with the
following charter:
  With the specific goal of increasing supercomputer RAS (reliability,
  availability, and serviceability), we intend to produce a
  machine-learning analysis system which enables content-novice
  analysts to efficiently understand evolving trends, identify
  anomalies, and investigate cause-effect hypotheses in large
  multiple-source event log sets.

Currently it provides two independant tools (teirify and slctify)
which address the first two items above by automatically generating
regular expressions of messages in your logfiles, categorized by
increasing anomaly: common, deviant, and anomalous.  Common are those
types which occur at least k times (k is an input argument), deviant
are messages which appear fewer than k times but are similar in
content to common messages, and anomalous are messages which are
completely anomalous in content and occurence.  A simple GUI is
included for efficient review of results.  This provides an efficient
means to define "normal", and thus provides a basis to detect
"abnormal".  See pdfs in doc/ieee_cluster04 for more details.

Posted to the log analysis mailing list by Jon Stearley.
http://www.cs.sandia.gov/sisyphus/

December 31st, 2004 | Jian Zhen | No Comments

Amazon/Red Cross Donation

| Posted in General Techologies

Amazon has setup a great way for everyone to donate cash to the Indian ocean tsunami victims.

As of this moment, amazon users have donated over $6.6 million!!

Wikipedia provides up-to-date information on the event.

The death toll from the Indian Ocean Earthquake and subsequent tsunamis on December 26 has exceeded 130,000 people in 13 countries from Malaysia to Somalia.

December 30th, 2004 | Jian Zhen | No Comments

LMon

| Posted in General Techologies

Anders Nordby released his new tool, LMon.

LMon is a package for near real-time monitoring of logs, sending e-mail
alerts upon known (rule hits) or unknown data (rule misses).

Features:

- Buffer multiple rule hits within a given interval, cap at a given maximum
number of lines, wait for a given interval before sending next alert.

- Auto-discovery of log rotation.

- Simplicity. LMon can run from the command line without configuration, or
be controlled from a central configuration file with multiple instances
monitoring different log files/sending alerts to different people. It is very
much intended to be simple (Keep It Simple, Stupid).

December 29th, 2004 | Jian Zhen | No Comments

War on Intellectual Property Leakage

| Posted in General Techologies

Approximately sixty to eighty percent of your company’s asset is defined as Intellectual Properties, or IP.

IP includes everything from patents, trademarks, brands, trade secrets, designs, architectures, copyrights, algorithms, software code, hardware schematics, inventions, business processes, and many other intangible assets. These are properties that may or may not have no physical presence. They exist mostly in the digital world or people’s minds.

A study by PricewaterhouseCoopers, the U.S. Chamber of Commerce, and the American Society for Industrial Security International estimated that American companies lost up to $59 billion in intellectual property and proprietary information between July 2000 and June 2001. The largest average dollar value of loss per incident occurred in research and development ($404,000), followed by financial data ($356,000).

Probably not surprising to information security professionals, most of the IP leakage incidents involve insiders. Insiders are generally considered “trusted” users who have access to the internal network, whether they are connected on the internal LAN or through VPNs. The insiders can be current and former employees, contractors or business partners.

Any one of these employees, contractors or business partners could be dissatisfied for whatever reason and decide to send a few design specs to the competitors. Once the secret is out, it is extremely difficult to contain it. The cost of IP litigation, if you choose to go that route, can cost from several hundred thousand dollars to several million dollars. This amount doesn’t even include the cost due to loss of reputation, brand, speed to market and other factors.

So how does a company go about securing their intellectual properties and make sure access to the IPs are tracked?

Enterprise Content Management

The first class of companies who attacked this problem is the Enterprise Content Management (ECM) vendors such as FileNet, Documentum, Interwoven, Open Text, Stellent and Vignette. These vendors generally provide centralized document management capabilities that allow users to

  • Organize and classify electronic documents
  • Search documents using keywords
  • Share documents with other users
  • Check-in and check-out documents for edit
  • Version control for all documents
  • Audit all access to documents

The main solution to the IP leakage problem by these vendors is all access to electronic documents are recorded and reported. These products will help manage and track documents when it’s stored centrally on the server. They can track who has accessed which file at what time. How many times files are accessed and how often people access these files.

Some of the more sophisticated products can also tell you the access behavior by individual users. For example, if a user who doesn’t normally access a certain section of the repository all the sudden starts to download all the files in that section, something suspicious may be going on and should be alerted.

But what happens when the file has been downloaded to the user’s desktop? Once that happens, these products can no longer protect or track the documents. What happens if the user emails the file via Yahoo Mail or Gmail? What happens if the user uploads the file to another server using FTP or HTTP? What happens if the user copies it to an USB drive or prints it out?

IP Leakage Detection

A whole new class of companies, including Vericept, Vidius, and Vontu, has been founded to detect IP leakage on the network. These companies’ products are designed to detected IP leakage by monitoring all the exit points in which information can leave the corporate network.

In general, when users intentionally or unintentionally leak intellectual properties, they will probably

  • E-mail the documents as attachments
  • Upload the documents to another server via FTP or HTTP
  • IM another user

All unencrypted traffic on the network can be sniffed out by package sniffers and have the content be examined. This is essentially what some of the products are doing. Most of the products in this category are basically re-purposing technologies from the IDS and content filtering world. These products will captures the contents from either the network or email stream; examine the content by either performing a keyword or regular expression search; and alert the administrators if any matches occur.

The detection mechanisms in these products are not unlike signature-based IDS. They also suffer the same high false positive rate problems as the IDS products. You will also need to spend quite a bit of time tuning and maintaining the products in order for it to accurately detect IP leakage.

However, some vendors, such as Vericept, claims to have additional technology that performs statistical or linguistic analysis on the content and are able to detect leakage much more accurately and efficiently.

IP Leakage Control

One major problem that the network-based detection products cannot solve is sneakerware leakage. Sneakerware leakage includes scenarios where the user copies the file onto removable media such as CDs, USB drives and floppies, or the user prints the documents out. The user can then carry these removable medias or printouts with them and no one will notice.

Another class of companies, including Verdasys, Liquid Machine, Authentica, and AegisDRM, are attacking the IP leakage problem a different perspective. They have designed agents that run on users’ desktops and track all user actions including opening and printing files, copying files to removable media, and sending files across the network. These products allow users to define Acceptable Use Policies, monitors all actions performed, and prevent or alert when a violation occurs. This class of companies is generally categorized as Digital Rights Management vendors.

In general, however, these products cannot detect whether a document contains confidential information. Administrators or users must explicitly mark documents as either confidential and should be protected, or not confidential. Administrators can also set up policies to globally disallow copying to removable medias, or file sharing via P2P networks.

The Future

What’s in the future in fighting against IP leakage?

As storage and security solutions are merging, as evidenced by the Symantec and Veritas marriage, we can expect comprehensive solutions that will integrate all of the above components. We can expect products that

  • Have centralized enterprise contents management capabilities
  • Have components that can monitor network exit points and match the outbound content with the central repository
  • Have agents that can monitor user activities

These three components will talk to each other to more accurately detect and prevent intellectual property leakage.

We will also probably see many of the pure play vendors in these three areas (ECM, DRM, IP Leakage Detection) be bought up by some of the bigger vendors such as Symantec and EMC/Documentum.

December 28th, 2004 | Jian Zhen | No Comments

Network Intelligence Knowledge Base

| Posted in General Techologies

I was searching the web for information on Cisco IDS and found this link. Obviously it’s not available as it has been password protected by Network Intelligence. However, if you use the Google Cache, you are able to see the content at the time Google indexed it.
A bit more poking around, I found a bunch [...]

More...
December 27th, 2004 | Jian Zhen | No Comments

Banks and Hospitals

| Posted in General Techologies

This is for you stats buffs who are doing market research.
23% of US banks are planning major updates to infrastructure
About 23% of all US banks are planning major initiatives related to core banking systems, according to Gartner. US banks lag behind financial institutions in other parts of the world such as Europe, where [...]

More...
December 25th, 2004 | Jian Zhen | No Comments

Advanced Visualization

| Posted in General Techologies

Terry Kim had a short piece on Beyond the Pie Chart. I agree w/ him to some extent. Most vendors put visualization up for the wow factor and not necessary for anything useful.
He then asked the question Who is the leader in advanced visualization?
I am not sure I can answer that question but here’s [...]

More...
December 22nd, 2004 | Jian Zhen | 1 Comment

Happy Holidays!

| Posted in General Techologies
More...
December 21st, 2004 | Jian Zhen | No Comments

Cisco Buys Protego

| Posted in General Techologies

I had a question a while back on why hasn’t the buy out begun in the log management market, and here’s Cisco’s answer.
Cisco Systems, Inc., today announced a definitive agreement to acquire privately-held Protego Networks, Inc. of Sunnyvale, CA, ….. Under the terms of the agreement, Cisco will pay approximately $65 million in cash for [...]

More...
December 20th, 2004 | Jian Zhen | No Comments
  • Page 1 of 2
  • 1
  • 2
  • >