To eval or not to eval
One of the biggest mistakes I have seen many organizations make is that they don’t evaluate the product they are buying. The organizations spent time creating a RFP, spent time reviewing the RFP responses, spent time talking to the vendors, even spent time doing due diligence on the vendors, but they don’t spend the time playing with the product before actually writing the check. They trusted the vendors’ white papers, the RFP responses and the marketing.
As I mentioned previously, the lack of clear understanding of the products is one of the five business mistakes you can make.
Without using a product, you will not understand whether it actually meet your requirements or not. If your requirement is root-cause analysis or forensics investigation, does the product you are choosing support that? Does it support drill downs for you to do investigation? Does it support ad-hoc reports for you to easily find the information you need? Does it keep the raw logs so you can actually see the data?
If your requirement is management reporting, does the product support PDF export? Does it support emailing? Does it have nice charts and graphs? (Believe me, management doesn’t care to see the raw data.) What kind of reports does it have? Does it let you create new custom reports and email those or just the canned reports?
If your requirement is to support 1000 logs per second, does the product support that type of performance? Does it have room for you to grow? Did you test it to see that it didn’t drop any logs? (Trust me on this one too, it happens! Since most logs are sent via syslog, they will be dropped silently!) What’s the peak and sustain performance of the software/appliance?
Does the product meet the security requirements of your corporate standards? Does it support your corporate authentication and authorization mechanisms? Does it support role-based authorization? Is the software installed on your server (or the appliance the vendor provides) locked down?
Vendors answer your RFP using the most general language possible and they always try to highlight certain features and hide others. They may interpret certain terms very broadly when you actually mean something very specific.
Be sure to understand your requirements and test the vendors of their claims. Never take the vendors’ word that their products meet your requirements. Put your hands on the product, use it, see how easy it is to perform your most important tasks.
There are generally two ways you can evaluate a product. Some vendors will install the software or appliance in your network so you can play with it. Some vendors will host a evaluation for you on their own network. Either way is fine, as long as you are in the driver’s seat and not the vendors’ SE.