In Part 1 of our log aggregation system round-up series, we discussed Papertrail, Logentries, Elastic Search, and Graylog2. While all of these services offered cheaper services, none of them presented with a complete system. In Part 2, we’ll focus on three services which offer advanced alerting. As any good DevOp knows, you need to know when to look at your logs, not just where to look. With CloudWatch Logs, Loggly, and Splunk, you get built in alerts to help you identify when patterns happen repeatedly so you can automatically notify your on-call staff, and make sure you respond to incidents quickly, minimizing the impact to your end users.
CloudWatch logs doesn’t necessarily belong in the log aggregation round-up since it’s really lacking any features that all of these other providers offer. CloudWatch Logs only allows you to send logs to your CloudWatch profile, and then alert when a search result returns more then a specific number of entries. You can graph that and set it up like any other CloudWatch metric, but it’s useless when it comes to actually diagnosing the issue and tracking down what went wrong. You can’t get to the raw logs, you can’t search, you can’t graph things after-the-fact.
The bonus is that CloudWatch Logs are incredibly cheap, and there’s no real reason to not also use them. It’s a great way to add additional verification on top of your existing checks.
- Cheap, pay-as-you-go model (Built into your AWS costs)
- Easy installation
- Easy integration with other CloudWatch alerts
- No Searching at all
- No way to dig into log files
Loggly offers one thing above most other providers, simplicity. Loggly is simple to use, simple to set up, and simple to get results from. Loggly’s new Generation 2 provides advanced features such as graphing, faceted searches, and built-in alerts. Under the hood, Loggly uses Elastic Search, so there are a few issues that you’ll need to be aware of.
First, Loggly can only handle certain types of log messages. You can append JSON to the end of your log messages in order to get in your custom fields, however you are limited to indexing only 100 unique fields. This isn’t per log, this is total, and unfortunately you can’t change those fields once the indexes are built. Since logs do time out after a given period of time, eventually those old logs will go away, but this can be very difficult for initial development.
Second, Loggly is backed by Elastic Search, so it is not very fast, and can not (currently) do real-time searches. Typically you will have a delay of a few minutes before a log line will show up in Loggly.
Overall, Loggly is very fast and easy to configure, and quick to get going. Their pricing model is designed around start-ups, low initial pricing and very customizable. If you want lower costs, you can choose less retention, or send less data. You can change anytime you want and pricing is a pay-as-you-go model (monthly), instead of requiring an annual subscription. Best of all, the entire sign-up process is online, and you don’t ever need to talk to a sales person if you don’t want to, but they’re certainly there if you need them. They also offer a free trial which is instant and painless to get started. There’s really no reason not to give them a shot.
- Low Price
- Flexible Pricing Model
- Easy to Set up
- Simple Alerting
- Advanced Dashboards
- Great support
- Limited indexing capabilities
- Some Reliability questions
- No add-on support
- Closed-platform, not extensible
Splunk has long been the defacto standard for Log Aggregation among enterprise clients. It has an advanced API, extendable dashboard, and even support for custom “apps”. Splunk has recently started trying to get into the Start-up market by offering their platform as a SaaS solution. Splunk Cloud is the result of that effort so far.
Splunk is by far the best log aggregation system I’ve ever tried, and it’s easy enough for non-technical people to use that it’s a no-brainer as the solution that should be your number 1 choice. It’s more then a Log Aggregation system, it’s a full-fledged reporting, searching, and analytics platform designed around parsing textual data. It has plugins for everything from reading AWS Logs (including billing), to performing custom REST requests and returning data from those requests. You can even make your own plugin using their custom language if you can’t find what you need.
Unfortunately, their business is severely behind the technology.
Splunk divides themselves into “geographic sales regions”, and they do not understand what it means to be a distributed company. When first approaching Splunk about purchasing Splunk Cloud, we were tossed around between several different salesmen because nobody knew who should get commission for the sale. As a consumer, I don’t care who gets commission, I just want to buy the product.
They also do not offer anything other then annual contracts. There is no simple online sign-up process, no pay-as-you-go model, no way to not be locked into using them at their current prices for another year. We were actually just about to buy into Splunk when Loggly Gen2 was announced. After a few weeks, Splunk announced a 33% price reduction which we would have not received if we had pre-purchased our yearly contract.
Splunk does not understand start-ups yet. They’re still focused on the “big sales” and don’t have a simple way to just use the platform. They have an AMI on the AWS Marketplace, but it’s a “Bring your own License” AMI, instead of building it into the price. If they had a simple sign-up, or an AMI that had a license included in it, I would drop any other logging platform in a heartbeat. It’s well worth the added costs, but not the added frustrations.
- Most advanced searching
- Post-processing of log lines
- Very advanced reporting
- Flexible Dashboards
- “Pivot” interface for non-technical users
- Advanced Alerting
- “Piping” of commands
- Add-on support
- 100% Uptime SLA
- Very expensive
- Does not offer pay-as-you-go model (Requires annual commitment)
- Very painful purchase/sales process
- Very complex setup process
- Requires third-party software installation on all servers
While we would prefer to use Splunk over any other service we’ve tested, we’re currently using Loggly. It has just enough for what we need, and offers the right combination of price and service.
There are many other logging platforms out there as well, and over the years I’ve had quite a few people sending me “hey check out this log system”, but most of them are all duplicates of one of these types of services. The biggest thing to keep in mind while looking at a log aggregation service is “what do I really need to answer”. Typically what you need is search, alerts, and graphs. Anything else is just a bonus.