Graylog Has Taken Over Our Centralized Application Logging

As all developers will know, centralized logging is the key to a happy life. Without it you spend your life trawling through servers to find the logs you need. Up until early 2015 we were achieving this with a SQL database to which all of our applications would directly write their logs.

There are many great solutions out there to support centralized logging such as Splunk, Logstash, SQL (love it or hate it, it's a stable system), etc but most of them are expensive. I am very happy that we happened to stumble across Graylog, which I have to say is by far the best centralized log collector I have used. We have not been using it in Production for over a year now.

Graylog sits on top of Elasticsearch so it scales very well. The front end to query your logs not only looks great but is blisteringly fast, with queries across millions of messages taking less than 100 milliseconds. The query syntax takes a little while to get used to but allows you to do some fairly complex querying on any number of user defined fields. It also allows you to setup log streams based on complex pattern matching rules which we use to categorize products and teams, and from these you can configure email alerts again based on your defined criteria. On any given busy sporting Saturday we are storing 20 million messages a day and hitting messages rates of up to 30k a minute which is very impressive for the hardware it's running on. We currently run 3 seperate instances to support our Dev, UAT and Production environments. As I write this our Production instance is holding 400 million messages which we are planning to scale up to 1 billion by the end of the year.

A couple of months ago we removed all of our SQL appenders and are now solely running with GELF (Graylog Extended Log Format) UDP appenders. The cost of a UDP publish is much lower than that of a SQL request so we have much more efficient logging, although we use Log4Net.Async to put all of our logging onto background threads to improve application efficiency. It's great to see such an impressive open source product being actively developed and made available for free.

If you are not using Graylog, you should be. Head over to their website and check it out.

comments powered by Disqus