Turbocharging your Logs

Logging can be a really powerful tool to help you to keep on top of problems in your applications. However all too often this information is lost amongst multiple log files spread out over many different servers. I’ve recently setup and configured a Graylog 2 server at work and thought I’d write about it and getting PHP applications to send both application and error messages to the server.

What is Graylog2?

Graylog2 is a logging server that can aggregate log messages from multiple sources. It’s an open source project and you can configure your servers and applications to send all log messages to Graylog for later analysis. Graylog stores them in an Elastic Search database and allows you to perform custom searches on them. You can also see in real time how many log messages are being received and processed. The real killer feature in Graylog is streams. A stream in Graylog is a predefined custom search. The server will display to you log messages that only match the stream criteria selected, graphing the results for you. You can also attach alarms to streams so that if the number of messages recorded for a stream exceeds a certain level over a period of time an email alert can be sent out.

Messages can be sent to Graylog via UDP or AMQP and there are libraries for many languages allowing you to send messages. UDP has the advantage of being quick in that the application does not need to wait for confirmation in the way that it would with TCP. However AMQP allows for log messages to be saved in the event that the Graylog server ever goes down. You can configure your servers to send messages to Graylog via Rsyslog so that even system logs can be aggregated in one place. Getting the server installed and running can be a little time consuming as it uses MongoDb and ElasticSearch for storage, Java for the Graylog server that accepts and stores incoming messages and Ruby on Rails for the web frontend. The best guide to getting all of this going in my opinion can be found at Getting Started with Graylog2 for Logging. I would add two ‘pro tips’ to this from my experience:

  1. If you’re using an Amazon EC2 instance to run your Graylog server don’t forget to open ports 514 and 12201 for UDP in the security configuration. Opening 514 allows your Graylog server to receive Rsyslog messages and 12201 is the default port that Graylog receives messages in Gelf (Graylog Extended Log Format) on. If you don’t make sure these ports are open you’ll spend hours like me wondering why your server isn’t receiving messages…
  2. Double check that the disk that ElasticSearch is saving data to has plenty of space. Graylog uses ElasticSearch to store log messages and depending on the volume of messages you’re handling this can mount up pretty quickly. You can configure how long messages are kept for (the default is 30 days) however.

Obviously, setting up and maintaining a Graylog server only makes sense if you have multiple servers to manage. Assuming that you do it can be a real godsend. There are libraries in many common languages for sending messages to a Graylog server and the Graylog project maintains a test server for you to try sending messages to (link on their website).

So, you now have a working Graylog server: how do you send messages to it from PHP?

Introducing Monolog

Monolog is a logging library for PHP 5.3 + is well worth checking out. It supports the 8 official logging levels defined in RFC 5424 and allows you to log messages to multiple formats. A Monolog logger has at least one handler attached to it, and handlers are used to process log messages. A handler takes care of recording log messages to different backends such as streams (files), emails or Graylog. When attaching a handler you also specify the log level that handler records, allowing you to specify multiple handlers, each recording a different log level. Handlers come with appropriate message formatters but this can be overridden, allowing you to send emails with JSON embedded data if you wish. Logging a message is as simple as calling a method with a string as the first argument and an optional associative array of extra ‘context’ data as a second argument. Monolog will take care of finding a handler that can handle the message and error level being logged.

Thanks to Matt Lehner Monolog has a handler for Graylog. To log messages to a Graylog server you’ll also need the Gelf-PHP package available from Packagist. Assuming a composer installation of Monolog and Gelf-PHP the following code will write test warning, error, info and debug log messages to your Graylog server on the default Gelf port of 12201:

Sending Logging Messages from Applications

So, how would you go about sending log messages from your applications to Graylog? Symfony2 uses Monolog by default for logging and it’s relatively simple to add the Graylog handler to your application config. This Gist gives information on how to do that. Once it’s all configured sending a message to Graylog is as simple as retrieving the logging service from the service container and adding a message.

It’s a little (but not much) harder with applications where a DIC is not available. We have an older application at work that uses Symfony 1.4 that I wanted to add logging for. To do that I ended up adding a class that returns a singleton (evil, I know) instance of Monolog, fully configured to send all messages to our Gelf server. The code for that looks something like this:

This code also attaches a processor to the Monolog instance that automatically collects all data from $_GET, $_POST, $_COOKIE and $_SERVER and adds it to the context of a log message, helping us to debug problems more easily.

Logging all PHP Errors to Graylog

Sending error messages from your application code is likely to only be half of the story though: what about errors from the PHP interpreter itself? We have some legacy code that I work with that produces a lot of errors and warnings and I wanted these recorded in Graylog too to give us a better chance of tracking them down. Luckily PHP allows you to register error handlers, exception handlers and shutdown functions. Using a combination of these I was able to capture all PHP errors and log them to Graylog. To do this I used the logging helper class from above and the following code:

This code is contained in a ‘logging.php’ file that is simply included in our index.php file for our application.

Don’t let the Gorillas put you off…

I really like Graylog2, despite the abundance of pink and Gorillas in the UI design. It’s a very useful application and is something that we’re going to be making increasing use of at work over the next year or two.

6 thoughts on “Turbocharging your Logs

  1. Great post Jeremy.

    I’m trying right now and works like a charm. Althoughyou have a typo in the line require_once ‘path/to/composer/autoload.php’);

    You should remove the parentheses.

    Regards.

  2. We use splunk storm at talenthouse. Have you ever used splunk before? It is a program that sounds pretty similar to the program you are proposing, except splunk handles the messages and storage for you, and also provides a nice search interface.
    And the cost of storage is also pretty low, you get 10 GB for free, as far as i know, and 50 GB log storage cost 50 $ monthly. Seems like a competitive alternative to me.

    1. Thanks Paul, that does sound interesting. On the flip side though the splunk service is hosted. For some people it might be more important for them to keep control of their log data themselves. The other issue, at least in the PHP world, is that AFAIK there is no Monolog handler for sending log messages to splunk. Do you know if there’s a PHP library to do this?

Leave a Reply