Sending Logs From Your Heroku App to Datadog

iRonin IT Team - Experts in software development
back-end development, heroku, rails, ruby, web

Are you using Datadog and Heroku? Would you like to send your Ruby on Rails application’s logs to Datadog for history purposes and easier processing? We will show you how to do it.

If you are not familiar with Datadog, it’s a very functional monitoring and analytics solution that provides a better overview of your applications.

Since it’s not possible to directly connect Heroku drains with Datadog logs (at least at the time of writing this article) we have to use a different solution. Up until recently it also wasn’t possible to customize the Datadog buildpack for Heroku, besides setting environment variables (which is useful but quite limited). Thanks to one of our Ronins pull request on the GitHub project, you now are able to provide an advanced configuration as part of your build process.

Datadog configuration

In the latest version of the buildpack you can customize the agent by writing configuration files in the datadog/conf.d directory in the root of your application.

Configuration files are simple yaml files and the buildpack will search for *.yaml files in the conf.d directory, then copy them to $HOME/.apt/etc/datadog-agent/conf.d/${filename}.d/conf.yaml so they can be picked up correctly by the agent upon start.

List of all available options for logs collection can be found here.

Pushing your application logs

To push your logs you need to:

1. Add the Datadog buildpack to your Heroku application:

heroku buildpacks:add --index 1 https://github.com/DataDog/heroku-buildpack-datadog.git

2. Provide the DD_API_KEY ENV variable with your application’s key:

heroku config:add DD_API_KEY=<your API key>

3. Enable the logging feature - it can be done by setting the DD_LOGS_ENABLED variable:

heroku config:add DD_LOGS_ENABLED=true

4. Provide a configuration for the agent to watch for your log files and push them to Datadog (configuration goes to to the /app/datadog/conf.d/logs.yaml file):

##Log section
logs:
  - type: file
    path: /app/log/*.log
    source: ruby
    sourcecategory: sourcecode

5. Ensure that either RAILS_LOG_TO_STDOUT variable is not set - or you are using ActiveSupport::Logger.broadcast to push your logs to STDOUT and file at the same time (the Datadog agent needs to watch log files in order to push them). 6. Specify tags for Datadog (optional but can help with managing your logs) by providing the DD_TAGS variable (i.e. env:production).

We also recommend to set the DD_LOG_LEVEL to critical to prevent swamping your output (i.e. when you connect to your container with the essential heroku run bash command) with messages from Datadog agent:

heroku config:add DD_LOG_LEVEL=critical

Bonus: logging to STDOUT and log file

If you would like to be able to check your Heroku logs the standard way (heroku logs --tail), as well as keep them in the Datadog, you need to push your messages to STDOUT and log file at the same time. This can be easily achieved with the ActiveSupport::Logger.broadcast method, which overrides original logging methods so that he messages can be pushed to the additional logger.

The code is as simple as this:

if ENV["RAILS_LOG_TO_STDOUT"].present?
  logger           = ActiveSupport::Logger.new(STDOUT)
  logger.formatter = Rails.application.config.log_formatter
  logger           = ActiveSupport::TaggedLogging.new(logger)
  Rails.logger.extend(ActiveSupport::Logger.broadcast(logger))
end

Saving time by building automation tricks instead of performing manual transfers always gives us a buzz. Does your DevOps configuration need tweaking? Let us analyze your systems and see where you can save time by implementing configuration tweaks. Trust the experts at iRonin to optimize your solutions and drop us a line!

Author's Bio
iRonin IT Team

Experts in software development

We are a 100% remote team of software development experts, providing web & mobile application development and DevOps services for international clients.

Similar articles
Comments

Bulletproof your development with remote team augmentation

Read how
This page is best viewed in portrait mode
Our websites and web services use cookies. We use cookies and collected data to enhance your experience, provide additional communication channels, improve marketing materials and enhance our offer. IRONIN SP. Z O.O. SP. K. is committed to protecting all the data that we collect or process in any way, especially data of personal nature. By accepting these terms you agree to our usage of cookies and processing your data, according to our Privacy Policy, and you declare that your browser settings reflect your preferences. Read more You have the right to revoke this agreement at any time, based on the terms of our Privacy Policy. You can change cookies settings in your browser. If you do not agree with us using cookies and processing your data, please change your cookies settings in your web browser and reject these terms. You can find more information about cookies, your data privacy This site uses cookies. By continuing to browse the site, you are agreeing to our use of cookies. data processing, and your rights in our Privacy Policy.