Logs and telemetry

Pixwel logs

Telemetry

We have many sources of telemetry:

• AWS CloudWatch
• DataDog
• PaperTrail
• NewRelic

Details for the above can be found in Pixwel Platform Operations and alarms talks about what alarms are available that will let us know something is wrong automatically so we can respond as fast as possible.

CloudWatch

We use CloudWatch to record metrics on the application.

There are two types of metrics:

1. One off event. E.g. “someone logged in”. We can average these out to determine “14 people logged in in the last 5 minutes” and derive a “LoginRate” metric. This approach will be familiar to anyone used to the StatsD/Librato type of approach.
2. A metric. Think of this as a gauge or a meter that we can read. This is things like “number of open work requests”. See GatherKpis.php for the main bulk of the collection of metrics.

We have an AWS role cloudwatch-write that is added when an instance starts. This allows the EC2 instances to talk to CloudWatch without needing keypairs.

Example:

// log a counter called 'some-metric' with a prefix of KPI
MetricsLogger::log('KPI', 'some-metric', 42, 'Count');

How PaperTrail works

Logs are automatically aggregated to Papertrail.

For local development, you can see logs in:

• /var/log/nginx/*
• /var/log/syslog

For all other systems, the logs are pushed to PaperTrail.

See the base box job which installs PaperTrail.

• remote_syslog is a binary supplied to us by PaperTrail themselves
• We wrap remote_syslog in an init.d to make it able to act as a service.
  • We took this from the provided samples from PaperTrail
• rsyslog service needs to be running - remote_syslog seems to use it somehow. It is a log processing system that comes with Ubuntu. You CAN configure papertrail to use it directly but using remote_syslog is easer.

Special case for the “magic closet”

The “magic closet” runs on OSX, not Ubuntu, is not provisioned through immutable infrastructure so any changes around logging must be made manually - see slurpee and services for more details on this setup.

Ingest logs

See the ingest documentation for details.

---

Back to docs index | Next page in recommended reading order >>