Using Lambda for a Server-less Stackdriver Alerting Webhook

Stackdriver now supports creating dashboards to understand how your AWS Lambda functions are being used.  These dashboards allow you to see how many times a given function is called, how often it fails, if it’s being throttled and how long the calls take.

Screen Shot 2015-04-21 at 1.47.34 PM

 

Note that the default read-only IAM role that we use doesn’t include support for listing Lambda functions but you can edit it to allow both the lambda:Get* and lambda:List* permissions.

In doing the work to support this, I wanted to have an example case that I could use to generate some metrics.  I realized that this would be the perfect way to allow people to take a Stackdriver alert notification, transform it and send it to another system.  Basically, it’s a webhook but it doesn’t require you to run any additional servers!

To do so, we’ll create a Lambda function which is executed as a result of messages on an SNS topic and then use that SNS topic as the delivery vehicle for our Stackdriver alert.

First, I need a Lambda function to handle the Stackdriver alert.  I go to Lambda in the AWS console and created a new function with the following code (replacing YOUR_API_KEY_HERE with my Stackdriver API key.

var http = require('https');
var STACKDRIVER_API_KEY = ‘YOUR_API_KEY_HERE’;

exports.handler = function(event, context) {
    console.log('From SNS:', event.Records[0].Sns.Message);
    var msg = JSON.parse(event.Records[0].Sns.Message);
    console.log('msg is:', msg);
    console.log('msg version is:', msg.version);
    if (msg.version !== 1) {
        context.fail('bad version');
        context.done();
        return;
    }

    var incident = msg.incident;
    FileTicket(msg.incident,
               function (status) { 
                   if (status == 'fail') {
                       context.fail();
                   } else {
                       context.succeed();
                   }
                   context.done();
               });
};

// files a ticket in my ticket system after massaging the data
// incident: hash of the incident
// completedCallback(status) : Callback with status message when the function completes.
function FileTicket(incident, completedCallback) {
    // don't create new tickets for incidents that aren't opened
    if (incident.state != 'open') {
        completedCallback('success');
        return;
    }
   
    // stackdriver event messages are pretty simple but you could
    // create whatever type of message body you want here.
    var eventMsg = {
        'message': 'New incident ' + incident.incident_id + ': ' + incident.summary
    };
    var eventBody = JSON.stringify(eventMsg);

    var options = {
        hostname: 'event-gateway.stackdriver.com',
        port: 443,
        path: '/v1/annotationevent',
        method: 'POST',
        headers: {
            'Content-Type': 'application/json',
            'Content-Length': eventBody.length,
            'x-stackdriver-apikey': STACKDRIVER_API_KEY
        }
    };

    var req = http.request(options, function(res) {
        res.setEncoding('utf8');
        res.on('data', function (body) {
            // sent successfully, yay!
            completedCallback('success');
        });
    });
    req.on('error', function(e) {
        // oh noes!
        console.log('problem with request: ' + e.message);
        completedCallback('error');
        return;
    });
    // write data to request body
    req.write(eventBody);
    req.end();
}

Basically, this will take the json from the Stackdriver alert, send the incident to FileTicket where you can do all of the formatting you need before sending it on to your ticketing system.  In this example,  I’m just generating a Stackdriver annotation event but you could do whatever transformation you need to send to any other system.

Next, we’ll need to create an SNS topic to be used for our Stackdriver alerts.  Create the topic and set it up to deliver to the Lambda function you just created.  Now, set it up as a Stackdriver notification method as described here.  If you test the connection before saving it and then go to the event log, you should see an event from your Lambda function.

Now you can associate this as a notification method for any of your policies.  Since we already generate events on alerts, this isn’t a particularly useful case.  But you could use the same approach to reformat the alert notification and send it to some other system of your choice.

Let us know how you’re using this and if you have ideas for future things that we could do!

Comments are closed.