Customizing CloudFormation Resources

Build Your Own CloudFormation Resources With AWS Lambda

Posted by Ryan S. Brown on Fri, Dec 4, 2015
In Mini-Project
Tags: cloudwatch, cloudformation, lambda

CloudFormation is a tool for specifying groups of resources in a declarative way. Each resource is actually a small block of JSON that CloudFormation uses to create a real version that is up to the specification provided. Now, AWS Lambda provides a way to implement your own resources with CloudFormation Lambda-backed custom resources.


Check out part one Deploying Lambda Functions with CloudFormation


Custom resources help a lot when CloudFormation doesn’t keep pace with the number of new services available in AWS. As problems go, this is a nice one to have, but it’s annoying if you like both immutable infrastructure and brand-new shiny services.

In this article, we’ll learn how to use custom resources to add support for more services. Keep in mind that you can make a Lambda custom resource do anything. You can create Github repositories, integrate with other service providers, or have a resource that kicks off a Jenkins build.

For this post, we’ll build on a CloudFormation template that sets up backups for your EBS volumes. The function contents were covered in a series earlier (check out EBS Backups Part 1 and EBS Backups Part 2). The template we’ll start with is from the earlier Lambda and CloudFormation post, and is available here.

Custom CloudFormation Resources

The custom resource is one of Andrew Templeton’s series of resources for API Gateway, CloudWatch logs, and DynamoDB streams. Take a second to thank him, @ayetempleton on Twitter.

Alright, now you’re back and we can try one out! Before we can set up the custom resources, we’ll make a stack that configures our EBS backup functions from the earlier series. The full template is available here.

Getting Started

The starting point for this post is this template from the post on deploying Lambda with CloudFormation. To it, we need to add the Lambda function that will clean up CloudWatch logs, then build a custom resource that calls that function to clean up our EBS snapshotter’s logs.

The Nested Stack

Like in the last post, we need an execution role to grant the function permission to do its job. The difference this time is that the log-janitor function only needs CloudWatch permissions.

"LogJanitorExecutionPolicy": {
    "DependsOn": [
        "LogJanitorExecutionRole"
    ],
    "Type": "AWS::IAM::Policy",
    "Properties": {
        "PolicyName": "LogJanitorExecutionPolicy",
        "Roles": [
            {"Ref": "LogJanitorExecutionRole"}
        ],
        "PolicyDocument": {
            "Version": "2012-10-17",
            "Statement": [{
                "Effect": "Allow",
                "Action": ["logs:*"],
                "Resource": ["arn:aws:logs:*:*:*"]
            }]
        }
    }
}

Once we have that role, we set up the Lambda function from andrew-templeton/cfn-cloudwatch-logs-janitor as an AWS::Lambda::Function resource.

"JanitorLambdaFunction": {
    "Type": "AWS::Lambda::Function",
    "DependsOn": [
        "LogJanitorExecutionPolicy"
    ],
    "Properties": {
        "Code": {
            "S3Bucket": "demos.serverlesscode.com",
            "S3Key": "2015-12-cloudformation-custom-resources/cfn-cloudwatch-logs-janitor.zip"
        },
        "Role": {
            "Fn::GetAtt": ["LogJanitorExecutionRole", "Arn"]
        },
        "Timeout": 60,
        "Handler": "index.handler",
        "Runtime": "nodejs",
    }
}

To use the Lambda function in a custom resource, we need to use the ARN as the ServiceToken (described in the custom resource docs). To get this to the parent stack, we’ll make that a stack output.

"Outputs": {
    "ServiceToken": {
        "Value": {"Fn::GetAtt": ["JanitorLambdaFunction", "Arn"]}
    }
}

You can get the whole template for the log janitor from here.

Referencing the Cleanup Function

In the parent template, we can add the log janitor stack as a resource.

"LogJanitorFunctionSubstack": {
    "Type": "AWS::CloudFormation::Stack",
    "Properties": {
        "TemplateURL": "https://s3.amazonaws.com/demos.serverlesscode.com/2015-12-cloudformation-custom-resources/cfn-log-cleaner-substack.template"
    }
}

Then we can reference the stack’s output (ServiceToken) in the LogJanitor resource we’ve configured. The resource acts like a wrapper, and calls the Lambda function from the child stack.

For each action taken by CloudFormation, an event is sent to the Lambda function for the request type. The request will be one of CREATE, UPDATE, or DELETE and will correspond with part of the resource lifecycle.

"SnapshotLogJanitor": {
    "Type": "Custom::LogJanitor",
    "DependsOn": [
        "LogJanitorFunctionSubstack",
        "EbsBackupSnapper"
    ],
    "Properties": {
        "ServiceToken": {"Fn::GetAtt": ["LogJanitorFunctionSubstack", "Outputs.ServiceToken"]},
        "RetentionInDays": "1",
        "LogGroupNamePrefix": {
            "Fn::Join": [
                "",
                [
                    "/aws/lambda/",
                    {"Ref": "EbsBackupSnapper"}
                ]
            ]
        }
    }
}

The Properties on the Custom:: resource are passed to the backing function specified by the ServiceToken. Our function takes the RetentionInDays and LogGroupNamePrefix parameters and, when deleted, sets up expiry so you don’t end up with CloudWatch logs for functions that don’t exist anymore.

Launching the Stack

now it’s time to launch what we’ve built. Just click Launch stack SelfCleaningStack to spin up a copy. You can view the full template here.

Click through the launch dialogs and then head to the AWS Lambda management console. Within a few minutes, a function named like “SelfCleaningStack-EbsBackupSnapper” will show up. Do a few test runs to generate CloudWatch logs.

Go to the CloudWatch management console and find the log group. Note that it’s set to never expire.

Cleaning Up

Now delete the stack. When the LogJanitor resource gets the DELETE signal, it will set the expiry of the CloudWatch log group to 1 day. This means you won’t pay for storage on logs for functions that no longer exist, which is always nice.

The NPM package has an installer that will deploy the resource automatically, but there are cases where it’s easier to deploy as a nested stack. For example, in a complex series of templates that may be used in multiple accounts (or as demos) it’s nice to have all resources and their cleanup self-contained and easily auditable.

Next Steps

As an extension to the logs-janitor function, you could add a CREATE action to have an expiration time set up for the life of the log group. This would help with cost control for long-lived stacks that contain Lambda functions, like deploying a Lambda-backed API.

Beyond enhancing existing AWS resources, you could make a custom resource that calls third-party services. Maybe your DNS is hosted with DNSimple but you need to create records for CloudFormation-defined instances. Their API is callable from Javascript, so making a wrapper resource would be trivial.

Thanks for reading! Keep up with future posts via RSS or subscribe to the mailing list.

As always, if you have an idea, question, comment, or want to say hi hit me on twitter @ryan_sb or email me at ryan@serverlesscode.com.


Tweet this, send to Hackernews, or post on Reddit