Creating a CloudFormation Custom Resource

CloudFormation is an extremely powerful service provided by AWS to simplify the provisioning of your resources and do so in a manner where you can deploy those same resources in a repeatable fashion. For example, if you need to deploy the same set of resources in multiple regions, instead of performing the same series of tasks in the AWS Console, spending the time creating CloudFormation templates can reduce errors and reduce the deployment time later.

However, as powerful as CloudFormation is, there are some things it cannot do well. For example, some resources need a VPC ID, while others need to know the CIDR Block for the VPC. We can determine the VPC ID by selecting it as one of the template parameters. However, if you need to know the CIDR block for that VPC, there is no method for CloudFormation to determine what it is.

Unless you use a Custom Resource.

A custom resource is a special or custom type that is backed by a Lambda function to perform a task which CloudFormation isn’t well suited to. For example, if we want to get the AMI ID to use for an EC2 instance, we would have to create mapping tables in our CloudFormation template listing the architecture type for each instance type we might deploy, and the AMI IDs for each architecture type. This doesn’t sound like fun, but when you add in that the AMI ID for the same image is different per region, creating that mapping table either makes your CloudFormation template impossible to use in an alternate region or creates a massive technical debt item to keep up to date.

Retrieving an AMI ID, or the CIDR block for a VPC are only two possible examples for a CloudFormation custom resource. Custom resources have a “request type” associated with the request, allowing the custom resource to create, update and delete whatever it is doing. Of course, if your custom resource is used to look up something, these request types are less meaningful, but they are there to help define more capabilities than what I am going to describe in this article.

A link to a sample custom resource for retrieving an AMI ID is included in the references section of this article.

Let’s assume for a moment we have already deployed our custom resource and we are now going to use it in our CloudFormation template.

VpcCidrBlock:
# Get the VPC CIDR Block
Type: Custom::cfnGetVpcCidr
Properties:
ServiceToken: !Join [ '', ['arn:aws:lambda:', !Ref "AWS::Region", ":", !Ref "AWS::AccountId", ':function:', 'cfnGetVpcCidr-', !Ref "AWS::Region" ]]
VpcId: !Ref VpcId
Region: !Ref AWS::Region

When CloudFormation executes this request, it uses the ServiceToken to find the function it is supposed to invoke and sends the parameters to that function. When our Lambda function receives the request, it includes the parameters and some other information to help CloudFormation know what to do later for an update or delete action.

Using the sample code above means we can deploy this function in multiple regions and accounts without having to modify the CloudFormation.

The request also includes a pre-signed URL which is used by the Lambda function to return the response to. All of the communication between the custom resource and CloudFormation is done through this pre-signed URL. If the custom resource doesn’t return the information to the pre-signed URL, the CloudFormation will not complete. If you didn’t specify a timeout for the stack creation, you could be waiting hours to be able to execute the stack again.

If you are building and testing a custom resource, set a short stack timeout so errors can be corrected and re-tested easily.

Before we look at our Python code to retrieve the CIDR block for our specified VPC, let’s look at the request and response components to our Lambda backed custom resource.

{
‘RequestType’: ‘Create’,
‘ServiceToken’: ‘arn:aws:lambda:us-west-2:AccountId:function:cfnGetVpcCidr-us-west-2’,
‘ResponseURL’: ‘https://cloudformation-custom-resource-response-uswest2.s3-us-west-2.amazonaws.com/pre-signed-URL’,
‘StackId’: ‘arn:aws:cloudformation:us-west-2:AccountId:stack/test20/2d213470-3bbd-11ea-a35f-06b8fd1f0384’,
‘RequestId’: ‘ba9e9fc5-4c6a-4b38-8459-3f9a3d838a4c’, ‘LogicalResourceId’: ‘VpcCidrBlock’, ‘ResourceType’: ‘Custom::cfnGetVpcCidr’, ‘ResourceProperties’:
{
‘ServiceToken’: ‘arn:aws:lambda:us-west-2:AccountId:function:cfnGetVpcCidr-us-west-2’,
‘VpcId’: ‘vpc-ID’, ‘Region’: ‘us-west-2’
}
}

In this sample request, there are several entries:

  • RequestType: This is one of create, update or delete. Used to help the function decide what to do. The only requests supported are those listed above.
  • ServiceToken: The service token tells CloudFormation where to send the requests to. The Service Token and required parameters are defined when creating the custom resource. This is always required in the template.
  • ResponseURL: This is the pre-signed URL the response must be sent back to.
  • StackId: The stack ID initiating the custom resource request.
  • RequesId: The Amazon assigned request ID.
  • LogicalResourceId: The name of the logical resource as it appears in the stack.
  • ResourceType: This is the name of the resource as defined in the stack.
  • ResourceProperties: The parameters which are passed to the function.
  • PhysicalId: If there is a physical resource associated with the request, or when the request is related to a previously submitted request, the physical ID provided back to the stack must be included.

The 1st ResourceProperties` component in the request provides the function inputs. In this case, we are providing the VPC ID, and the region, so we can look up the details for the VPC.

After our function executes, we return the following response to CloudFormation using the pre-signed URL provided in the Request data.

{
“StackId”: “arn:aws:cloudformation:us-west-2:AccountId:stack/test20/2d213470-3bbd-11ea-a35f-06b8fd1f0384",
“RequestId”: “ba9e9fc5-4c6a-4b38-8459-3f9a3d838a4c”,
“LogicalResourceId”: “VpcCidrBlock”,
“PhysicalResourceId”: “cfnGetVpcCidr-us-west-2-$LATEST”,
“Data”: {
“CidrBlock”: “172.31.16.0/21”,
“Reason”: “OK”
},
“Reason”: “More information in CloudWatch.“,
“Status”: “SUCCESS”
}

When responding to CloudFormation, the majority of the information we sent in the request is also passed back. If we are paying attention to the “create”, “update” and “delete” directives in the request, our custom resource can not only create a new resource but also update and delete it. However, for the custom resource to perform updates and deletes, it has to pass back a PhysicalResourceId as the identifier so future actions can be taken against this resource. In the example above, since there is no "physical" resource being provisioned, the PhysicalResourceId was set to the Lambda function.

Let’s now look at the Python 3.8 code to implement our sample resource. We’ll walk through the code in sections. Because I have used this code for several functions, the handler function sets everything up, calls the get_details function to retrieve the desired data, processes the results and sends the response back to the pre-signed URL.

I should point out there are some tools that can simplify the creation of the custom resource code, but sometimes the best way to understand how it works is to do it yourself. You get a better appreciation of what these other tools are doing to simplify your development cycle.

#
# create variable to hold the response
#
response_data = {}
print(f”INFO REQUEST RECEIVED”)
print(event)
for k,v in event.items():
print(f”key: {k} value: {v}“)
#
# retrieve data from the request
#
request_type = event.get(‘RequestType’, None)
resource_type = event.get(‘ResourceType’, None)
response_url = event.get(‘ResponseURL’, None)
request_properties = event.get(‘ResourceProperties’, None)
vpc_id = request_properties[‘VpcId’]
region = request_properties[‘Region’]

This code section from the handler function just sets things up. It prints everything in the event dictionary passed into the function, meaning it is saved into our CloudWatch logs. Even if you don’t do this, take advantage of logging information from your function into CloudWatch as it makes debugging in a production environment easier as you can see both the data received, and the data returned.

The request_type, resource_type and response_url values are passed into the function and used to determine what you should be doing (create, update, delete), the INSERT RESOURCE TYPE HERE, and finally, where the response should be returned to. The response_url is a pre-signed URL passed into the function on every invocation and is how the response is provided. If we don't send a response back to this URL, the stack will never complete.

The ResourceProperties from the event dictionary contains the arguments or parameters for the function. In this case, we have two: (a) the VPC ID we want the CIDR block for, and (b) the region the VPC is located in.

Almost all of the information provided to the function must be provided in the response.

#
# The values must be copied to the response verbatim
#
response_data[‘StackId’] = event.get(‘StackId’, None)
response_data[‘RequestId’] = event.get(‘RequestId’, None)
response_data[‘LogicalResourceId’] = event.get(‘LogicalResourceId’, “”)
response_data[‘PhysicalResourceId’] = event.get(‘PhysicalResourceId’, f”{context.function_name}-{context.function_version}“)

This is what happens next. We take all of the data we received in the event dictionary and put it in our response_data variable to be returned to CloudFormation once the function has completed.

With what we have now, we can get the CIDR for the VPC Using Boto3, the AWS SDK for Python.

try:
ec2 = boto3.resource(‘ec2’, region_name=Region)
except ClientError as e:
return {
“StatusCode” : 400,
“Message” : e
}
try:
vpc = ec2.Vpc(VpcId)
except ClientError as e:
return {
“StatusCode” : 400,
“Message” : e
}

print(f”REQUEST for VPC {VpcId}“)
cidr = vpc.cidr_block

We create a connection to AWS using the boto3.resource function. Assuming it doesn't fail, we try the ec2.vpc call, using the VPC ID provided from the CloudFormation stack to get the details about the VPC. There are a lot of details available, but the only one we are interested in is the CidrBlock.

Whether we got the CidrBlock or generated an error, a response is returned to the handler function where we evaluate the response and configure the remaining pieces in the response data.

if answer[‘StatusCode’] == 400:
print(f”FAILED: {answer[‘Message’]}“)
response_data[‘Status’] = “FAILED”
response_data[‘Reason’] = f”{answer[‘Message’]}”
else:
response_data[‘Status’] = “SUCCESS”
response_data[‘Data’][f”CidrBlock”] = answer[‘CidrBlock’]
response_data[‘Data’][f”Reason”] = answer[‘Message’]

If our response is bad, we set the status to FAILED and the error message to the Reason variable. By putting a value into the Reason variable, the user can see what happened in the CloudFormation stack. If our request succeeded, the status is set to SUCCESS, and the CidrBlock data added to the response. We can then access that value in our CloudFormation template.

The last thing we need to do is send the response back to CloudFormation.

json_response = json.dumps(response_data)
headers = {
‘content-type’: ‘’,
‘content-length’: str(len(json_response))
}
print(“SENDING REPONSE”)
print(json_response)
try:
response = requests.put(response_url,
data=json_response,
headers=headers)
print(f”CloudFormation returned status code: {response.reason}“)

Here we convert the response data to JSON, add the headers to the response object, and send it back using the Python requests library.

Here is the entire function.

import json
import requests
from collections import namedtuple
import random
import boto3
from botocore.exceptions import ClientError

def get_details(VpcId, Region):
try:
ec2 = boto3.resource(‘ec2’, region_name=Region)
except ClientError as e:
return {
“StatusCode” : 400,
“Message” : e
}
try:
vpc = ec2.Vpc(VpcId)
except ClientError as e:
return {
“StatusCode” : 400,
“Message” : e
}

print(f”REQUEST for VPC {VpcId}“)
cidr = vpc.cidr_block
if cidr is None:
return {
“StatusCode” : 400,
“Message” : “No CIDR returned”
}

return {
“StatusCode” : 200,
“VpcId” : VpcId,
“Region” : Region,
“CidrBlock” : cidr,
“Message” : “OK”
}

def handler(event, context):
“”"
function: handler
description: this function is called by lambda to perform the task
“”"
#
# create variable to hold the response
#
response_data = {}
print(f”INFO REQUEST RECEIVED”)
print(event)
for k,v in event.items():
print(f”key: {k} value: {v}“)
#
# retrieve data from the request
#
request_type = event.get(‘RequestType’, None)
resource_type = event.get(‘ResourceType’, None)
response_url = event.get(‘ResponseURL’, None)
request_properties = event.get(‘ResourceProperties’, None)
vpc_id = request_properties[‘VpcId’]
region = request_properties[‘Region’]

#
# The values must be copied to the response verbatim
#
response_data[‘StackId’] = event.get(‘StackId’, None)
response_data[‘RequestId’] = event.get(‘RequestId’, None)
response_data[‘LogicalResourceId’] = event.get(‘LogicalResourceId’, “”)
response_data[‘PhysicalResourceId’] = event.get(‘PhysicalResourceId’, f”{context.function_name}-{context.function_version}“)

#
# Add the data key
#
response_data[‘Data’] = {}
response_data[‘Reason’] = f”More information in CloudWatch.”

response = {}
answer = get_details( vpc_id, region)

if answer[‘StatusCode’] == 400:
print(f”FAILED: {answer[‘Message’]}“)
response_data[‘Status’] = “FAILED”
response_data[‘Reason’] = f”{answer[‘Message’]}”
else:
response_data[‘Status’] = “SUCCESS”
response_data[‘Data’][f”CidrBlock”] = answer[‘CidrBlock’]
response_data[‘Data’][f”Reason”] = answer[‘Message’]

#
# save the response in S3 using the responseURL parameter
#
json_response = json.dumps(response_data)
headers = {
‘content-type’: ‘’,
‘content-length’: str(len(json_response))
}
print(“SENDING REPONSE”)
print(json_response)
try:
response = requests.put(response_url,
data=json_response,
headers=headers)
print(f”CloudFormation returned status code: {response.reason}“)
except Exception as e:
print(f”send(..) failed executing requests.put(..): {e}“)
raise

# return the response
#
return response_data

This sample doesn’t make use of the RequestType value, but in reality, it should do so to simplify requesting new data (which always has a RequestType of Create, and when deleting stack resources.

CloudFormation Custom Resources allow us to extend the capability of CloudFormation in many different ways, from retrieving information like our AMI ID or VPC CIDR Block to creating, updating and deleting other possibly more complex resources.

AWS CloudFormation Custom Resources

AWS CloudFormation Lambda Backed Custom Resources

AMI ID Lookup Custom Resource Examples

Custom Resource Helper Script

Chris is a highly-skilled Information Technology AWS Cloud, Training and Security Professional bringing cloud, security, training and process engineering leadership to simplify and deliver high-quality products. He is the co-author of more than seven books and author of more than 70 articles and book chapters in technical, management and information security publications. His extensive technology, information security, and training experience make him a key resource who can help companies through technical challenges.

This article is Copyright © 2020, Chris Hare.

Chris is the co-author of seven books and author of more than 70 articles and book chapters in technical, management, and information security publications.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store