Deploying CI/CD Components in AWS With NodeJs

Written by Topher ZiCornell

CI/CD Deployments: The Problem Set

In the rapidly changing tech world, we need many tools to adapt and stay ahead. We've previously talked about the power of using Continuous Integration and Continuous Delivery (CI/CD) techniques in a Software As A Service (SaaS) environment in CI/CD in a Serverless World, and a key ingredient in making CI/CD effective is scriptable deployments.

Let's say I'm a software engineer (which is true) who is writing a shiny new wizbang. I need to get that wizbang deployed out into our testing environment to make sure it works with everything else in our ReachForce ecosystem. I can truck over to our deployment console, click on the buttons, type in the fields, link up all the input and output pathways and end up with a joyous, fully deployed wizbang.

However it took me 30 minutes of manual clicking and typing, and I had umpteen different opportunities to mistype every key, address, and setting I entered. That won't work in a CI/CD environment and is dangerous to do in any environment. But there is a better way.

The Power of CloudFormation Templates: Scripting Deployments

CloudFormation is an Amazon Web Services (AWS) tool for deploying components. It's based on the concept of templates. You write a template for the component you want to deploy in JSON, then kick off that template to actually do the deployment. When deploying in a development environment you want the wizbang to talk to a development database, whereas in a staging or production environment you want it to talk to the staging or production database. So you use variables in your templates for things like this: things that might change from deployment to deployment.

The Pain of CloudFormation Templates: Death by 1000 Cuts

Well, yeah, that's cool and all, but now let's say I have 30 new wizbang-like components to deploy. All of them look very similar to each other. Since I wrote that template for the first wizbang, I can just reuse it for the 29 others, right? Unfortunately, there are several things in a CloudFormation template that can't be handled by a variable. Any time the structure of the template is different a whole new template is needed. That means in this scenario we would need to maintain up to 30 templates that were nearly identical. Ick!

Enter the Template Generator: The Magic of Code

A template generator is a program you run which would generate a CloudFormation template (CFT) for you based on the instructions you gave it. For example, you can tell it to build a template that looks exactly like that one over there except replaces the entire stream section with a queue section instead. This is a tool that allows us to maintain the deployment needs for those 30 wizbang-like components without having to maintain 30 separate but eerily similar templates.

There are CFT generators based in several languages. Scripting languages which support using a Domain-Specific Language (DSL) such as Ruby are particularly attractive because you can make the scripts read more like the templates they're representing.

NodeJs is a Native JSON Generator: Connecting the Dots

As it turns out NodeJS is perfectly suited to be a natural CloudFormation template generator. CloudFormation templates can be written in JSON. I will confess to you, but don't tell anybody: It took me a disturbingly long time to realize the JS part of NodeJS and JSON were related. It's one of those things that is sitting there in plain sight but you just don't see because you're so focused on figuring out some clever way to solve your problem. It's the "JS" part. JSON stands for JavaScript Object Notation, and NodeJS is server-side JavaScript. So, NodeJS effectively speaks native CloudFormation template JSON.

Now, with all of that, let me walk through a completely fictional, fabricated, and imagined solution to the deployment problem. Here's the scenario: we have two components, a wizbang, and a foobar. Both are Lambdas, one connects to an RDS DB while the other publishes messages to a Kinesis stream.

I'll work in NodeJS 6.10 since that's currently the highest version AWS allows for Lambdas and I don't like having to switch between versions just for deployment. Also, I use a plug-in here called deep-assign.

Here's a quick little function which will accept an Object and overlay it over our base definition for what we think a Lambda should be. All the standard, common stuff we would set up for a Lambda is in the base definition, leaving only the overrides for the overlay Object. We have token replacement with the map of tokens being passed in as the config argument.

 const createLambdaTemplate = (overlayTemplate = {}, config = {}) => {
let baseTemplate = {
Resources: {
LambdaFunction: {
Type: 'AWS::Lambda::Function',
Properties: {
runtime: 'nodejs6.10', // nodejs6.10 | nodejs4.3 | java8 | python3.6
Code: {
S3Bucket: 'the-great-reachforce-code-bucket',
S3Key: `${config.codePath}/${config.componentName}`
FunctionName: { 'Fn::Sub': `\${AWS::StackName}-${config.componentName}` },
Handler: config.handlerFunction,
MemorySize: 128, // MB, in increments of 64
Role: config.iamRole,
Timeout: 10, // max execution time, in seconds

return deepAssign({}, baseTemplate, overlayTemplate)

For the tokens, though, I tried to use a range of patterns to demonstrate a few of the details available here. Here are a few notes to point some things out:

  • Every single item in the baseTemplate can be replaced by something the overlayTemplate. The MemorySize can be overridden, the Timeout can be overridden. Everything.
  • Nothing in the baseTemplate will be overridden unless there is an overriding setting in the overlayTemplate. This includes nested settings. So, if there's a Code block in the overlayTemplate but it does not include an S3Bucket item, the S3Bucket from baseTemplate will still be used.
  • All the CloudFormation template (CFT) functions are still available. For example, the FunctionName can still be set using a CFT Sub. This is useful to pull CFT properties, such as the AWS::StackName reference.
  • All the NodeJS functionality you already know and love are also available. For example, backtick string templates work just fine, and I could have set the Type by doing ['AWS','Lambda','Function'].join('::') if I really wanted to be That Kind of Guy.

The output of this function can be written to an S3 bucket and deployed using the aws-sdk plugin. From that point on the deployment would be exactly the same as you would do for any CloudFormation template. We also use the aws-sdk plugin to derive template stacks and deploy the whole shebang from NodeJs tools. We could just as easily do it using REST calls (and actually did in an earlier version).

The Bow on Top

All in all, using NodeJs to generate CloudFormation templates is a powerful and extremely flexible tool to keep in any AWS CI/CD toolbox.

Data Quality Management AWS

Subscribe to the Blog