This post began as a tutorial for deploying something real to AWS using Pulumi. As I was writing it I realised that it was getting quite long so apologies in advance! I do start with a quick introduction to my experience with the tools and ending with getting Pulumi set up.
I’m going to talk very briefly about my journey through exploring a few IaC (Infrastructure as Code) tools. Now, I’m by no means an expert in this space, but I have deployed non-trivial infrastructure, and for better or worse I am managing infrastructure for my team. I’ve tried various different methods, each with their own pro’s and con’s, lets get into it!
Terraform
Terraform was my first attempt, it is hugely popular and is pretty much where you get directed if you do a quick google search. I got some resources up and running relatively quickly and the documentation is quite good (which is really helpful when you aren’t sure what you’re doing!). The downside, at least for me, is that my non trivial application soon grew quite complex and managing all of the Terraform modules and dealing with the DSL (Domain Specific Language) i.e. HCL (Hashicorp Configuration Language) just felt really limiting. Now I do want to say that I’m sure someone else could have done a much better job if they had known HCL but I didn’t have the time to invest, so I barely brushed the surface of it’s capability.
I will note that it does seem odd to be trying to program in a config oriented thing. It’s quite awkward.
AWS CDK
Next I gave the AWS CDK a go, using my teams language of choice - Python. I was hoping that the use of a real programming language would ease some of my pains around module reuse and organisation, also my dislike of the weird programming syntax in HCL. I also had high hopes for the AWS CDK as it has first class support (It’s maintained by AWS) and there are some really nice high level constructs, which are basically application abstractions.
I got off to a good start, It was really nice to already know the language I was working in, it made thinking about the task at hand a little easier. I really do feel like infrastructure deployment lends itself to an OO style. I did eventually run into a few issues. The first was that the high level constructs are only really useful if you are doing something that is trivial and you don’t want to customise it. The second problem I ran in to while trying to mimic the application I already had working in Terraform was that (at the time) the CDK was waiting on some functionality to be exposed by the underlying descriptive language, CloudFormation. This was a no-go for me, I didn’t want to have to start hacking about and calling the aws-cli from Python just to make my infrastructure deploy with one command. I then went searching for another solution armed with more knowledge about what I really wanted.
CDKTF
During my desperate search to have something that was like the AWS CDK I stumbled across a small article about Terraform recently bringing to life their own CDK - CDKTF. I decided to give this a go, I was pretty excited because it output Terraform instead of CloudFormation and so I figured if I could do it in plain-old Terraform I could surely do it using their CDK. I also want to note that they use the same library created by AWS, jsii. This is the library that AWS uses to generate it’s CDK for multiple languages.
My time using the AWS CDK was not wasted as CDKTF was using the same nomenclature and style etc, it just happened to be generating Terraform instead of CloudFormation. I did end up mimicking exactly what I had in my original Terraform implementation (but in a nicer way) and I was really happy with it. I did have to do a few hacks to get access to the underlying Terraform complex types, but It wasn’t too bad, plus I could hide it behind functions now that I had a real programming language.
The real downside I found was that CDKTF is still so new that it just doesn’t have the user base to find help, nor does it have any nice abstractions, or a standard for how these should be implemented. It only had wrappers for the raw AWS resources, no equivalents of the plethora of Terraform modules that you see out in the wild. I did write my own constructs of course but I didn’t want to commit to anything like that seeing as the CDKTF might change in the near future, and all my effort would have been wasted. My other gripe with it was that the CLI tool just sucked - the deploy command would re-write the entire terminal, looking like it was flashing and if you wanted the Terraform plan output you had to synthesize first with cdktf synth and then change into the output directory and actually use terraform plan.
Pulumi
Through all of my searching and experimenting a colleague of mine mentioned they had spotted an article somewhere mentioning Pulumi as an IaC solution. So here I am, writing a post as I attempt to mimic some of the infrastructure I have up and running with CDKTF.
Getting started with Pulumi
First things first, you need to get Pulumi up and running, I’ll give super brief instructions because you can follow their much better documentation for your specific OS.
NOTE: this assumes you already have your aws cli tool up and running and you know a little about it. Pulumi also links to this stuff on their docs - so I wont repeat it here.
Install Pulumi
brew install pulumi
Start a new project
Initialise a project by creating a directory and using the pulumi new command to create a new project.
mkdir pulumi && cd pulumi
pulumi new aws-python
At this point, if you haven’t used Pulumi before, you’ll be prompted to login or sign up. By default Pulumi manages your state and deploys your services from their cloud environment. If you are uncomfortable with this then there are other options to manage your state - but that is well out of scope for this post!
You should go through the following form to get the rest of the details to set up your project and stack, once complete the output should be similar to this:
➜ pulumi git:(master) ✗ pulumi new aws-python
This command will walk you through creating a new Pulumi project.
Enter a value or leave blank to accept the (default), and press <ENTER>.
Press ^C at any time to quit.
project name: (pulum) litec
project description: (A minimal AWS Python Pulumi program)
Created project 'litec'
Please enter your desired stack name.
To create a stack in an organization, use the format <org-name>/<stack-name> (e.g. `acmecorp/dev`).
stack name: (dev)
Created stack 'dev'
aws:region: The AWS region to deploy into: (us-east-1) ap-southeast-2
Saved config
...
...
Your new project is ready to go! ✨
To perform an initial deployment, run 'pulumi up'
The next step will be to deploy to AWS. There are many ways to make sure Pulumi uses the correct AWS credentials (if you have multiple). The method I use is to set my AWS_PROFILE using direnv, which will source the .envrc file every time I enter the project directory.
echo 'export AWS_PROFILE=dev' > .envrc && direnv allow
Now that is out of the way lets bring up our first stack!
pulumi up
Finally you should see the output as below signalling that you have successfully created an AWS S3 bucket using Pulumi!
➜ pulumi git:(master) ✗ pulumi up
Previewing update (dev)
View Live: https://app.pulumi.com/XXX/litec/dev/previews/b2450488-2de5-4aad-b968-ed837e86eec5
Type Name Plan
+ pulumi:pulumi:Stack litec-dev create
+ └─ aws:s3:Bucket my-bucket create
Resources:
+ 2 to create
Do you want to perform this update? yes
Updating (dev)
View Live: https://app.pulumi.com/XXX/litec/dev/updates/1
Type Name Status
+ pulumi:pulumi:Stack litec-dev created
+ └─ aws:s3:Bucket my-bucket created
Outputs:
bucket_name: "my-bucket-3992a83"
Resources:
+ 2 created
Duration: 21s
This post was virtually a bad copy of Pulumi’s own tutorial, but it’s just to lay the groundwork for the next part of the series where we will create a VPC construct from scratch with the following features:
Public and private subnetsInternet gatewayElastic IPNAT gateway