Infrastructure As Code Explained

Unless your company is completely out of touch in 2022, you have some footprint in the Cloud. Data engineers do quite a fair bit of development for applications that run in the Cloud, and as a result, need to have the infrastructure in place for it. In a lot of places, this creation is done by the developers who need it, generally with some oversight from a platform team to make sure best practices are being followed.

While creating something like an S3 bucket is relatively straightforward, there will come a time when you’ll want to have that infrastructure setup automated (after all, who doesn’t get a kick out of putting something together that does all your work for you?), especially if you’re going to repeat creating the same things over and over again (transient EMR clusters for example). This is where infrastructure as code comes into play. By specifying all your configuration needs and going through the proper CI/CD practices, you’ll be able to use those configurations whenever you need. Pretty cool, huh?

What are the benefits of following IaC? Those include the following:

Convinced yet?

There are two different styles to developing IaC: declarative and imperative. In the declarative approach, your code defines the desired state of the final solution. The execution of getting to that state is handled by the tool itself, which makes everything idempotent.

In the imperative approach, you define the steps needed to get to that final solution. This gives you more control over the process from start to finish, but this doesn’t guarantee idempotence.

The two main styles of tools for accomplishing IaC are cloud-native and cloud-agnostic. As can probably be guessed by the name, the cloud-native brand are tools that the Cloud providers create in order for developers to accomplish IaC. For AWS, the main tool for this is CloudFormation, which follows the declarative approach. Your configuration is defined in either JSON or YAML, which CloudFormation then uses to build up your resources.

AWS also offers the Cloud Development Kit (CDK for short), which is open-source and allows you to create resources using common programming languages (such as Python or Java). This takes the imperative approach in doing so.

Cloud-agnostic tools are open-source and can support different types of Cloud provider. The most popular one is Terraform, where configurations are written in a language known as HCL, which is based on Go. Terraform uses the declarative approach to creating resources.

I have been using Terraform rather exclusively for IaC for the last 3–4 years or so and I must say that it works really well. For one, I’m always a proponent of open-source solutions, and the ability to support multiple Cloud providers can definitely come in handy (you never know when everything within your company might move from AWS to GCP, for example). The two main disadvantages to Terraform are that it maintains state (which is actually a good thing but becomes a mess to resolve when it gets screwed up) and that supporting new features from a Cloud provider might be a little slower than it would be in the Cloud-native tools, who get those features faster. At the end of the day, though, I still think it’s the clear winner on what to use.

Hopefully, I convinced you of the benefits of infrastructure as code. It takes a little time to get up to speed, but it’s just so useful in the long run. Once again, thanks to Mariusz for the inspiration of this post. I’d recommend checking out the blog I linked in the beginning for more background on IaC.

Related posts:

The Indian Real Estate sector, one of the most globally recognized industries in the world, is well complemented by the demand for office spaces as well as urban and semi-urban accommodations…