How to Organize Infrastructure as Code in your Source Code Repository

Moving to the cloud allows you to manage infrastructure in new and incredibly powerful ways. Unlike traditional, manually managed infrastructure, the cloud empowers Infrastructure as Code (IaC) in which entire infrastructures can be implemented and managed with automation. While IaC offers real benefits in ensuring environmental consistency, growing the pace of innovation, and increasing overall quality, to effectively manage your code, it is important to codify and version it, which is best done through a source code repository. Yet, that is often easier said than done with teams in agile DevOps environments needing to effectively navigate, retrieve and collaborate on code. At Flux7, we have implemented several DevOps best practices for organizing source code repositories which makes this collaboration easier, in turn saving time and potential rework.


If your organization is concerned about security, you may find our paper: Effectively Balancing DevOps and Security useful.   

A Quick Repo Review

Source code repositories are central file storage locations that support various services and tools, such as version control, release management, and more. A source code repository should be in place before you even start down the IaC path as it will enable your team to define IaC, store all your artifacts, and update configs while maintaining consistency. Moreover, it ensures that the entire team is on the same page with regard to the resources required to support an application.

Repo Organization for Infrastructure as Code

The DevOps team here at Flux7 tends to work with a variety of the most popular source code repository hosts like Github, Bitbucket, and AWS CodeCommit. However, the best practices we are about to share can be applied almost universally to any source code repository you choose. Note that while how you arrange your source code repository may be a matter of personal preference, the overarching goal should always be to ensure that anyone needing to access it can easily find what they need. With that in mind, here are the best practices we’ve established in our work with dozens of enterprises and hundreds of projects. Note that “X” used here is a stand-in for the project name. Also, note that this repo structure is applicable to both TerraForm and CloudFormation, the two most commonly used infrastructure as code languages today.

• X-modules – In this repository, we store all Terraform or AWS CloudFormation modules. These contain modules that can be imported and re-used by other business units and development teams to build their solutions. Some example of resources we add to modules include:
  
  • vpc – Template for standard VPC
  • ad – AWS MS Active directory service
  • alb – Application Load Balancer
  • api – API Gateway
  • asg – An auto-scaling group of ec2 instances
  • cloudfront – A cloudfront distribution
  • ebs-backup – Lambda to take EBS snapshots
  • ec2 – An ec2 instance templates with all corporate tags
  • efs – Template for Elastic File System volume
  • elasticache – Template for Elasticache
  • elasticsearch – Template for ElasticSearch
  • elb – Elastic Load Balancer
  • iam – IAM roles, cross-account and federated
  • jenkins – Template to create Jenkins master
  • lambda – Template to create Lambda functions
  • persistent_asg – Template to create an auto-scaling group which preserves IP addresses and EBS volumes
  • rds – RDS Databases

• X-shared-services – This repository stores the infrastructure code of the resources in the shared service account. By shared services, we mean that some of the applications and/or resources are shared across multiple environments. For example, a single Jenkins server might be used to deploy code to ‘dev’, ‘qa’ and ‘prod’ environments and as such the Jenkins server would belong to a shared services environment. An example shared services that might be found here are Jenkins, ELK, or Splunk for logging, Prometheus or similar tool for monitoring, Transit VPC, etc.

• X-network-iam-SolutionName – This repository stores the infrastructure code for the network and IAM related elements like VPCs, Subnets, etc. specific to a given solution.

• X-app-SolutionName – This repository stores solution-specific TerraForm code for elements of the application itself, e.g., ELBs, EC2 instances, Auto-scaling groups, etc.

Source Code Repository & DevOps

We recommend developers use a commercial-grade code repository system that enables them to safely and securely store code while simultaneously offering the agility needed to collaborate on projects. As DevOps proponents, we also favor systems that help increase the speed of the development cycle.

Even more to the point, CI/CD is a core pillar of DevOps automation and one that should be supported via your code repository system. For example, for a Fortune 500 airline, we set up an infrastructure CI/CD pipeline using CloudFormation, AWS CodeCommit, AWS CodePipeline, and Jenkins. Developers would use a Git flow methodology, and once they merged code to the master branch in CodeCommit, polling would alert the CI/CD to a new commit. CodePipeline would create a CloudFormation ChangeSet, and queue it up for review by the Ops team. Once the Ops team approved the changeset by manually pressing a button, CodePipeline would trigger a CloudFormation stack update. This allowed for very high full-stack agility while not losing central governance.

For greater detail on implementing DevOps Automation and CI/CD in your environment, check out these resources:

• A Primer: What is DevOps? And how do Code, Config, CI/CD & Containers Relate
• Setting Up an Efficient CI/CD Pipeline
• AWS CodePipeline, CloudFormation & Continuous Delivery

Get Started with AWS

Receive AWS tips, DevOps best practices, news analysis, commentary and more. Sign up for our IT Modernization blog here and set your topic and frequency preferences. Or, download our guide on getting started with AWS, establishing a secure AWS enterprise architecture with Flux7 Landing Zones.