Reimagining CI/CD Pipelines as Composable Blocks with Bryan Liles

Bryan Liles, Senior Staff Engineer at VMWare, talked at the DeliveryConf for ideas about patterns and recommendations when building CI/CD pipelines. Liles recommends thinking about CI/CD as patterns instead of implementations, like merely using Jenkins or Spinnaker. It should be possible to build a platform with composable blocks with replaceable components and agnostic to a technology stack.

Liles talked about how many companies traditionally build CI/CD pipelines, the purpose of having these pipelines is to move an application to a production environment. In essence, a delivery process gets the application code from source code management. Then, the pipeline compiles the application, and store its artifacts in a repository as a bundle. Finally, the pipeline ships the bundle to all different environments like dev, test, and prod. Liles said that if a delivery pipeline looks complex, perhaps the application stack shouldn’t be any more complicated.

According to Liles, engineers could reimagine deployment pipelines as composable blocks as patterns instead of implementations, like merely using Jenkins or Spinnaker. For instance, a pipeline could have an integration block for building the application, run tests, and create artifacts. This block could be using Jenkins, but in the future could use another tool like Circle CI. Another block in the pipeline could be the delivery process block that deploys the application, solicits approval, and run data migrations. Both the integration and delivery blocks have the same base components: a webhook trigger, a workflow, notifications, and approvals. And the integration and delivery blocks are part of a bigger composable block, the deployment pipeline.

Reimagining CI/CD Pipelines as Composable Blocks with Bryan Liles

Liles said, "we’ve to think more about composability in our systems," a deploy pipeline should be a platform-as-a-service (PaaS) that push an app, and then ships the app to its destination. Liles proposed to rename the deployment pipeline to a composable PaaS, where engineers can change components effortlessly. An example of this pattern is Kubernetes, which is a platform for building platforms, Liles said. So, with Kubernetes, a deployment pipeline could have the same base components as the integration and delivery blocks, but using different tools. For instance, an integration block could use Tekton or Argo for the trigger, workflow, notification, and approval blocks.

Moreover, Liles said that tools like Jenkins, Spinnaker, or ConcourseCI helped to define how to build CI/CD pipelines and how to use them. However, it’s essential to take the lessons learned from these tools and leave them behind, Liles said. Therefore, he recommends to think differently about tools by saying:

We really need to think differently about our tools, the status quo got us to where we are, but it won’t get us to where we need to be. We should be thinking bigger and differently.

Finally, Liles said that tooling is complicated, and the ecosystem should work on building smaller and composable tools. He recommends starting to research about how to improve the ecosystem together instead of making it more polished. The blocks he proposed are only the start, and he said: "the point should be to think more about composability in our systems."