Apr 10, 20233 min read

Dynamic Environments in Kubernetes: Why Fixed Staging is an Anti-Design Pattern - Blog #21

Updated: Apr 12, 2023

Summary

A software team usually has numerous predefined test environments for virtual machines and bare metal servers to test an application before it goes into production. At least three environments (QA/staging/production) are usual, but larger companies may have more. The "integration environment" setting collects all developer features after they are merged to the mainline branch. Feature isolation is the most important factor in the integration environment.

Single Integration Layer

When numerous developers merge features and something breaks, a single integration environment makes it difficult to identify the offending feature(s). If three devs merge their features on a staging environment and the deployment fails (build, integration tests, metrics explode), there are several scenarios pop up and it’s hard for the team to manage it efficiently.

Booking Paradigm

Companies usually use the "booking Paradigm" to fix this:

Each developer "books" the staging environment to try their feature alone.

Developers try features in the company's "staging" environments (as a single environment can quickly become a bottleneck).

Dynamic Staging

Developers must coordinate on environments and track their cleaning actions, making this scenario difficult. If multiple developers need a database changeset with their feature, they must ensure that each staging environment's database includes only their changes and not the previous developer's. Multiple permanent staging environments can also rapidly experience configuration drift, where environments are supposed to be the same but are not after several ad hoc changes.

Of course, dynamic environments that are created and destroyed on demand are the answer. Kubernetes simplifies this:

There are many approaches to this problem, but one minimum requirement is that each Open Pull Request generates its own unique, sandboxed environment. When a pull request is merged or closed, or after a certain period of time has passed, the environment is destroyed immediately.If you continue to use permanent environments for feature testing, you are making life difficult for your developers and wasting system resources when your environments are not in use.