K8s Anti-Design Pattern Series: Manual Kubectl Edit/Patch Deployments

Introduction

In Kubernetes deployments, configuration dispersion can be a significant issue, especially when two or more identical environments no longer share the same setup due to ad hoc deployments or adjustments. This can lead to reverse-engineering a running instance to determine the machine's original setup, which can be a time-consuming process.

Kubernetes offers a lot of flexibility when it comes to deploying applications using the 'kubectl' tool. This tool has in-built apply, edit, and patch commands that allow developers to modify resources on a running cluster. However, the issue arises when developers abuse this technique by making changes on the fly in the cluster without documenting them outside of it. Such changes can cause issues and are one of the common causes of unsuccessful deployments.

One of the problems with making ad hoc adjustments is that they are usually done for quick fixes or as a temporary measure. These adjustments are never documented, leading to undocumented changes to the cluster. Furthermore, if the configuration of the production environment has changed from that of the testing environment, the deployment will fail in production even though it succeeded in staging.

To avoid these issues, it is strongly recommended that manual distributions not be performed using Kubectl. Instead, the GitOps paradigm suggests that all deployments be managed by the deployment tool and, ideally, documented there. By exclusively using Git commits for deployments, developers can keep track of cluster activity, be fully aware of the precise contents of each cluster, and have a simple way to clone or recreate an entire system.

Git change history provides a comprehensive record of cluster activity, and reading the Git configuration is a simple way to clone or recreate an entire system. Additionally, resetting the cluster to an earlier commit will restore its prior configuration. Most significantly, in the event of a failed deployment, developers can quickly identify the most recent change that affected it and how that change impacted its configuration.

All editing and patching done with kubectl should be considered experimental. Attempting to make manual resource changes to a cluster in production is extremely risky and should be avoided. It's important to have a well-thought-out deployment process, but it's also crucial to concur with your colleagues that this misuse of kubectl is unacceptable.

In summary, manual deployments using the kubectl tool can cause configuration dispersion and lead to undocumented changes and unsuccessful deployments. To avoid these issues, the GitOps paradigm is recommended, where all deployments are managed by a deployment tool and documented in Git commits. By avoiding manual distributions using Kubectl, a well-thought-out deployment process can be maintained.