> For the complete documentation index, see [llms.txt](https://dinush-chathurya.gitbook.io/kustomize-e-book/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://dinush-chathurya.gitbook.io/kustomize-e-book/additional-topics/15-pitfalls-and-lessons-learned.md).

# Pitfalls and Lessons Learned

Even with best practices, real-world Kustomize projects encounter challenges. In this chapter, we’ll explore common pitfalls, how to avoid them, and lessons learned from production incidents.

***

## 1. Over-Patching & Complexity

**Pitfall:** Creating too many small patches leads to confusing, brittle overlays.

* **Symptom:** Hard to trace which patch controls a given change.
* **Solution:**
  * Group related changes into fewer, well-named patch files (e.g., `replicas-and-resources.yaml` instead of separate `replicas.yaml` and `resources.yaml`).
  * Use `components/` for reusable snippets (logging sidecars, monitoring).

***

## 2. Configuration Drift

**Pitfall:** Cluster state drifts from Git because of manual edits.

* **Symptom:** `kubectl diff` shows unexpected changes; ArgoCD reports “OutOfSync.”
* **Solution:**
  * Enable `selfHeal` and `prune` in your GitOps tool’s `syncPolicy`.
  * Block out-of-band changes via admission controllers.
  * Educate teams to never edit live clusters directly.

***

## 3. Slow Builds & Large Manifests

**Pitfall:** `kustomize build` and `kubectl apply -k` become slow as resource counts grow.

* **Symptom:** CI jobs time out; deployments stall.
* **Solution:**
  * Split monolithic overlays into smaller “apps of apps.”
  * Cache generated YAML in CI artifacts.
  * Optimize `resources:` lists and avoid unnecessary files.

***

## 4. Incorrect Patch Ordering

**Pitfall:** Patches fail to apply because of wrong order or targets.

* **Symptom:** Base values remain unchanged after build.
* **Solution:**
  * Reference patches **after** base resources in `kustomization.yaml`.
  * Ensure `metadata.name` and `kind` match exactly.
  * Inspect merged output with `kubectl kustomize overlays/dev`.

***

## 5. Secret Leakage

**Pitfall:** Sensitive data accidentally committed in plain text.

* **Symptom:** Secrets appear in Git history or build logs.
* **Solution:**
  * Use SealedSecrets or External Secrets rather than `secretGenerator` literals.
  * Add local secret files to `.gitignore`.
  * Scan for secrets in CI (e.g., truffleHog, GitGuardian).

***

## 6. CRD & CR Version Mismatches

**Pitfall:** CRDs and CRs get out of sync; apply fails due to schema errors.

* **Symptom:** “unknown field” or validation errors when applying CRs.
* **Solution:**
  * Pin CRD versions in `base/crds/`.
  * Apply CRDs separately before CRs (e.g., `kubectl apply -k base/crds`).
  * Use `kubectl apply --dry-run=server` in CI to catch schema incompatibilities.

***

## 7. Tooling & Plugin Incompatibilities

**Pitfall:** Plugins or Kustomize versions break builds.

* **Symptom:** `kustomize build` errors referencing plugin code.
* **Solution:**
  * Version-lock your Kustomize binary (e.g., use a Docker image tag).
  * Test upgrades in an isolated environment.
  * Contribute fixes upstream or maintain a fork if needed.

***

## Lessons Learned

1. **Document Everything**\
   Keep a top-level `README.md` explaining structure, patch conventions, and promotion workflows.
2. **Adopt Incrementally**\
   Start with one service or overlay and expand as the team gains confidence.
3. **Automate Validation**\
   Lint, diff, and dry-run every overlay on PRs—no manual steps allowed.
4. **Perform Regular Audits**\
   Quarterly reviews to remove unused patches, bases, and deprecated resources.
5. **Train Your Team**\
   Host knowledge-sharing sessions so everyone follows the same patterns and tools.

***

## Summary

By understanding these common pitfalls and applying the recommended solutions, you can keep your Kustomize projects robust, maintainable, and scalable. Learning from real incidents helps you build resilient GitOps workflows and avoid repeating mistakes.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://dinush-chathurya.gitbook.io/kustomize-e-book/additional-topics/15-pitfalls-and-lessons-learned.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.