This is the single entry point for any AI agent (or new contributor) working on the KubeKey v4 codebase. It covers what the project is, how the code actually flows, and the conventions you must follow when changing it.
Read it top to bottom once; afterwards use Β§3 (Code Logic Map) and Β§8 (File-to-Concern Map) as lookup tables.
Further reading:
- README.md β user-facing intro.
- docs/en/framework/README.md β writing custom playbooks.
KubeKey v4 is a Go-based task execution framework modeled on Ansible. Its primary use case is installing and managing Kubernetes clusters, but the core engine is generic: it loads playbook projects (YAML), executes tasks across hosts via connectors (SSH/local/Kubernetes/Prometheus), and provides built-in modules (command, copy, template, image, etc.).
Two binaries are produced:
kkβ CLI tool that runs playbooks locally or inside a Kubernetes pod.kk-controller-managerβ Kubernetes operator that watchesPlaybookCRs and spawns executor pods.
A third build target, capkk, provides the Cluster API infrastructure provider (built with the clusterapi tag).
kubekey/
βββ cmd/kk # CLI binary entry
βββ cmd/controller-manager # Operator binary entry
βββ api/ # Separate Go module for CRD Go types
β βββ core/v1 # Playbook / Inventory / Config CRDs
β βββ core/v1alpha1 # Task CRD
β βββ project/v1 # Playbook YAML types (play/role/block/...)
βββ pkg/ # Core packages
β βββ executor/ # Playbook/role/block/task execution engine
β βββ project/ # Project loading (builtin/local/git)
β βββ modules/ # Built-in modules
β βββ variable/ # Variable merging and lookup
β βββ connector/ # SSH/local/k8s/prometheus connectors
β βββ converter/ # BlockβTask conversion, template rendering
β βββ manager/ # commandManager/controllerManager/webManager
β βββ controllers/ # Kubernetes reconcilers and webhooks
β βββ proxy/ # Hybrid REST API proxy
β βββ web/ # HTTP services
β βββ const/ # Constants, scheme, workdir helpers
β βββ utils/ # Small utilities
βββ builtin/core/ # Embedded playbooks/roles (requires "builtin" tag)
βββ plugins/ # Optional community playbooks/roles
βββ config/ # Generated CRDs, Helm charts, Kustomize
βββ docs/ # Documentation
β βββ en/framework/ # User-facing framework docs
βββ Makefile # Build targets, generate, test, lint
βββ go.mod # Main module
βββ go.work # Workspace including ./api
βββ version/ # Build-time version injection
Exact code paths for the most important flows, so you can jump straight to the right function when debugging or adding a feature.
kk binary entry:
cmd/kk/kubekey.go:main()
βββ app.NewRootCommand().Execute()
Root command construction:
cmd/kk/app/root.go:NewRootCommand()
βββ options.AddProfilingFlags() # pprof/gops
βββ options.AddKlogFlags()
βββ options.AddGOPSFlags()
βββ newRunCommand()
βββ newPlaybookCommand()
βββ newVersionCommand()
βββ newWebCommand()
βββ internalCommand... # built-in commands registered by init()
Built-in commands registration (gated by //go:build builtin):
cmd/kk/app/builtin.go:init()
βββ imports cmd/kk/app/builtin/* packages
cmd/kk/app/builtin/create.go:init()
βββ internalCommand = append(internalCommand, newCreateCommand())
cmd/kk/app/builtin/add.go:init()
cmd/kk/app/builtin/delete.go:init()
cmd/kk/app/builtin/init.go:init()
cmd/kk/app/builtin/precheck.go:init()
cmd/kk/app/builtin/artifact.go:init()
cmd/kk/app/builtin/certs.go:init()
Each built-in command constructs a CommonOptions and calls CommonOptions.Run().
Built-in command flow (example: create cluster):
cmd/kk/app/builtin/create.go:newCreateCommand()
βββ cmd.RunE = func(...)
βββ options.NewCommonOptions()
β βββ sets up Playbook/Inventory/Config references
βββ options.Complete()
β βββ resolve inventory/config files
β βββ apply --set overrides
β βββ determine workdir
βββ options.Run()
βββ manager.NewCommandManager(playbook, inventory, config)
βββ commandManager.Run()
βββ executor.NewPlaybookExecutor(...).Exec(ctx)
Arbitrary playbook flow (kk run):
cmd/kk/app/run.go:newRunCommand()
βββ options.KubeKeyRunOptions.Complete()
βββ project.New() for git/local project
βββ build Playbook CR pointing at that project
βββ options.Run() -> CommandManager -> PlaybookExecutor
In-cluster executor (kk playbook):
cmd/kk/app/playbook.go:newPlaybookCommand()
βββ PlaybookOptions.Complete()
βββ read Playbook CR from API server
βββ read Inventory/Config CRs
βββ Run() -> CommandManager -> PlaybookExecutor
All three binaries converge on the Manager interface in pkg/manager/manager.go.
Command manager:
pkg/manager/command_manager.go:NewCommandManager()
βββ Run(ctx)
βββ create controller-runtime client for local file storage
βββ if local run and not dry-run: create/update Playbook CR locally
βββ PlaybookExecutor.Exec(ctx)
Controller manager:
cmd/controller-manager/controller_manager.go:main()
βββ app.NewControllerManagerCommand().Execute()
βββ pkg/manager/controller_manager.go:NewControllerManager().Run(ctx)
βββ create controller-runtime manager
βββ register enabled controllers via options.Register()
βββ mgr.Start(ctx)
Controllers register in pkg/controllers/core/register.go:init() and pkg/controllers/infrastructure/register.go:init().
Web manager:
pkg/manager/web_manager.go:NewWebManager().Run(ctx)
βββ create local REST config via pkg/proxy
βββ build go-restful container
βββ pkg/web.NewCoreService()
βββ pkg/web.NewSchemaService()
βββ pkg/web.NewUIService()
βββ http.ListenAndServe()
Project factory:
pkg/project/project.go:New(ctx, playbook, update)
βββ if playbook address looks like git: newGitProject()
β βββ go-git clone/pull into workdir
βββ else if BuiltinsProjectAnnotation is set: builtinProjectFunc()
β βββ builtin/core.BuiltinPlaybook embed.FS
βββ else: newLocalProject()
βββ os.DirFS(path)
Playbook marshaling:
pkg/project/project.go:MarshalPlaybook()
βββ ReadFile(playbook.yaml)
βββ yaml.Unmarshal -> kkprojectv1.Playbook
βββ resolve import_playbook recursively
βββ load vars_files
βββ load roles:
β βββ read defaults/main.yaml
β βββ read meta/main.yaml dependencies
β βββ recursively marshal dependency roles
βββ expand include_tasks
βββ validate playbook/role/block
YAML project types live in api/project/v1/:
playbook.go:Playbookβ top-level list of Plays.play.go:Playβ hosts, gather_facts, vars_files, roles, pre_tasks/tasks/post_tasks.role.go:Role/RoleInfoβ dependencies, name, blocks.block.go:Blockβ nested block/rescue/always or leaf task.base.go:Baseβ name, connection, vars, environment, run_once, ignore_errors, become.taggable.go:Taggableβ tags logic (always/never/all/tagged).conditional.go:Whenβ conditional evaluation.
Executor creation:
pkg/executor/playbook_executor.go:NewPlaybookExecutor(client, playbook, variable, logOutput)
βββ returns *playbookExecutor{ option{...}, project }
Playbook execution:
pkg/executor/playbook_executor.go:Exec(ctx)
βββ project.MarshalPlaybook() -> kkprojectv1.Playbook
βββ set Playbook phase Running
βββ for each Play:
β βββ select hosts from inventory by pattern
β βββ group hosts by serial batches
β β βββ pkg/converter/converter.go:GroupHostBySerial()
β βββ for each batch:
β β βββ gather_facts (if play.gather_facts != false)
β β β βββ setup module on each host
β β βββ run pre_tasks
β β βββ for each role:
β β β βββ roleExecutor.Exec(ctx)
β β βββ run tasks
β β βββ run post_tasks
β βββ update Playbook status
βββ set Playbook phase Succeeded/Failed
βββ store final result
Role execution:
pkg/executor/role_executor.go:Exec(ctx)
βββ merge role defaults into variable system
βββ recursively execute dependency roles
β βββ dependency role inherits parent role's when/tags/ignore_errors
βββ for each block in role:
βββ blockExecutor.Exec(ctx)
βββ blocks inherit role's when conditions
when defined on a role is merged with parent conditions and passed down to all blocks and tasks within that role.
Block execution:
pkg/executor/block_executor.go:Exec(ctx)
βββ evaluate tags: skip block if tags don't match
βββ merge block's when condition with parent when conditions
βββ if block has nested block/rescue/always:
β βββ run block tasks
β βββ on failure: run rescue tasks
β βββ always: run always tasks
βββ else (leaf task):
βββ taskExecutor.Exec(ctx)
βββ all inherited when conditions are evaluated per host
when conditions are cumulative: a block or task must satisfy its own when expressions and all inherited when expressions from parent blocks and roles.
Task execution:
pkg/executor/task_executor.go:Exec(ctx)
βββ if loop: expand loop items
βββ create/update kkcorev1alpha1.Task CR
βββ for each host in parallel (wait.Group):
β βββ evaluate per-host when condition
β βββ create progress bar
β βββ FindModule(moduleName)
β βββ moduleExecFunc(ctx, ExecOptions)
β βββ evaluate failed_when
β βββ handle ignore_errors
β βββ store register/result variables
βββ update Task CR status
Module discovery:
pkg/executor/block_executor.go:MarshalBlock()
βββ iterate over UnknownField(s) in kkprojectv1.Block
βββ first unknown key that matches a registered module -> ModuleName
Variable structure:
pkg/variable/variable.go:value
βββ Config kkcorev1.Config
βββ Inventory kkcorev1.Inventory
βββ Hosts map[string]host
β βββ RemoteVars map[string]any # gather_facts
β βββ RuntimeVars map[string]any # set_fact, register
βββ Result map[string]any
Lookup precedence:
pkg/variable/variable_get.go:GetFunc
βββ resolves in order:
1. Config vars
2. Host-specific inventory vars
3. Group vars (for groups containing host)
4. Inventory vars
5. Runtime vars
6. Remote vars
Merge paths:
pkg/variable/variable_merge.go
βββ MergeRemoteVariable() # gather_facts -> Hosts[host].RemoteVars
βββ MergeRuntimeVariable() # set_fact/register -> Hosts[host].RuntimeVars
βββ MergeHostsRuntimeVariable() # cross-host variable injection
βββ MergeResultVariable() # task result -> Playbook.Status.Result
Persistence:
pkg/variable/source/file_source.go
βββ reads/writes per-host vars to
<workdir>/runtime/<namespace>/<playbook>/variable/<hostname>.yaml
Factory:
pkg/connector/connector.go:NewConnector(host, vars, logger)
βββ connector.type == "local" -> localConnector
βββ connector.type == "ssh" -> sshConnector
βββ connector.type == "kubernetes" -> kubernetesConnector
βββ connector.type == "prometheus" -> prometheusConnector
βββ default:
βββ localhost -> localConnector
βββ otherwise -> sshConnector
SSH connector (pkg/connector/ssh_connector.go): Init() parses auth (password/key); ExecuteCommand() via golang.org/x/crypto/ssh; PutFile() / FetchFile() via sftp.
Local connector (pkg/connector/local_connector.go): ExecuteCommand() via os/exec; PutFile() / FetchFile() operate on the local filesystem.
Fact gathering (pkg/connector/gather_facts.go): local/ssh connectors implement HostInfo(), collecting OS, arch, hostname, IP, memory and CPU facts.
Modules must stay decoupled from the local OS: always go through the Connector interface, never hardcode Linux paths or bash-only syntax.
Registry:
pkg/modules/internal/options.go
βββ RegisterModule(fn, names...)
βββ FindModule(name)
βββ ModuleExecFunc signature
Module list (registered in pkg/modules/module.go):
| Module | Package | Key file |
|---|---|---|
| add_hostvars | pkg/modules/add_hostvars |
add_hostvars.go |
| assert | pkg/modules/assert |
assert.go |
| command/shell | pkg/modules/command |
command.go |
| copy | pkg/modules/copy |
copy.go |
| debug | pkg/modules/debug |
debug.go |
| fetch | pkg/modules/fetch |
fetch.go |
| gen_cert | pkg/modules/gen_cert |
gen_cert.go |
| http_get_file | pkg/modules/http_get_file |
http_get_file.go |
| image | pkg/modules/image |
image.go, image_deprecated.go, repository.go |
| include_vars | pkg/modules/include_vars |
include_vars.go |
| prometheus | pkg/modules/prometheus |
prometheus.go |
| result | pkg/modules/result |
result.go |
| set_fact | pkg/modules/set_fact |
set_fact.go |
| setup | pkg/modules/setup |
setup.go |
| template | pkg/modules/template |
template.go |
Representative implementations:
pkg/modules/command/command.go:ModuleCommand(ctx, opts)
βββ render args through template
βββ build command string
βββ opts.Connector.ExecuteCommand(cmd)
βββ return stdout/stderr
pkg/modules/copy/copy.go:ModuleCopy(ctx, opts)
βββ resolve src/content
βββ optionally template content
βββ opts.Connector.PutFile(data, dst, mode)
βββ return result
pkg/modules/template/template.go:ModuleTemplate(ctx, opts)
βββ read src template
βββ render with variables
βββ opts.Connector.PutFile(rendered, dst, mode)
βββ return result
Rendering:
pkg/converter/tmpl/template.go:ParseFunc()
βββ if string contains "{{" and "}}":
β βββ text/template.Execute()
βββ else return original string
Custom functions (pkg/converter/tmpl/functions.go), on top of Sprig:
toYaml / fromYaml / toToml
ipInCIDR / ipFamily / isIP
pow / subtractList
fileExists / unquote / getStringSlice
toLowerByteUnit
mapToNamedStringArgs
when conditions (api/project/v1/conditional.go) are always wrapped as templates and rendered to a boolean-like result. They can be defined at role, block and task level:
- Role-level
whenis inherited by all blocks and tasks in that role. - Block-level
whenis merged with parent block/role conditions and inherited by nested blocks and leaf tasks. - Task-level
whenis evaluated per host right before module execution. - All inherited conditions must evaluate to true for a task to run.
Tags (api/project/v1/taggable.go):
Tags []string
AlwaysTag = "always"
NeverTag = "never"
AllTag = "all"
TaggedTag = "tagged"
IsEnabled(onlyTags, skipTags) bool
Tags are inherited the same way as when: role β block β nested block/task. Use JoinTag() to merge parent tags into child tags. Runtime filtering uses playbook.Spec.Tags (only run matching) and playbook.Spec.SkipTags (skip matching), typically set via CLI --tags and --skip-tags.
| Tag | Meaning |
|---|---|
always |
Always runs unless explicitly skipped by always in skipTags. |
never |
Never runs unless explicitly included. |
all |
Matches every block except those tagged never. |
tagged |
Matches any block that has at least one tag. |
Matching rules for onlyTags:
- A block with
alwaysruns. allortaggedruns everything exceptnever.- Otherwise the block runs if any of its tags intersect with
onlyTags. - Blocks without matching tags are skipped.
Matching rules for skipTags:
allskips everything except blocks taggedalways(unlessalwaysis also in skipTags).- Any tag intersection with
skipTagsskips the block. taggedskips all tagged blocks.
Like when, declare tags at the highest applicable scope; do not repeat the same tag at every nested level (see Β§4.4).
Playbook controller:
pkg/controllers/core/playbook_controller.go:Reconcile()
βββ fetch Playbook CR
βββ if no executor Pod exists:
β βββ create Pod running "kk playbook --name <name> --namespace <ns>"
βββ watch owned Pods
βββ sync Playbook status from Pod phase/logs
CAPKK controllers live in pkg/controllers/infrastructure/: inventory_controller.go, kkcluster_controller.go, kkmachine_controller.go.
Registration:
pkg/controllers/core/register.go:init()
βββ options.Register(&PlaybookReconciler{})
βββ options.Register(&PlaybookWebhook{})
pkg/controllers/infrastructure/register.go:init()
βββ registers Inventory/KKCluster/KKMachine reconcilers
Hybrid REST config:
pkg/proxy/transport.go:RestConfig()
βββ if no k8s cluster: use file-based storage for Task/Inventory/Playbook
βββ if cluster exists: forward non-local resources to API server,
keep Task local
This lets kk run without a Kubernetes cluster while still using controller-runtime clients.
Web services (pkg/web/service.go):
NewCoreService() # /api/v1/playbooks, /inventories, /logs
NewSchemaService() # schema listing and config
NewUIService() # static SPA UI
NewSwaggerUIService() # swagger UI
NewAPIService() # OpenAPI JSON
High-level flow of builtin/core/playbooks/create_cluster.yaml:
1. native/root role on all hosts
2. hook/pre_install.yaml
3. load defaults + precheck on all hosts
4. on localhost:
generate certs, download binaries/images
5. on etcd/k8s_cluster/image_registry/nfs:
run native role
6. on etcd hosts (when external):
etcd prepare/install
7. on image_registry hosts:
docker + registry
8. on localhost (when registry configured):
push images
9. on k8s_cluster hosts:
CRI install
kubernetes pre/init/join
certs renewal
custom labels/taints
10. on a random control plane host:
CNI + storage class
11. hook/post_install.yaml
Default variables are loaded from builtin/core/defaults/ and merged before playbook execution.
Follow these conventions when producing or modifying code.
Choose the appropriate log level.
| Level | Usage |
|---|---|
klog.Info |
Main business events. |
klog.Warning |
Recoverable abnormal situations. |
klog.Error |
Errors requiring attention. |
klog.V(4) |
Framework execution flow. Examples: project, proxy, variable, connector, web, manager, controllers, executor. |
klog.V(5) |
Extension modules. Examples: module, converter. |
klog.V(6) |
Debug information. May include detailed intermediate values and execution flow. |
Wrap errors only where they originate.
- Lower layers should use
errors.Wrap(or equivalent) to add context. - Upper layers should return the error directly unless adding meaningful business context.
- Do not repeatedly wrap the same error.
KubeKey uses github.com/cockroachdb/errors with errors.Wrapf / errors.Join.
Keep names concise. Prefer meaningful short names. Avoid unnecessary abbreviations and verbose names.
Avoid:
tmpData
managerObject
projectConfigurationPrefer:
cfg
proj
mgr
connPrefer modifying existing code instead of introducing new abstractions.
- Do not introduce new structs or interfaces unless there is a clear benefit.
- Keep APIs stable.
- Minimize public surface.
- Favor composition over inheritance-like patterns.
- Do not repeat inherited conditions (e.g.
when,tags) at every level; declare them at the highest applicable scope (see Β§3.8).
- Package aliases:
kkcorev1,kkcorev1alpha1,kkprojectv1. pkg/constis imported as_constto avoid keyword collision.- Options structs have
Flags()andComplete()methods. - Modules return
(stdout, stderr, err)triples. - Controllers and built-in commands register via
init()gated by build tags. - Templates use Go
text/template+ Sprig + custom functions inpkg/converter/tmpl/.
Important Makefile targets:
| Target | Purpose |
|---|---|
make kk |
Build kk binary with BUILDTAGS=builtin |
make build-kk-dev |
Dev build with branch-based version |
make controller-manager |
Build operator image |
make generate |
Deepcopy, CRDs, RBAC, modules, goimports |
make generate-manifests-kubekey |
Generate CRDs to config/kubekey/crds/ |
make verify |
Verify generated artifacts and modules are up to date |
make test |
Run unit/integration tests with envtest |
make lint |
Run golangci-lint |
Build tags:
builtinβ includes embedded playbooks/roles and built-in CLI commands.clusterapiβ CAPKK controller-manager image build.
- Find where a built-in command is defined: search
cmd/kk/app/builtin/*.gofor the command name. - Find where a module is implemented: search
pkg/modules/<name>/<name>.goand confirm registration inpkg/modules/module.go. - Trace variable values: start at
pkg/variable/variable_get.goand add logging inGetFunc. - Trace task execution: add logging in
pkg/executor/task_executor.gobeforeFindModule. - Test a module locally: look at existing
*_test.gofiles; many use the fake connector frompkg/modules/internal/test.go. - Regenerate CRDs:
make generate-manifests-kubekey. - Build with built-ins:
make kk(setsBUILDTAGS=builtin).
Follow Conventional Commits:
<type>: <short description>
Common types:
feat:β new featurefix:β bug fixrefactor:β code refactoringdocs:β documentation onlytest:β testschore:β build, dependencies, tooling
Examples:
feat: support multiple ssh private keys
fix: preserve proxy configuration during reconnect
A PR description should include:
- What changed and why.
- How it was implemented (briefly).
- Testing performed.
- Risks / breaking changes.
| Concern | File |
|---|---|
| CLI root | cmd/kk/app/root.go |
| CLI options base | cmd/kk/app/options/option.go |
| Built-in commands | cmd/kk/app/builtin/*.go |
| Controller options | cmd/controller-manager/app/options/controller_manager.go |
| Playbook execution | pkg/executor/playbook_executor.go |
| Role execution | pkg/executor/role_executor.go |
| Block execution | pkg/executor/block_executor.go |
| Task execution | pkg/executor/task_executor.go |
| Project loading | pkg/project/project.go |
| Git project | pkg/project/git.go |
| Local project | pkg/project/local.go |
| Builtin project | pkg/project/builtin.go |
| Variable core | pkg/variable/variable.go |
| Variable get | pkg/variable/variable_get.go |
| Variable merge | pkg/variable/variable_merge.go |
| Variable source | pkg/variable/source/file_source.go |
| Connector factory | pkg/connector/connector.go |
| SSH connector | pkg/connector/ssh_connector.go |
| Local connector | pkg/connector/local_connector.go |
| Kubernetes connector | pkg/connector/kubernetes_connector.go |
| Prometheus connector | pkg/connector/prometheus_connector.go |
| Module registry | pkg/modules/module.go |
| Module internals | pkg/modules/internal/options.go |
| Template functions | pkg/converter/tmpl/functions.go |
| Template rendering | pkg/converter/tmpl/template.go |
| BlockβTask converter | pkg/converter/converter.go |
| Playbook CRD | api/core/v1/playbook_types.go |
| Inventory CRD | api/core/v1/inventory_types.go |
| Config CRD | api/core/v1/config_types.go |
| Task CRD | api/core/v1alpha1/task_types.go |
| Project YAML types | api/project/v1/playbook.go, play.go, block.go, role.go, base.go, taggable.go, conditional.go |
| Playbook controller | pkg/controllers/core/playbook_controller.go |
| Web services | pkg/web/service.go |
| REST proxy | pkg/proxy/transport.go |
| Constants/workdir | pkg/const/common.go, pkg/const/workdir.go, pkg/const/scheme.go |