BitBake Overview
About 779 wordsAbout 3 min
2025-12-21
What BitBake Is
BitBake is a generic task execution engine — conceptually similar to GNU Make but with a Python-based metadata language, a parallel task scheduler, and a shared-state cache system. It is maintained as a standalone project by the OpenEmbedded community and used by both Yocto and OpenEmbedded builds.
The core job: read a collection of text files (recipes, configuration, classes), construct a dependency graph of all tasks needed to satisfy a build request, then execute those tasks in the correct order — parallelising where possible, skipping tasks whose outputs are cached.
Parsing Phases
BitBake startup follows four sequential phases before any task executes:
Phase 1: Configuration Parsing
└─ Read bitbake.conf → site.conf → local.conf → bblayers.conf
└─ Result: global DataStore (the `d` object at conf scope)
Phase 2: Recipe Parsing
└─ For each .bb file found via BBFILES:
├─ Parse recipe variables into per-recipe DataStore
├─ Apply inherited .bbclass files (inject task functions)
└─ Apply .bbappend files from higher-priority layers
Phase 3: Dependency Resolution
└─ Build task dependency graph from DEPENDS, RDEPENDS, [deptask], [recrdeptask]
└─ Generate task hash (signature) for each node
Phase 4: Task Execution
├─ Check sstate-cache for matching hash
├─ If hit: restore cached output via do_setscene (~1 second)
└─ If miss: execute task, then write output to sstate-cacheThe DataStore (d Object)
Every recipe runs with its own DataStore — a key/value store holding all variables for that recipe's context. In Python code within a recipe or class, d is always the DataStore:
python do_my_task() {
# Read a variable
srcdir = d.getVar('S')
# Write a variable
d.setVar('MY_FLAG', '1')
# Expand a variable (process ${} references)
workdir = d.expand('${WORKDIR}/output')
# Append to a variable
d.appendVar('CFLAGS', ' -DMY_DEFINE=1')
}Variable Assignment Operators
BitBake has distinct assignment operators with different expansion semantics. This is the #1 source of hard-to-diagnose build problems.
| Operator | When ${VAR} expands | Notes |
|---|---|---|
VAR = "val" | Parse time | Standard assignment |
VAR := "val" | Forced parse time | Immediately expands RHS |
VAR ?= "val" | Parse time | Only if VAR not already set |
VAR ??= "val" | Parse time | Lowest priority default |
VAR += "val" | Parse time | Appends with a space |
VAR .= "val" | Parse time | Appends without space |
VAR:prepend = "val" | At variable read time | Applied after all = assignments |
VAR:append = "val" | At variable read time | Applied after all = assignments |
VAR:remove = "val" | At variable read time | Removes token from the value |
The :append and :prepend operators use a deferred model — stored as modifiers, applied to the final value only when the variable is read. This is why a .bbappend can use SRC_URI:append = " file://patch.patch" and guarantee the result, regardless of parse order.
sstate-cache: Task Hash Mechanics
The sstate-cache is BitBake's most important performance feature. Each task has a signature hash computed from:
- The set of input variables that affect that task's output
- The task's function body (shell or Python code)
- The signatures of all tasks it depends on
If the computed hash matches an entry in the sstate-cache, BitBake runs do_setscene — restoring cached output in seconds instead of recompiling.
# Inspect what went into a task's hash signature
bitbake-dumpsig -d tmp/stamps/<arch>/<recipe>/<ver>/do_compile.sigdata.<hash>
# Compare two signatures to find why a cache miss occurred
bitbake-dumpsig -d <sig1> <sig2>Common cause of unexpected cache misses: a variable included in sigdata that changes between builds (e.g., a DATE variable, hostname, or a SRCREV pointing at a moving branch). Add such variables to BB_HASHBASE_WHITELIST to exclude them from hash computation.
Task Execution and Parallelism
# Number of BitBake tasks running simultaneously (across recipes)
BB_NUMBER_THREADS = "8" # typically = number of logical CPUs
# Parallelism within a single make invocation
PARALLEL_MAKE = "-j 8"BitBake maintains a runqueue of ready tasks (tasks whose dependencies are all complete) and feeds them to a thread pool of BB_NUMBER_THREADS workers. Multiple recipes can compile simultaneously.
Anonymous Python Functions
BitBake supports anonymous Python functions that run at parse time (not at task execution time), useful for computing variable values dynamically:
# Runs when the recipe is parsed, before any task executes
python () {
machine = d.getVar('MACHINE')
if machine.startswith('raspberrypi'):
d.setVar('ENABLE_GPU', '1')
else:
d.setVar('ENABLE_GPU', '0')
}Useful Introspection Commands
# Dump the complete variable environment for a recipe (invaluable for debugging)
bitbake -e myrecipe | grep ^WORKDIR=
bitbake -e myrecipe | grep ^SRC_URI=
bitbake -e myrecipe > /tmp/env.txt
# Show which layer provides a recipe
bitbake-layers show-recipes | grep my-recipe
# Show the task dependency graph (generates .dot files)
bitbake -g core-image-minimal
dot -Tsvg task-depends.dot > task-deps.svg
# List all tasks for a recipe
bitbake -c listtasks myrecipe
# Open an interactive shell with the recipe's full build environment
bitbake -c devshell myrecipe