What BitBake Is

BitBake is a generic task execution engine — conceptually similar to GNU Make but with a Python-based metadata language, a parallel task scheduler, and a shared-state cache system. It is maintained as a standalone project by the OpenEmbedded community and used by both Yocto and OpenEmbedded builds.

The core job: read a collection of text files (recipes, configuration, classes), construct a dependency graph of all tasks needed to satisfy a build request, then execute those tasks in the correct order — parallelising where possible, skipping tasks whose outputs are cached.

Parsing Phases

BitBake startup follows four sequential phases before any task executes:

Phase 1: Configuration Parsing
  └─ Read bitbake.conf → site.conf → local.conf → bblayers.conf
  └─ Result: global DataStore (the `d` object at conf scope)

Phase 2: Recipe Parsing
  └─ For each .bb file found via BBFILES:
     ├─ Parse recipe variables into per-recipe DataStore
     ├─ Apply inherited .bbclass files (inject task functions)
     └─ Apply .bbappend files from higher-priority layers

Phase 3: Dependency Resolution
  └─ Build task dependency graph from DEPENDS, RDEPENDS, [deptask], [recrdeptask]
  └─ Generate task hash (signature) for each node

Phase 4: Task Execution
  ├─ Check sstate-cache for matching hash
  ├─ If hit:  restore cached output via do_setscene (~1 second)
  └─ If miss: execute task, then write output to sstate-cache

The DataStore (d Object)

Every recipe runs with its own DataStore — a key/value store holding all variables for that recipe's context. In Python code within a recipe or class, d is always the DataStore:

python do_my_task() {
    # Read a variable
    srcdir = d.getVar('S')

    # Write a variable
    d.setVar('MY_FLAG', '1')

    # Expand a variable (process ${} references)
    workdir = d.expand('${WORKDIR}/output')

    # Append to a variable
    d.appendVar('CFLAGS', ' -DMY_DEFINE=1')
}

Variable Assignment Operators

BitBake has distinct assignment operators with different expansion semantics. This is the #1 source of hard-to-diagnose build problems.

Operator	When ${VAR} expands	Notes
VAR = "val"	Parse time	Standard assignment
VAR := "val"	Forced parse time	Immediately expands RHS
VAR ?= "val"	Parse time	Only if VAR not already set
VAR ??= "val"	Parse time	Lowest priority default
VAR += "val"	Parse time	Appends with a space
VAR .= "val"	Parse time	Appends without space
VAR:prepend = "val"	At variable read time	Applied after all = assignments
VAR:append = "val"	At variable read time	Applied after all = assignments
VAR:remove = "val"	At variable read time	Removes token from the value

The :append and :prepend operators use a deferred model — stored as modifiers, applied to the final value only when the variable is read. This is why a .bbappend can use SRC_URI:append = " file://patch.patch" and guarantee the result, regardless of parse order.

sstate-cache: Task Hash Mechanics

The sstate-cache is BitBake's most important performance feature. Each task has a signature hash computed from:

1. The set of input variables that affect that task's output
2. The task's function body (shell or Python code)
3. The signatures of all tasks it depends on

If the computed hash matches an entry in the sstate-cache, BitBake runs do_setscene — restoring cached output in seconds instead of recompiling.

# Inspect what went into a task's hash signature
bitbake-dumpsig -d tmp/stamps/<arch>/<recipe>/<ver>/do_compile.sigdata.<hash>

# Compare two signatures to find why a cache miss occurred
bitbake-dumpsig -d <sig1> <sig2>

Common cause of unexpected cache misses: a variable included in sigdata that changes between builds (e.g., a DATE variable, hostname, or a SRCREV pointing at a moving branch). Add such variables to BB_HASHBASE_WHITELIST to exclude them from hash computation.

Task Execution and Parallelism

# Number of BitBake tasks running simultaneously (across recipes)
BB_NUMBER_THREADS = "8"   # typically = number of logical CPUs

# Parallelism within a single make invocation
PARALLEL_MAKE = "-j 8"

BitBake maintains a runqueue of ready tasks (tasks whose dependencies are all complete) and feeds them to a thread pool of BB_NUMBER_THREADS workers. Multiple recipes can compile simultaneously.

Anonymous Python Functions

BitBake supports anonymous Python functions that run at parse time (not at task execution time), useful for computing variable values dynamically:

# Runs when the recipe is parsed, before any task executes
python () {
    machine = d.getVar('MACHINE')
    if machine.startswith('raspberrypi'):
        d.setVar('ENABLE_GPU', '1')
    else:
        d.setVar('ENABLE_GPU', '0')
}

Useful Introspection Commands

# Dump the complete variable environment for a recipe (invaluable for debugging)
bitbake -e myrecipe | grep ^WORKDIR=
bitbake -e myrecipe | grep ^SRC_URI=
bitbake -e myrecipe > /tmp/env.txt

# Show which layer provides a recipe
bitbake-layers show-recipes | grep my-recipe

# Show the task dependency graph (generates .dot files)
bitbake -g core-image-minimal
dot -Tsvg task-depends.dot > task-deps.svg

# List all tasks for a recipe
bitbake -c listtasks myrecipe

# Open an interactive shell with the recipe's full build environment
bitbake -c devshell myrecipe

Contributors: Vasu Grover

Previous pageIntroduction

Next pagePoky Reference System