Parallel and batch

Three settings decide how fast a queue drains and how much testing it does: a branch’s parallel and batch, and the host’s [runs] max. This page shows what each one does over time, why a queue tests fewer at once after a failure, and which settings suit the test suite you have.

Combinations

sluice never tests a submission on its own. Say A, B and C are queued on main, in that order. sluice forms three combinations: main + A, main + A + B and main + A + B + C. Each is what main would hold if everything up to that submission landed. A submission lands only after a combination holding it and everything ahead of it has passed.

batch decides which of those combinations are tested. parallel decides how many at once.

Parallel: every combination (batch = false)

This is the default. sluice tests each combination as soon as it is formed, up to parallel at once. It bets that the submissions ahead will pass: main + A + B starts before main + A has finished.

When everything passes, each submission lands about one test run after it was submitted. The cost is runs: four submissions are four runs at the same time, each in its own checkout and, if the tests start containers or an emulator, with its own copy of them.

Batch: the longest first (batch = true)

sluice tests only the longest combination, and starts nothing else on the branch while that run is under test. Submissions that arrive meanwhile wait. When the run ends, whatever queued up is tested together as the next batch.

No timer collects a batch. A batch is whatever queued while the last run was under test: on a quiet branch that is one submission, which starts at once; on a busy branch batches grow by themselves.

If a batch passes, every submission in it lands together, each with its own merge commit naming the same verdict.

Under batch, parallel is the largest batch. With parallel = 4, sluice tests at most the first four queued submissions together, and the rest wait for the next batch.

When something fails

With batch = false, a failure names its submission at once: main + A + B failed and main + A passed, so B broke the build. Every combination with B in it is thrown away and formed again without B.

With batch = true, a failed batch doesn’t say which submission broke it. sluice then tests the shorter combinations it skipped, as many at once as the branch allows, and blames the newest submission of the first one that failed. That costs one more run before anyone is told, and the rest of the batch waits for it.

Either way, only the blamed submission’s agent is told, and the others land without it.

Fewer at once after a failure

parallel is a ceiling, not a fixed number. A branch starts out testing parallel combinations at once. Each blamed submission halves that number, and each combination that passes adds one back. It never falls below one or rises above parallel.

With batch = false, everything stacked on a failing combination is thrown away. When failures come in a streak, because the branch is broken, a test is flaky or the host is short of memory, testing many at once throws away most of those runs. Testing fewer wastes less, and each pass raises the number again. With batch = true the same number is the size of the batch, so after failures the batches are smaller too.

sluice status shows where it stands:

$ sluice status main
main at c575d3a · testing 1 at once (up to 4)

No setting turns this off. A branch stuck at one is a branch whose submissions keep failing. If unrelated submissions all fail the same check, look for the cause in the tests or on the host. sluice status <id> names each failed check’s log.

The host’s bound

parallel is per branch. The host’s [runs] max, in ~/.config/sluice/config.toml, caps every branch together, along with the runs agents start in their own worktrees with sluice exec. With no max, the bound is a quarter of the host’s logical CPUs, and never less than one. A branch with room under its own number still waits while the host’s bound is full. reserved keeps some of the bound for submissions, so agents’ own runs can’t take all of it (configuration).

If each run starts containers, an emulator or a database, set max to how many of those the host holds in memory at once, whatever its count of cores. Two runs that each need 6 GB on a 16 GB host is max = 2.

Runs at once also have to stay out of each other’s way. sluice gives each run its own checkout and its own ports ([checkout] ports). Anything else the tests write, such as an output directory, a lock file or a port they choose themselves, has to live inside the checkout. Tests that share one break as soon as two runs overlap. The sign is unrelated submissions failing the same check, and each of those failures halves how many the branch tests at once.

Choosing

Start from how long the gate takes and how often submissions arrive.

the branch	settings	why
a gate of a few minutes	batch = false, parallel = 2 to 4	runs are cheap; each submission lands about one run after it was submitted, and a failure names its submission at once
a gate of 15 minutes or more, and submissions that arrive while a run is under test	batch = true, parallel = 4 to 8	one run lands a whole batch, so the host does a fraction of the testing
runs that each start a large stack	batch = true, and the host’s max set by memory	batch = false needs a stack for every combination at once
failures are common	batch = false	a failed batch costs a second run before anyone is told
a slow gate with few submissions	either	with nothing queued behind it, a submission is tested alone in both modes

The two diagrams above are the trade in numbers. On a 20-minute gate with four submissions a minute apart, batch = false runs four at once and lands all four by minute 23; batch = true runs two, one at a time, and lands them by minute 40. When B breaks the build, batch = false tells B at minute 20 and batch = true at minute 40. Choose batch = true when runs are scarce, because the host holds few stacks or runs cost money, and a later landing is acceptable. Choose batch = false when the host has room for several runs and each submission should land as soon as it can.

Set timeout to about twice the gate’s usual time. A run that hangs then fails in minutes, not after the default hour. A timeout counts as a failure, so it halves how many are tested at once too.

Parallel runs and batches together

On one branch, batch = true tests one combination at a time, so one branch can’t have both. Several branches can: each branch has its own queue, and all of them share the host’s bound. A repository whose agents land on bolts (bolt/*) and that lands each bolt on main tests every bolt’s queue at once.

A common split is batch = false on bolt/*, with a short gate of the fast checks, and batch = true on main, with the full gate:

[branches."bolt/*"]
required = ["#check-fast"]
parallel = 3

[branches.main]
required = ["#check-fast", "#integration"]
parallel = 4
batch = true
timeout = "45m"

A queue that tests one at a time

When sluice status says testing 1 at once and submissions wait, check these in order:

1. batch = true on the branch. One run at a time is what batch does.
2. Recent failures. Each one halved the number, and it grows back by one for each pass.
3. The host’s bound. [runs] max is full, perhaps with agents’ sluice exec runs.
4. One submission at a time. If each submission arrives after the last one landed, there is only ever one combination to test.

Every key is in the configuration reference.