Splitting and balancing a scripted test
scripts/run_unit_tests.py (the rut task) runs a test listed in
_SPLIT_TEST_GROUPS as N tasks with GROUP_ID = 1..N. A chunk whose header ends
in (G) runs only in group G; a chunk with no suffix runs in every group. The
goal is N groups, in file order (group 1 first … group N last), that finish at
the same time. Every group must still pass on its own.
The mechanics are in scripts/balance_chunk_groups.py, and its docstring is
the reference for them. This skill covers the procedure and the judgement calls
that the script can't make.
Tools
- Timing collection is automatic.
shell_script_tester.cc records every
executed chunk to $CHUNK_TIMINGS_FILE. run_unit_tests.py sets that variable
for every task and merges the results of passing tasks into
test-chunk-times.txt, next to the -t file (unittest/). Chunks that run
in every group are averaged.
balance_chunk_groups.py <script> <N> [--explain] [--write] splits the
grouped chunks into N contiguous groups, minimizing the slowest one. It never
cuts through a detected dependency. Without --write it prints a table and a
diff.
balance_chunk_groups.py <script> --init does the first split of an
unsplit script. It gives every chunk except the INCLUDEs, entry point,
Setup and Cleanup (plus any --common ID) a (1) placeholder for the
balancing step to renumber.
balance_chunk_groups.py <script> <N> --estimate balances by each chunk's
line count instead of its timings. It gives a provisional split for a script
with no timings yet.
balance_chunk_groups.py --merge FILE --skip PREFIX merges a failed
task's timings, leaving out its failing chunks. The timings are in
<logs-dir>/workerW/T/chunk-timings.txt.
Running the test
Run it the way rut does, so the timings reflect real parallelism. Take the
server from .vscode/settings.json (env.main-server, env.mysql-port), as
the rut task in .vscode/tasks.json does. Keep the plan, report and log files
out of bld/bin:
cd bld/bin && MYSQL_PORT=3311 PATH=~/servers/<main-server>/bin:$PATH \
python3 ../../scripts/run_unit_tests.py --binary ./run_unit_tests \
--shell-binary ./mariadb-shell -t ../../unittest/test-execution-times.txt \
--gtest_filter='*<test_name>' --execution-plan-file <tmp>/plan.txt \
--report-file <tmp>/report.html --logs-dir <tmp>/logs
Rebuild run_unit_tests first if the tester has changed. A run takes about as
long as the slowest group; an unsplit test takes its whole length (10+ minutes
for the copy tests), so run it in the background.
Run it against MySQL too, from the MySQL-linked build (bld-mysql/bin)
with a MySQL server first on PATH (~/servers/<version>/bin). Remove every
MariaDB server directory from PATH as well: the runner and the harness pick
the vendor from the first mariadbd/mysqld on PATH, and mariadbd is
looked for first. Pass a -t file of its own, such as
../../unittest/test-execution-times-mysql.txt. Chunk timings are still merged
into the shared test-chunk-times.txt next to it, so a MySQL run overwrites
MariaDB's timings for every chunk it times. Rerun on MariaDB before you
rebalance.
Procedure
- Measure. For a split test, the last
rut run's timings are usually
enough. For an unsplit test, either run it once as it is (fine if it takes a
few minutes), or, for a long test, skip ahead: do the first split with
--estimate, run that split (about 1/N of the time), and rebalance on the
timings it records.
- If a group failed, read its
test-output.log. Failures are often porting
problems that also distort the timings; see Pitfalls.
- Fix what's in scope, then
--merge --skip the failed task's timings.
- Timings are keyed by chunk id, so they stay valid when the number of groups
changes.
- First split (unsplit test only):
- Run
--init. Check that the chunks it left common really are needed in
every group, using --common for extra setup chunks.
- Add the test to
_SPLIT_TEST_GROUPS with N.
- Balance. Run
balance_chunk_groups.py <script> N --explain and read the
output:
- The table shows each group's
current and balanced time, including
the chunks common to every group.
- A slowest group made of one big block means a dependency is holding a
large range together;
--explain names it. Don't move code to break it.
See "Keeping the diff small".
- The cuts list the chunks on either side of every group boundary. Check
each one by reading the code (next step).
- Review every cut by hand. The script only sees Python-level dependencies
(variables and functions defined in an earlier chunk, setup…cleanup runs,
consecutive
BUG#nnn chunks). It can't see state left on disk or in the
server by an earlier chunk, such as a dump directory, a created schema, or a
switched session user. A safe group start begins with its own
EXPECT_SUCCESS/dump, a TEST_*_OPTION, or a wipe and setup of its own.
- For a dependency the script misses, put
# balance: keep-with-previous in the dependent chunk. Add it even when the
current cut is fine, so a later rebalance won't split the pair.
- For a longer run that must stay together, put
# balance: begin-block in
its first chunk and # balance: end-block in its last. The typical case is
prepare user privileges, switch user … reconnect to the user with full privileges: every chunk between them runs as the restricted test user.
dump_tables and dump_instance already carry these markers.
- A chunk reading a variable left over from an earlier chunk (for example a
loop variable) is a real dependency. Define it locally, adding one line,
rather than keeping the whole range together.
- For a false positive, use
# balance: allow-cut-before.
- Also check every chunk that looks at the whole instance, not only the
ones next to a cut: a
dump_instance, a dump or EXPECT_SUCCESS with no
schema filter (None), compare_servers(), snapshot_instance(), or an
asserted total such as "N indexes were built" or "N tables". Such a chunk
depends on what every earlier chunk left on the server, including the
groups it no longer runs after. In file order it may run after a wipe
(wipeout_server(), drop_all_schemas()) that its group no longer
includes, or it may count something another group leaked. See "Hidden
whole-instance state" below.
- Write with
--write, then run the test split. All groups must pass,
on both vendors: run it against MySQL as well as MariaDB (see "Running
the test"). That run also records timings with real parallelism.
- Rebalance only if it's worth it. Rerun the balancer on the fresh timings.
A gain of a few percent on the slowest group is within run-to-run noise and
isn't worth the churn of moving suffixes. About 10% or more is worth it.
After rebalancing, rerun.
- After code changes, run
graphify update ..
Keeping the diff small
These test files are compared with upstream MySQL Shell, so:
- Never move chunks. Group suffixes are expected to change. Moving code
makes the diff unreadable.
- When a dependency welds a big range together, split the defining code off
into a new common chunk: a new
#@<> ... header with no suffix. For
example, dump_tables' WL13804-FR13 - CRC of the test schema tables… adds two
lines and frees the whole range. When the definition sits in the middle of an
expensive chunk, you can instead repeat a cheap assignment in a new common
chunk right after it, which leaves the original untouched; see
dump_instance's targetVersion used by the MySQL HeatWave Service tests below. The new chunk runs in every group, so it must
be cheap, and it must be safe to run on whatever state each group has at that
point.
- A long block held together by session or schema state can often be
split by making its cheap setup and teardown chunks common, instead of
keeping everything between them in one group. Every group then runs that
stretch of the file in the same state as the unsplit run: for example, as the
restricted user between a "switch user" chunk and its "reconnect" chunk.
Then pin only what really shares state. Check the cost first: every group
pays for the common chunks.
- But keep CPU-heavy chunks together. Chunks that start background load
(
Generate_transactions, which creates 500 tables and runs a DDL/DML loop in
another shell) slow each other and every other group down when they run in
parallel groups. The balancer can't model that, because it treats chunk
times as independent. In dump_instance, spreading them over three groups
raised the slowest group from about 300s to 407s, and even groups that hadn't
changed got 10–15% slower. They're now one begin-block…end-block. Timings
recorded while such chunks overlapped are inflated, so rerun with them
serialized before trusting a rebalance.
- An empty
() suffix is not a group. The tester treats it as "run
everywhere". The balancer assigns it a real group.
Hidden whole-instance state
A chunk that dumps, compares or counts the whole instance can pass in file
order only because of something another group did, or because of a leftover
that group never cleaned up. Splitting then breaks it in a way no cut review
catches. Both cases below were found only by a MySQL run: the chunks are
not __server_is_maria_db, so a split verified on MariaDB never ran them.
- A wipe in an earlier group. util_dump_instance's
BUG#35550282 - option is set, schema is not dumped does an ocimds dump of the whole instance.
In file order it runs after the WL14244 helpers have run
wipeout_server() and WL15311 - setup has re-created the test schemas
empty. Its group ran neither, so it still had the deliberately incompatible
mysqlaas fixtures, and the compatibility check failed. Fix: make the
chunk's own setup restore the state it has in file order (here
drop_all_schemas() + create_all_schemas()), rather than pinning it to the
earlier group.
- A leak in an earlier group. util_dump_and_load's
BUG#33414321 - table with a secondary engine asserted that 13 indexes were built for the whole
instance. One of the 13 was wl14506.no_pk's, left on the source server
because WL14506: cleanup ran DROP SCHEMA on the global session, which
EXPECT_PK() had switched to the destination. Fix: fix the leak
(WL14632: cleanup had the same bug), then correct the expectation to the
count without it, so it holds in every order.
What to look for:
- Cleanups that use the global
session after a helper that calls
shell.connect(). They clean the wrong server, and the leftovers show up
later, in one group only. Drop through the named sessions (session1,
session2) instead.
- Hard-coded totals over the whole instance, such as index, table or file
counts. In a split run the total only matches if every group sees the same
instance.
- To confirm a suspected case, run the test unsplit (see Pitfalls for how).
If the chunk passes there and fails in its group, it is this class and not a
product bug.
Pitfalls
- Timings depend on parallelism. Eight workers make each group about 25%
slower than six do, so compare groups within the same run.
- The
rut server may be a debug build (mariadbd --version shows
-debug). It is about 3× slower per row in LOAD DATA, which inflates every
dump/load/copy chunk. The balance still holds; the absolute times don't.
VER(>=8.2.0) passes on MariaDB 12+. MySQL-only chunks guarded only by
version, for example WL15887 HeatWave account checks, then run on MariaDB and
fail. Gate them with and not __server_is_maria_db: the setup, the tests
and the cleanup. If only the cleanup is gated, a debug point such as
copy_utils_force_mds set in the setup stays on and breaks every later chunk
in that group.
copy_utils_force_mds enables the ocimds checks, which MariaDB rejects.
A MariaDB variant of such a test must not set it.
- Flaky under load:
WL15298 - test sessionInitSql option in copy_schemas
has deadlocked once (1213): four loader threads insert into the same table.
Before blaming a new cut for a failure, check whether the chunk depends on
anything before it.
- A baseline must run with no
GROUP_ID at all. The tester runs only
group G's chunks when GROUP_ID is set, even to 0. run_unit_tests.py now
leaves it unset for tasks that aren't split; before that fix, an unsplit run
of a tagged script "passed" while running only its common chunks. To get an
unsplit baseline of a split test, run a copy of run_unit_tests.py with the
test's _SPLIT_TEST_GROUPS line removed, and check that its duration makes
sense.
- Group totals barely include overhead. The .inc files take under 0.1s and
process start/teardown about 1s. The common chunks cost what
Setup costs,
usually a sandbox deployment. run_unit_tests.py builds the sandbox
boilerplate once per run and shares it through
MARIADB_SANDBOX_BOILERPLATE_DIR, so a deployment is a copy, not a
mariadb-install-db; that took copy_schemas' Setup from 27s to 10s per
group. If a test's Setup suddenly costs about 20s more, check that the runner
still printed "Preparing the sandbox boilerplate…".
Reporting
Report to the user:
- the predicted per-group table, and the measured one after the split run;
- the slowest group and what limits it (for example, one indivisible block);
- each marker or new common chunk you added, and why;
- every failing chunk, split into "caused by the split" and "already failing";
- which vendors the split was run on. A split verified on one vendor only has
not run the other vendor's gated chunks.
1---2name: split-scripted-test3description: Split a long scripted test (unittest/scripts/auto/*/scripts/*_norecord.py) into N parallel chunk groups for run_unit_tests.py, or rebalance the groups of a test that is already split, so every group takes about the same time. Use when asked to split, shard, parallelize, balance or rebalance a scripted test, change its number of groups, or find out why one group of a split test is slower than the others.4metadata:5 source: https://skillmd.com/skills/mariadb-corporation/split-scripted-test6---78# Splitting and balancing a scripted test910`scripts/run_unit_tests.py` (the `rut` task) runs a test listed in11`_SPLIT_TEST_GROUPS` as N tasks with `GROUP_ID` = 1..N. A chunk whose header ends12in `(G)` runs only in group G; a chunk with no suffix runs in every group. The13goal is N groups, in file order (group 1 first … group N last), that finish at14the same time. Every group must still pass on its own.1516The mechanics are in `scripts/balance_chunk_groups.py`, and its docstring is17the reference for them. This skill covers the procedure and the judgement calls18that the script can't make.1920## Tools2122- **Timing collection is automatic.** `shell_script_tester.cc` records every23 executed chunk to `$CHUNK_TIMINGS_FILE`. `run_unit_tests.py` sets that variable24 for every task and merges the results of **passing** tasks into25 `test-chunk-times.txt`, next to the `-t` file (`unittest/`). Chunks that run26 in every group are averaged.27- **`balance_chunk_groups.py <script> <N> [--explain] [--write]`** splits the28 grouped chunks into N contiguous groups, minimizing the slowest one. It never29 cuts through a detected dependency. Without `--write` it prints a table and a30 diff.31- **`balance_chunk_groups.py <script> --init`** does the first split of an32 unsplit script. It gives every chunk except the INCLUDEs, `entry point`,33 `Setup` and `Cleanup` (plus any `--common ID`) a `(1)` placeholder for the34 balancing step to renumber.35- **`balance_chunk_groups.py <script> <N> --estimate`** balances by each chunk's36 line count instead of its timings. It gives a provisional split for a script37 with no timings yet.38- **`balance_chunk_groups.py --merge FILE --skip PREFIX`** merges a **failed**39 task's timings, leaving out its failing chunks. The timings are in40 `<logs-dir>/workerW/T/chunk-timings.txt`.4142## Running the test4344Run it the way `rut` does, so the timings reflect real parallelism. Take the45server from `.vscode/settings.json` (`env.main-server`, `env.mysql-port`), as46the `rut` task in `.vscode/tasks.json` does. Keep the plan, report and log files47out of `bld/bin`:4849```sh50cd bld/bin && MYSQL_PORT=3311 PATH=~/servers/<main-server>/bin:$PATH \51 python3 ../../scripts/run_unit_tests.py --binary ./run_unit_tests \52 --shell-binary ./mariadb-shell -t ../../unittest/test-execution-times.txt \53 --gtest_filter='*<test_name>' --execution-plan-file <tmp>/plan.txt \54 --report-file <tmp>/report.html --logs-dir <tmp>/logs55```5657Rebuild `run_unit_tests` first if the tester has changed. A run takes about as58long as the slowest group; an unsplit test takes its whole length (10+ minutes59for the copy tests), so run it in the background.6061**Run it against MySQL too**, from the MySQL-linked build (`bld-mysql/bin`)62with a MySQL server first on `PATH` (`~/servers/<version>/bin`). Remove every63MariaDB server directory from `PATH` as well: the runner and the harness pick64the vendor from the first `mariadbd`/`mysqld` on `PATH`, and `mariadbd` is65looked for first. Pass a `-t` file of its own, such as66`../../unittest/test-execution-times-mysql.txt`. Chunk timings are still merged67into the shared `test-chunk-times.txt` next to it, so a MySQL run overwrites68MariaDB's timings for every chunk it times. Rerun on MariaDB before you69rebalance.7071## Procedure72731. **Measure.** For a split test, the last `rut` run's timings are usually74 enough. For an unsplit test, either run it once as it is (fine if it takes a75 few minutes), or, for a long test, skip ahead: do the first split with76 `--estimate`, run that split (about 1/N of the time), and rebalance on the77 timings it records.78 - If a group failed, read its `test-output.log`. Failures are often porting79 problems that also distort the timings; see Pitfalls.80 - Fix what's in scope, then `--merge --skip` the failed task's timings.81 - Timings are keyed by chunk id, so they stay valid when the number of groups82 changes.832. **First split** (unsplit test only):84 - Run `--init`. Check that the chunks it left common really are needed in85 every group, using `--common` for extra setup chunks.86 - Add the test to `_SPLIT_TEST_GROUPS` with N.873. **Balance.** Run `balance_chunk_groups.py <script> N --explain` and read the88 output:89 - **The table** shows each group's `current` and `balanced` time, including90 the chunks common to every group.91 - **A slowest group made of one big block** means a dependency is holding a92 large range together; `--explain` names it. Don't move code to break it.93 See "Keeping the diff small".94 - **The cuts** list the chunks on either side of every group boundary. Check95 each one by reading the code (next step).964. **Review every cut by hand.** The script only sees Python-level dependencies97 (variables and functions defined in an earlier chunk, setup…cleanup runs,98 consecutive `BUG#nnn` chunks). It can't see state left on disk or in the99 server by an earlier chunk, such as a dump directory, a created schema, or a100 switched session user. A safe group start begins with its own101 `EXPECT_SUCCESS`/dump, a `TEST_*_OPTION`, or a wipe and setup of its own.102 - For a dependency the script misses, put103 `# balance: keep-with-previous` in the dependent chunk. Add it even when the104 current cut is fine, so a later rebalance won't split the pair.105 - For a longer run that must stay together, put `# balance: begin-block` in106 its first chunk and `# balance: end-block` in its last. The typical case is107 `prepare user privileges, switch user` … `reconnect to the user with full108 privileges`: every chunk between them runs as the restricted test user.109 dump_tables and dump_instance already carry these markers.110 - A chunk reading a variable left over from an earlier chunk (for example a111 loop variable) is a real dependency. Define it locally, adding one line,112 rather than keeping the whole range together.113 - For a false positive, use `# balance: allow-cut-before`.114 - **Also check every chunk that looks at the whole instance**, not only the115 ones next to a cut: a `dump_instance`, a dump or `EXPECT_SUCCESS` with no116 schema filter (`None`), `compare_servers()`, `snapshot_instance()`, or an117 asserted total such as "N indexes were built" or "N tables". Such a chunk118 depends on what *every* earlier chunk left on the server, including the119 groups it no longer runs after. In file order it may run after a wipe120 (`wipeout_server()`, `drop_all_schemas()`) that its group no longer121 includes, or it may count something another group leaked. See "Hidden122 whole-instance state" below.1235. **Write** with `--write`, then run the test split. All groups must pass,124 **on both vendors**: run it against MySQL as well as MariaDB (see "Running125 the test"). That run also records timings with real parallelism.1266. **Rebalance only if it's worth it.** Rerun the balancer on the fresh timings.127 A gain of a few percent on the slowest group is within run-to-run noise and128 isn't worth the churn of moving suffixes. About 10% or more is worth it.129 After rebalancing, rerun.1307. After code changes, run `graphify update .`.131132## Keeping the diff small133134These test files are compared with upstream MySQL Shell, so:135136- **Never move chunks.** Group suffixes are expected to change. Moving code137 makes the diff unreadable.138- **When a dependency welds a big range together**, split the defining code off139 into a **new common chunk**: a new `#@<> ...` header with no suffix. For140 example, dump_tables' `WL13804-FR13 - CRC of the test schema tables…` adds two141 lines and frees the whole range. When the definition sits in the middle of an142 expensive chunk, you can instead repeat a cheap assignment in a new common143 chunk right **after** it, which leaves the original untouched; see144 dump_instance's `targetVersion used by the MySQL HeatWave Service tests145 below`. The new chunk runs in every group, so it must146 be cheap, and it must be safe to run on whatever state each group has at that147 point.148- **A long block held together by session or schema state** can often be149 split by making its **cheap** setup and teardown chunks common, instead of150 keeping everything between them in one group. Every group then runs that151 stretch of the file in the same state as the unsplit run: for example, as the152 restricted user between a "switch user" chunk and its "reconnect" chunk.153 Then pin only what really shares state. Check the cost first: every group154 pays for the common chunks.155- **But keep CPU-heavy chunks together.** Chunks that start background load156 (`Generate_transactions`, which creates 500 tables and runs a DDL/DML loop in157 another shell) slow each other and every other group down when they run in158 parallel groups. The balancer can't model that, because it treats chunk159 times as independent. In dump_instance, spreading them over three groups160 raised the slowest group from about 300s to 407s, and even groups that hadn't161 changed got 10–15% slower. They're now one `begin-block`…`end-block`. Timings162 recorded while such chunks overlapped are inflated, so rerun with them163 serialized before trusting a rebalance.164- **An empty `()` suffix is not a group.** The tester treats it as "run165 everywhere". The balancer assigns it a real group.166167## Hidden whole-instance state168169A chunk that dumps, compares or counts the whole instance can pass in file170order only because of something another group did, or because of a leftover171that group never cleaned up. Splitting then breaks it in a way no cut review172catches. Both cases below were found only by a MySQL run: the chunks are173`not __server_is_maria_db`, so a split verified on MariaDB never ran them.174175- **A wipe in an earlier group.** util_dump_instance's `BUG#35550282 - option176 is set, schema is not dumped` does an `ocimds` dump of the whole instance.177 In file order it runs after the `WL14244` helpers have run178 `wipeout_server()` and `WL15311 - setup` has re-created the test schemas179 empty. Its group ran neither, so it still had the deliberately incompatible180 `mysqlaas` fixtures, and the compatibility check failed. **Fix:** make the181 chunk's own setup restore the state it has in file order (here182 `drop_all_schemas()` + `create_all_schemas()`), rather than pinning it to the183 earlier group.184- **A leak in an earlier group.** util_dump_and_load's `BUG#33414321 - table185 with a secondary engine` asserted that 13 indexes were built for the whole186 instance. One of the 13 was `wl14506.no_pk`'s, left on the source server187 because `WL14506: cleanup` ran `DROP SCHEMA` on the global `session`, which188 `EXPECT_PK()` had switched to the destination. **Fix:** fix the leak189 (`WL14632: cleanup` had the same bug), then correct the expectation to the190 count without it, so it holds in every order.191192What to look for:193194- **Cleanups that use the global `session`** after a helper that calls195 `shell.connect()`. They clean the wrong server, and the leftovers show up196 later, in one group only. Drop through the named sessions (`session1`,197 `session2`) instead.198- **Hard-coded totals over the whole instance**, such as index, table or file199 counts. In a split run the total only matches if every group sees the same200 instance.201- **To confirm a suspected case**, run the test unsplit (see Pitfalls for how).202 If the chunk passes there and fails in its group, it is this class and not a203 product bug.204205## Pitfalls206207- **Timings depend on parallelism.** Eight workers make each group about 25%208 slower than six do, so compare groups within the same run.209- **The `rut` server may be a debug build** (`mariadbd --version` shows210 `-debug`). It is about 3× slower per row in `LOAD DATA`, which inflates every211 dump/load/copy chunk. The balance still holds; the absolute times don't.212- **`VER(>=8.2.0)` passes on MariaDB 12+.** MySQL-only chunks guarded only by213 version, for example `WL15887` HeatWave account checks, then run on MariaDB and214 fail. Gate them with `and not __server_is_maria_db`: the setup, the tests215 **and** the cleanup. If only the cleanup is gated, a debug point such as216 `copy_utils_force_mds` set in the setup stays on and breaks every later chunk217 in that group.218- **`copy_utils_force_mds` enables the `ocimds` checks**, which MariaDB rejects.219 A MariaDB variant of such a test must not set it.220- **Flaky under load:** `WL15298 - test sessionInitSql option` in copy_schemas221 has deadlocked once (1213): four loader threads insert into the same table.222 Before blaming a new cut for a failure, check whether the chunk depends on223 anything before it.224- **A baseline must run with no `GROUP_ID` at all.** The tester runs only225 group G's chunks when `GROUP_ID` is set, even to `0`. `run_unit_tests.py` now226 leaves it unset for tasks that aren't split; before that fix, an unsplit run227 of a tagged script "passed" while running only its common chunks. To get an228 unsplit baseline of a split test, run a copy of `run_unit_tests.py` with the229 test's `_SPLIT_TEST_GROUPS` line removed, and check that its duration makes230 sense.231- **Group totals barely include overhead.** The .inc files take under 0.1s and232 process start/teardown about 1s. The common chunks cost what `Setup` costs,233 usually a sandbox deployment. `run_unit_tests.py` builds the sandbox234 boilerplate once per run and shares it through235 `MARIADB_SANDBOX_BOILERPLATE_DIR`, so a deployment is a copy, not a236 `mariadb-install-db`; that took copy_schemas' `Setup` from 27s to 10s per237 group. If a test's Setup suddenly costs about 20s more, check that the runner238 still printed "Preparing the sandbox boilerplate…".239240## Reporting241242Report to the user:243- the predicted per-group table, and the measured one after the split run;244- the slowest group and what limits it (for example, one indivisible block);245- each marker or new common chunk you added, and why;246- every failing chunk, split into "caused by the split" and "already failing";247- which vendors the split was run on. A split verified on one vendor only has248 not run the other vendor's gated chunks.