jitcatch
jitcatch never calls a model. You write the tests. You decide whether a failure is a regression. The CLI prepares a brief, runs each test on the parent and on the change, and applies deterministic rules.
Do not commit the candidate tests unless the human asks to promote one.
Prerequisites
Ruby 3.3 or newer, and git. No gems. minitest and prism ship with Ruby. RSpec candidates need rspec-core: in the project's bundle when it has a Gemfile, otherwise installed for the Ruby that runs jitcatch. The subject must be a git repository that can add worktrees. The tool reads commits only. Uncommitted files are not in the diff.
gem install jitcatch installs the jitcatch command, with no runtime dependency. In an app with a Gemfile, such as a Rails app, the gem can sit in the :development, :test group instead (gem "jitcatch", require: false); run it as bundle exec jitcatch. Each side still loads its own bundle, so the line does not need to be in either commit. From a checkout of the jitcatch repository, put its bin directory on PATH, or run ruby /path/to/jitcatch/bin/jitcatch. jitcatch --help prints the commands, and jitcatch --version prints jitcatch 0.2.1, with the commit after it (jitcatch 0.2.1 (1fbcd18)) when jitcatch runs from its own git checkout; quote that line with any output you quote. The version line of every report and version in --json hold the number only.
Exit status 0 is success, including when catching tests were found. Exit status 1 is a usage error. Exit status 2 is a git error. Exit status 3 means run tested nothing: a side failed its setup check before any test ran, no candidate file loaded on either side, or every row timed out, or found no database or no table, on a side (below).
Choose base and head
BASE is the parent whose behavior should survive: the branch you would land onto, or git merge-base of that branch and the change. HEAD is the revision under review (a branch, HEAD, or a SHA). Both must resolve as git rev-parse --verify REF^{commit}. One exception: to review the first commit of a repository, pass ROOT^ (or ROOT~1) as BASE. The parent of a root commit is read as the empty tree, so every file is added. On run, the base side is an empty directory, so every test is fail_on_base.
When BASE is not an ancestor of HEAD (for example, a main branch that moved on after the change branched off), the diff also undoes the commits only BASE has, and their methods show up as changed. context and run print warning: base is not an ancestor of head under head, with the count and the merge base. Pass that merge base as BASE and run again. When BASE and HEAD resolve to the same commit, they print warning: base and head are the same commit, and every row is no_signal or fail_on_base.
Pass --repo DIR when that repository is not the working directory.
In a monorepo whose gem sits in a directory with its own lib/ and Gemfile, such as ruby/, run every command from that directory (cd ruby && jitcatch run BASE HEAD catch), or pass --repo ruby from the top. Candidates, the bundle check, --setup, and each rerun: line then run in ruby/ of each checkout, with -Ilib as ruby/lib and BUNDLE_GEMFILE as ruby/Gemfile, so a candidate needs no $LOAD_PATH or GEM_PATH workaround. Put TESTS_DIR under that directory, and run each rerun line from there. The report prints directory ruby under head. File paths in the change, the methods, and the frames stay from the repository root (ruby/lib/shelf.rb).
Commit messages from BASE to HEAD are the intent the CLI can see (git log BASE..HEAD, oldest first, commits separated by ---). The CLI leaves out the trailer block that git parses at the end of each message (Co-Authored-By:, Signed-off-by:, and the like). Git counts a last paragraph made only of Word: text lines as trailers too, so a closing line such as Note: callers must refresh. is also left out. When the intent looks cut short, read git log BASE..HEAD. Also read the pull request or the task when you have it. The tool does not.
Context
jitcatch context --repo DIR BASE HEAD
jitcatch context --json --repo DIR BASE HEAD
jitcatch context --repo DIR --coverage FILE BASE HEAD
jitcatch context --repo DIR --all-readers BASE HEAD
Omit --repo when the working directory is the repository.
Read intent: first. Then the changed files, dependencies: (each gem whose Gemfile.lock version changed, as gem old -> new, shown only when one did), constants (k01, …), and methods (m01, …). Each method has a change kind, signature, visibility, call sites, reachable_from, existing tests, and the old and new source.
reachable_from goes through constructors: a caller of Runner.new reaches Runner#initialize and the methods it calls, and that hop does not count toward the four calls reachable_from follows. reachable_from keeps the eight nearest. run reads reached_by (--json), every caller at any depth, so a test that enters high (a request spec through a controller and a service, or a CLI test of CLI.start) still reaches a changed method five or more calls down; its row is usually related:guess with that method in the list.
Change kinds:
moved: the method left one file while head defines it in another, test files included. The line shows its new place andmoved_from=the old file, and a call to it is not a call to a removed method. A constant that moves to another file with the same value ismovedwithmoved_from=as well, and its readers are notaffected; with a new value it ismodified, withmoved_from=.affected: the method did not change its body. It reads a constant that did (via=k01).- A class or module that only one side defines anywhere is listed as an
addedorremovedconstant, with its header line as the source. A file that reopens an existing class adds nothing.
What counts as a method, in any Ruby:
- Methods defined in a block given to a constant (
Point = Struct.new(:x) do,Class.new do,Data.define do) are listed under that constant (Point#norm), and the constant's source is the text beforedo. - A reader or writer that
attr_reader,attr_writer,attr_accessor, or aStruct.new/Data.definemember list defines is a method too (Point#x,Point#x=), with the one-name form as its source (attr_reader :seed,Struct.new(:x)). A name dropped from that line isremoved, the names left on it are unchanged, and aNoMethodErroron the dropped name, called from the test, isdiscard:signature_change.Struct.new(*FIELDS)andattr_reader(*FIELDS)count when the same file setFIELDSto a list of symbols earlier; a list built at run time or in another file gives no members. - A
defor a reader inside a conditional of the class body (attr_reader :x if cond,if,unless,else) counts, read as if every branch ran in order, so a bareprivatein a branch carries past the conditional.
Rails and Grape definitions
- On an Active Record model (a class whose superclass chain in the repository reaches
ActiveRecord::Base), a caller ofcreate,create!, orbuildreachesinitializetoo, as a caller ofnewdoes. On another class those three names reach nothing. scope :recent, -> { ... }is the class methodPost.recentwith the whole statement as its source.delegate :name, to: :authorisPost#name(Post#author_namewithprefix: true).enum :status, { draft: 0 }gives each valuePost#draft?,Post#draft!,Post.draft, andPost.not_draft, with that one value as the source.- A definition counts too in an
ActiveSupport::Concernblock:included doandprepended dodefine instance methods of the module,class_methods doits class methods. - A request to a path helper (
post things_path) reaches the controller actionconfig/routes.rbdraws for it, with the helper and controller names the app'sconfig/initializers/inflections.rbgives (person_pathforresources :people,APIKeysControllerwithinflect.acronym "API"). So a changed action'stests:lists the files whose requests reach it (via POST things_path) and the controller tests that call it aspost :create, not a grep of its name. A helper the routes file does not draw gets awarning:. - A Grape endpoint (
post "/refresh" doinsideresource :things do, in a class underGrape::API) isThings#POST /things/refresh. A request spec reaches it by sending that verb to a path ending in those segments::idmatches any one, an interpolation in the test matches any one segment, and a string constant the file assigns reads as its string. An endpoint path the source computes otherwise is named with that part in braces and gets awarning:.
Visibility, test files, and executables
visibility= follows private, protected, and public (bare, with names, or before a def), the same inside class << self, and private_class_method / public_class_method (with names or before def self.x). A private method is not an entry point. Test it through a public caller in reachable_from. A method after module_function, or named by it, is listed as the public module method Mod.name. A method of a class or module that private_constant names in the same file, or of anything nested in it, is listed as private, because a test cannot name its receiver. Visibility set by send, or by private :name written above its def, is not seen.
Methods in test files (under test/ or spec/, or named test_*.rb or *_test.rb) are not listed, and are not entry points. The changed test files still appear under files:.
An executable under exe/ or bin/ with no suffix and a ruby line after #! is read as Ruby. Its code outside a def, a class, or a module is one method, exe/tool <main> (kind: "script" in --json), so a change to its usage text or to the case that picks a command is listed with its old and new source. Its tests: are the test files that write its path, and a candidate that starts it (IO.popen(["ruby", "exe/tool"])) mentions it, so a row can name it as its related id.
Tests, entry points, and readers
tests: (none) means no test file names that method. Then entry_point_tests: lists, when it finds any, the test files that call a public reachable_from entry point: the file names the entry point's class (Cart.new, not only Cart::LIMIT) and calls a method of that name (.total). Each file shows the entry points it matched. Read those tests before you write yours. They show how the repository builds the object and which inputs already reach the changed method. Like tests:, this line is a grep. It misses tests that drive the code through a subprocess, and it can list a file that calls a same-named method on another object. A tests: grep hit on a common name such as call can be any file that defines or calls that name. For a reader or member from attr_reader or Data.define (Targets::Hit#code), a hit counts only in a file that also names its class or the namespace around it (Hit or Targets). The code of a fixture project under test/ or spec/ (test/fixture/depot/app/, lib/, config/, db/) is not a test file there.
In a Rails app, a changed initialize shows built_by: above tests:: create, create!, and build on an Active Record model, and build(:loan) and create(:loan) for each factory under spec/factories or test/factories that builds the class. Its tests: lists only the test files that call Owner.new or one of those, as spec/models/loan_spec.rb via create(:loan), and no grep hit for the word initialize. factories_skipped: lists each factory whose class the source does not settle (a class: computed at run time, an unknown parent, or a class no file defines, as an inflection such as APIKey causes). A factory that builds a Ruby value (class: Hash, Array, Struct, OpenStruct, and the like) is left out and not listed. A test that builds from one of those names no constructor. A factory also counts for the classes its associations build (borrower, association :borrower, or association in an attribute block), so a changed Borrower callback lists create(:loan) when :loan builds a borrower. An association in a trait lists the call with the trait, create(:loan, :cosigned), and a test that writes create(:loan, :cosigned) mentions it; a global trait and a trait that includes another are not followed.
A method that a model's class body names in a callback or a validation (after_save :count_save, or if: :checked? on a validates) shows run_by:: the calls that run it by Rails's rules (Loan.create!, Loan#save, a factory of the class), with the lines that name it. An on: :create validation lists no update, and an on: :update one no create. build(:loan) does not save, so it counts only in a test that also saves or validates (.save!, be_valid). Its tests: lists the test files that make those calls, next to the grep hits. A method of the app that makes one of those calls (LoanDesk.open calling Loan.create!) is in reachable_from, with its callers, so a test of LoanDesk.open names the callback.
A save on a parameter, a local, or an instance variable the source does not type counts for the model it is named after: loan.save! for Loan, but record.save! for none. Any other call on such a value, and on a belongs_to or has_one reader (borrower.approved_cents inside Loan), is a call edge to that model's method too, so reachable_from follows @loan.approve in a service up to its controller. A callback in a Concern or a parent class is not followed.
The brief ends with one entrypoints <Const>: section for each class or module that holds a changed method. Each line is a public method of that class on head (on base when head has none) and its role: the change kind (modified, added, affected, removed) first, then unchanged, at most 12 lines. An unchanged method in a changed method's reachable_from comes before the other unchanged ones, and (+3 more) counts the methods the cap left out. The cap only shortens the list: a method left out still counts as a mention through reachable_from. A class that defines initialize on either side lists Owner.new first in its group instead, with the role of Owner#initialize: Mismatch.new removed means head dropped the constructor and .new now runs the inherited one. A class that makes new private (private_class_method :new) shows no Owner.new; build it through the public method that calls new. Pick the methods your tests call from this list. run counts these names as mentions too, so calling one gives a row its touches:unchanged_caller tag when it reaches a changed method.
Read readers: before you pick what to test, in any Rails change, and always when a changed method says callsites: (none by name; 3 under readers:). It lists, per changed value (a method, a scope, a default_scope, a constant, a callback, a validation, a column default in db/schema.rb or attribute, a config. setting), the code at head that reads that value without calling it by name. A call in an ERB template is a call site, under callsites:, so it is not repeated here. A real run, on a change that adds Branch#to_param returning code:
readers: 1 for 1 changed value
Branch#to_param (added) method m01 aac7821:app/models/branch.rb:4 readers=1
100 route_param_mismatch aac7821:app/views/branches/show.html.erb:2
rank: the URL value and the lookup now disagree
why: @branch goes through Branch#to_param (now `code`) into params[:branch_id]; ClosuresController#index reads params[:branch_id] by id (Branch.find(params[:branch_id]), app/controllers/closures_controller.rb:11)
test: Render app/views/branches/show.html.erb:2 with a Branch whose to_param differs from its id, follow the URL branch_closures_path(@branch) builds, and check ClosuresController#index finds that same Branch.
not seen: receivers named unlike their model (`x = Branch.first; link_to x`); ...
For each row, from the top: open the place (git show aac7821:app/views/branches/show.html.erb), read why: against the intent, and write the candidate test: describes. It states what the old code satisfied, so it passes on base: build the record so the old and the new value differ (a code that is not the id), and drive the entry point the row names (render the view or send the request, then follow the URL). A row whose effect the intent asks for still gets a test: its reported result is then intended. rank: says why a row scores what it does; a low row (bare STATUSES is defined nowhere visible, in a file the change edits) is the first to skip. also: lists more places of the same kind for the same value, and one test usually covers them. note: this line is itself part of BASE..HEAD marks a row the author edited, so it was in front of them. The 10 highest rows print, and the first line counts the rest; --all-readers prints every one. A row finds a record's model by the name of the variable that holds it (@branch, home_branch, branches). not seen: names what no row can show: a reader there needs a git grep of your own. In a history with no app/, config/routes.rb, or config/application.rb, such as a gem's, it names only run-time dispatch names and config read at load time. A constant's read_by: lists the changed methods that read it by name, and a plain read inside one of those is not repeated under readers:.
--coverage FILE replaces the grep of existing tests. The file is JSON: "format": "jitcatch-coverage-v1" and a tests array of { "id", "file", "lines" }, where lines maps repo paths to line numbers. Without it, the brief says existing_tests grep.
--json prints the same brief ("command": "context").
Write candidate tests
Write 3 to 6 candidate files in a scratch directory you can delete. Use the framework the project's own tests use: Minitest files, or RSpec specs named *_spec.rb (see RSpec candidates below). One directory can hold both.
A catching test states a behavior the old code already satisfies. Aim it at a changed or affected method, and at unchanged callers that reach that method (reachable_from). Prefer a boundary, a rounding case, or an error path the diff could disturb. One claim per test method. A test that only describes the new feature fails on base and is not evidence. Assert on one field, not on a whole returned Hash: a whole-Hash test (such as a branchout characterization test) fails on any added key, even one the intent asked for.
To use branchout characterization tests as candidates, run branchout explore on a checkout of BASE, not of HEAD: a test emitted on head pins the new behavior and is fail_on_base. Pass --require-by-name, so the emitted test requires the target through the load path (require "shop/cart") and each side loads its own worktree's code. Pass --format rspec for an RSpec project. Without --require-by-name, check where the emitted require_relative or absolute path points. A path that resolves inside the worktree is fine as written: tests emitted under .jitcatch/ of the base checkout keep require_relative "../lib/legacy_register.rb", which each side resolves in its own worktree. A path outside the worktree makes both sides load the same file, so every row is no_signal; rewrite it to require "<lib>". On jitcatch itself, 14 tests emitted on head with an outside path gave 14 no_signal; emitted on base with the require rewritten, they gave 8 reported.
The mentions and the touches: tags read the test method's body and the bodies of the methods in the same file that it calls, directly or through other such methods. A helper in another file (require "test/support/...") is not read, so call the changed method or its public caller from the candidate file. Prefer a public entry point over send(:private_helper): a private helper can move between base and head, and the test then fails on base.
Each file:
require "minitest/autorun", then require the library the way the repo loads it (require "shop"whenlib/is on the load path). The worktree root is on the load path too, so a test can reuse the repository's own helpers by path from the root (require "test/support/repo"). When you start from a copy of a repository test, rewriterequire_relative "../test_helper"torequire "test/test_helper": the copy runs from.jitcatch/, where the relative path points at the worktree root and fails withLoadErroron both sides.- Name it
*_test.rbortest_*.rb. When any file matches, other*.rbfiles are not executed. At most 40 files run. - The runner copies the directory into each worktree at
.jitcatch/and runs one process per file:ruby -Ilib -I. -r<jitcatch>/lib/jitcatch/minitest_report.rb .jitcatch/<file> --seed 1. That file writes each result to.jitcatch-report.json, so a row names one test whatever the test prints and underparallelize_me!. A*_spec.rbfile runs through RSpec instead (below). - When the worktree has a
Gemfile, the process also gets-rbundler/setupandBUNDLE_GEMFILEset to that worktree'sGemfile. Base and head then each load the gem versions their ownGemfile.locknames, so a test can catch a dependency bump. Only gems in the bundle load. When theGemfiledoes not listminitest, no Minitest candidate can load it, andrunstops withsetup base failedandminitest is not in the bundle. For RSpec candidates the check asks forrspec-coreinstead. Do not addrequire "bundler/setup"to the file yourself. - Do not depend on the original path, on order, or on the clock. The base run and the head run of a file run at the same time, each in its own worktree, so do not write a fixed path outside the worktree (use
Dir.mktmpdir) or listen on a fixed port. When a candidate must, pass--serial, which runs one process at a time. The child keeps onlyPATH,HOME,USER,LOGNAME,TMPDIR,TEMP,TMP,LD_LIBRARY_PATH,GEM_HOME, andGEM_PATH, and setsLANGandLC_ALLtoC.UTF-8andTERMtodumb. Other variables, such asRUBYOPTfrombundle execorCI, do not reach it. When the library reads a variable of its own (for exampleQUINT, the command it runs), pass it with--env QUINT. Without it, the library falls back to its default, which can time out on both sides. - A failure message shows the root of the worktree as
{worktree}({worktree}/lib/cart.rb, and{worktree}forDir.pwdalone) and Ruby's prefix as{ruby}/, so an absolute path is still visibly a path. Where it named the copy at{worktree}/.jitcatch, it namesTESTS_DIRas the rerun line does ({worktree}/candidates/fixtures/a.json, or{worktree}/../catch/tests/fixtures/a.jsonoutside the repository), and a backtrace frame names it without{worktree}/(candidates/cart_test.rb), so a file a test reads by__dir__is named where you edit it. A relative.jitcatch/, such as__FILE__, stays as printed, except at the start of a line in Ruby's report of a file that fails to load (.jitcatch/x_test.rb:2:in), which names the file as the rerun line does. An absoluteTESTS_DIRgets no{worktree}/. The worktrees sit underDir.tmpdir, so a claim such asrefute_includes File.expand_path(__FILE__), Dir.tmpdirfails on both sides. A failure message may name a path under the temp directory. The rerun check compares the whole message. It masks the name directly underDir.tmpdir, an object address (#<Class:0x00007f...>), and a time of day with its date (a record printed byhave_attributesshowscreated_at: "2026-11-04 19:02:03.212197000 +0000"), so a newDir.mktmpdirname, a new anonymous class, or the time a record was saved is not a disagreement. When a rerun fails the same way (same status and exception class) and only the message moved, such as a pid, a random record id, or a value that moves between runs, base runs the test--rerunstimes too. If base passes each time, the row stays a candidate with the tagmessage_varies; if not, it isflaky, and the rerun line readsrerun 1: fail Minitest::Assertion (same outcome, message differs) <first line that moved>, followed bybase rerun 1:lines. - A file that builds a repository, starts processes, or runs a slow fixture needs more than the default 10 seconds. Raise
--timeout. A timed-out file is one row (<file>#timeout). A timeout on both sides, or on head beside a base that took more than half of--timeout, isdiscard:timeout; a timeout on base with a passing head isfail_on_base. A timeout on head beside a base that passed in at most half of it is reported with the taghead_only_timeoutand the hinthint: base passed in 0.1s; head did not finish in 5s, so head may loop or wait forever: read the changed code for a loop or a wait whose end the test stubs, such as a deadline checked against a frozen clock. The text report prints ahint:under that row, such ashint: base and head timed out after 10s; raise --timeout if the file needs longer; when head hasconfig/application.rb("rails_app": truein--json), it adds, as a Rails app's first run in a new checkout can (it loads db/schema.rb). A Rails app boots in each new checkout jitcatch makes, andmaintain_test_schema!loads the schema into that checkout's database, so when head hasconfig/application.rbthe default is 60 seconds. When a row timed out, its hint first names the variables your shell sets that the child did not get and is likely to wait without: a network setting (HTTPS_PROXY,NO_PROXY,SSL_CERT_FILE,NODE_EXTRA_CA_CERTS, and the like) when head or a candidate loads an HTTP client such asnet/httporfaraday(by arequireor aGemfile.lockentry) or starts a command that fetches (npx --yes ...,curl ...,bundle install,gem install,git clone), or a name head's Ruby files or a candidate read fromENVwith a literal (hint: base and head timed out after 10s; jitcatch did not pass HTTPS_PROXY QUINT, so add --env HTTPS_PROXY --env QUINT if the file needs them, then raise --timeout if it needs longer;env_not_passedin--json). A name--envpassed that your shell does not set comes before those (--env QUINT passed nothing, because QUINT is not set, so set it if the file needs it). Pass the ones the file needs with--envbefore you raise--timeoutfurther. A command that a script file or a computed string runs is not seen, so a file that fetches that way gets no proxy names; pass them yourself.
RSpec candidates
A file named *_spec.rb runs with the project's RSpec, one process per file, from the worktree root:
ruby -Ilib -I. -e 'require "rspec/core"; require "rspec/support/object_formatter"; RSpec::Core::RubyProject.instance_variable_set(:@project_root, Dir.pwd); RSpec::Support::ObjectFormatter.default_instance.max_formatted_output_length = nil; exit(RSpec::Core::Runner.run(ARGV))' -- \
--format json --out .jitcatch-report.json --seed 1 --backtrace --no-color .jitcatch/<file>_spec.rb
- Start the file with
require "spec_helper"orrequire "rails_helper", as the project's specs do. RSpec putsspec/andlib/on the load path and reads the worktree's.rspec, so its--require spec_helperalso loads. The launch pins RSpec's project root to the worktree, so those are the worktree's ownspec/andlib/even whenTMPDIRis under another project that holdsspec/. Do not writerequire_relative "spec_helper": the copy runs from.jitcatch/, where that path does not exist. - A
*_spec.rbfile that requiresminitest(minitest/spec) runs as Minitest. - One example is one row, named by its full description. The text report shows the example's line after the file (
message_lines_spec.rb:31). Two examples with the same description get(line N)on the later one. expect(...).tofailures arefail(tagassertion), any other error iserror,pendingisskip. An error while RSpec loads the file is one<file>#loadrow.raise_errorthat caught an arityArgumentErrorcounts as a signature change, asassert_raisesdoes.- An example's mentions come from its own block, the
let,subject, andbefore/after/aroundblocks of the groups around it, those groups' arguments, and the same-file helpers they call.describe "#total"anddescribe ".build"name that method.described_class.newnamesOwner.new, anddescribed_class.parseorIndex.parsepicks that class'sparseamong changed methods of the name. Aletorsubjectname is the spec's own helper, not a mention. When its block builds one class, a call on it is a call on that class: withlet(:cart) { Cart.new }orsubject(:cart) { described_class.new },cart.totalnamesCart#total. An implicitsubjectandspec/supporthelpers are not read, so call the changed method in the example or itslet. - A Rails
rails_helperloads in each worktree. Read Rails apps below before you write candidates that touch the database.
On jitcatch itself, across the change that drops the ---Backtrace--- lines from an assert_raises message, three specs that branchout explore --format rspec --require-by-name emitted on base for JitCatch::MinitestParser.parse, in candidates/backtrace/ of a clone, gave:
tests:
t01 no_signal JitCatch::MinitestParser.parse returns summary: false when stdout is "" minitest_parse_spec.rb:11
t02 no_signal JitCatch::MinitestParser.parse returns summary: true when stdout is "T#test_a = 0.00 s = F\n..." minitest_parse_spec.rb:21
t03 reported JitCatch::MinitestParser.parse returns summary: true, status: "fail" when stdout is "ArTest#test_raises = 0..." minitest_parse_spec.rb:16 m02
counts generated=3 no_signal=2 fail_on_base=0 candidates=1 flaky=0 discarded=0 reported=1
assess:
c01 JitCatch::MinitestParser.parse returns summary: true, status: "fail" when stdout is "ArTest#test_raises = 0..." base=pass head=fail m02 assertion,report,touches:unchanged_caller
hash keys differ: "tests"
-...ror>\nMessage: <\"wrapped k\">\n---Backtrace---\n------------...
+...ror>\nMessage: <\"wrapped k\">", "message_first_line" => "[Ke...
^ column 301
rerun: bundle exec ruby -Ilib -I. -e 'require "rspec/core"; require "rspec/support/object_formatter"; RSpec::Core::RubyProject.instance_variable_set(:@project_root, Dir.pwd); RSpec::Support::ObjectFormatter.default_instance.max_formatted_output_length = nil; exit(RSpec::Core::Runner.run(ARGV))' candidates/backtrace/minitest_parse_spec.rb:16
That row is intended: the message no longer ends in the two lines that framed the frames. m02 is the private MinitestParser#parse_issue, and the spec reaches it through parse, an unchanged caller.
Across the commit that pinned the project root, with rspec-core added to both sides of a clone, two hand-written specs next to two that branchout emitted for JitCatch::Assessor.apply, in candidates/root/, gave this row. The emitted example for b_spec.rb already fails on base, so it is fail_on_base:
counts generated=4 no_signal=2 fail_on_base=1 candidates=1 flaky=0 discarded=0 reported=1
assess:
c01 JitCatch::Framework::RSpec reruns a row with rspec base=pass head=fail m02 assertion,report,touches:changed_method
-"rspec {tests}/a_spec.rb:7"
+"ruby -Ilib -I. -e 'require \"rspec/core\"; RSpec::Core::Ruby...
^ column 3
rerun: bundle exec ruby -Ilib -I. -e 'require "rspec/core"; require "rspec/support/object_formatter"; RSpec::Core::RubyProject.instance_variable_set(:@project_root, Dir.pwd); RSpec::Support::ObjectFormatter.default_instance.max_formatted_output_length = nil; exit(RSpec::Core::Runner.run(ARGV))' candidates/root/rerun_spec.rb:4
Not supported yet: shared example groups (an example from a group in another file reads the whole candidate file for mentions), an error outside every example when some example ran (such as in after(:suite)). A project whose bundle has no rspec-core cannot run RSpec candidates: run stops with exit status 3 and says so.
Rails apps
Write RSpec candidates that start with require "rails_helper", as the app's specs do (or Minitest files that start with require "test/test_helper"). They can use the app's factories and its test database. Four things differ from a plain gem.
The test database. Base and head boot the app at the same time. SQLite at a relative path, the Rails default storage/test.sqlite3, is a file in each worktree, so the two sides never meet. A database server (PostgreSQL, MySQL) has one test database name for both sides, and each side's maintain_test_schema! loads its own db/schema.rb into it while the other side runs. On a change that removed a column, 1 to 6 of 6 candidates came back as reported <file>#load rows with bin/rails aborted!, and a collision left the database without its environment data, so every later run failed until the database was dropped and created again. jitcatch sets TEST_ENV_NUMBER in each checkout, the variable parallel_tests sets: 1 and 2 for base and head of the first worker, 3 and 4 for the second worker, and so on. When config/database.yml on both revisions names the test database with it, each checkout uses a database of its own:
test:
<<: *default
database: app_test<%= ENV["TEST_ENV_NUMBER"] %>
Create those databases once, two for each worker you will pass, before the first run. When the test helper calls ActiveRecord::Migration.maintain_test_schema!, Rails loads each revision's schema into them when it boots. Without that call, a database that db:create alone made has no tables, so also pass run --setup 'bin/rails db:create db:schema:load', which loads each side's own schema:
for n in 1 2; do RAILS_ENV=test TEST_ENV_NUMBER=$n bin/rails db:create; done
When none exists, every file fails to load on both sides and run exits 3 (ActiveRecord::NoDatabaseError). When only head's is missing, its rows are discard:no_database; create it and run again. When head's has no tables, its rows are discard:no_table; load the schema there and run again. When database.yml does not use TEST_ENV_NUMBER on both revisions, run finds it before any candidate runs (it reads each side's database.yml as Active Record does, with that side's TEST_ENV_NUMBER and any DATABASE_URL that --env passes) and runs one process at a time, as --serial does, whatever --workers says; each process loads its revision's schema when it starts. The report then prints serial and two lines: warning: base and head use one test database, lending_test, so jitcatch ran one process at a time; ..., and a second warning: that names the database.yml change and the db:create loop above. That change helps only when BASE and HEAD both have it; for past commits, keep the run one process at a time. With --env DATABASE_URL and a URL that names a database on a server, the second line names --number-database-url instead: Rails reads a DATABASE_URL that names a database over database:, so only the URL can split it. A URL that names no database takes database: from database.yml, so the line stays the database.yml one. An SQLite URL at an absolute path gets give DATABASE_URL a relative path, as sqlite3:storage/test.sqlite3, so each checkout opens a file under its own root; --number-database-url refuses an SQLite URL with the same advice. That flag adds each checkout's TEST_ENV_NUMBER to the database name in the URL (app_test_wt1 becomes app_test_wt1_1 and app_test_wt1_2), which fits worktrees that each pass their own DATABASE_URL; the report prints them as numbered_databases app_test_wt1_1 app_test_wt1_2, and you create them once. A database.yml that reads only in the booted app gives no answer, so pass --serial yourself there. On the sample app with PostgreSQL 16 and a schema change, six files took 32 s one at a time and 9 s with numbered databases at a load average of 2. --env cannot pass TEST_ENV_NUMBER, because jitcatch sets it.
A change to the schema. When the diff touches db/schema.rb, db/structure.sql, or db/migrate/, context and run print warning: head changes the database schema (...). A database made once holds one schema, so a row can fail on a missing column and read as a regression (a request spec that gets a 500 for an unknown attribute). When the warning says the helper does not call maintain_test_schema!, pass --setup 'bin/rails db:create db:schema:load': jitcatch runs it once in each checkout, before the candidates, with that side's TEST_ENV_NUMBER, numbered DATABASE_URL, and RAILS_ENV=test. A failed --setup exits 3 with jitcatch: --setup failed, so no test ran. When the database refused the connection there, each side names the variables your shell sets that the run did not get, as the database refused the connection, and jitcatch did not pass DB_PORT from your shell: add --env DB_PORT; add those flags and run again. A reported row whose head answered 500 has hint: head answered 500, and a Rails app writes the error behind it to log/test.log; ...: run its rerun line and read log/test.log before you judge it.
Files git ignores. Each checkout has only what git tracks. config/master.key is not there, so an initializer that reads a credential at boot (Rails.application.credentials.fetch(:secret_key_base)) fails in every checkout with KeyError: key not found: :secret_key_base, while the same command passes in the repository. Pass the key:
RAILS_MASTER_KEY=$(cat config/master.key) jitcatch run --env RAILS_MASTER_KEY BASE HEAD TESTS_DIR
Pass other variables the app reads at boot the same way (--env DATABASE_URL, --env REDIS_URL). RAILS_ENV is not passed; rails_helper sets it to test. --env takes a name, not NAME=VALUE: export the value first. env_unset DB_HOST is a note, not an error: the shell does not set it, so each side ran with the app's own default.
Time. Each file boots the app once on each side. On the sample app (4 CPUs, load average 1 to 3), six files with one example each took 9.8 s, --workers 2 took 6.0 s, and one file with the same six examples took 1.9 s. Put the examples in few files, and raise --timeout above the app's boot time. bootsnap starts with an empty cache in each checkout. One file took 3.7 s with bootsnap and 2.5 s with DISABLE_BOOTSNAP=1 jitcatch run --env DISABLE_BOOTSNAP; six files took 8.0 s with bootsnap and 9.8 s without it, because later files reuse the cache. Disable it for one or two files.
The brief. The existing-test grep finds a spec that calls loan.approve! on a record a factory built, and entry_point_tests needs the class name in the file, which create(:loan) does not write. A changed Loan#initialize is different: its tests: lists the specs that call create(:loan) or Loan.create!. So is a changed callback or validation method: its run_by: names the saves that run it, and its tests: lists the specs that make them. An on: :create callback lists no update, and build(:loan) counts only in a spec that also saves or validates the record. Read the model's spec under spec/models/ as well.
Run
jitcatch run --repo DIR BASE HEAD TESTS_DIR
jitcatch run --json --repo DIR --reruns 2 --timeout 10 --workers 2 BASE HEAD TESTS_DIR
--reruns defaults to 2: that many extra runs on head for a pass-on-base / fail-on-head result. 0 skips the flake check. --timeout defaults to 60 seconds per process when head has config/application.rb and to 10 seconds otherwise (minimum 1
…(truncated)