yigitkonur Run TestSprite Backend
Use TestSprite as an independent HTTP client against a reviewed deployment. Build executable Python tests from repository truth, run them through TestSprite, correlate the returned evidence to one pinned run, fix the demonstrated layer, deploy the exact verified revision, and finish with a fresh external run.
This workflow is TypeScript-backend-first. For another stack, substitute that repository's route, schema, test, build, and deployment conventions; TestSprite still exercises the service over HTTP.
Value boundary
TestSprite can expose defects that same-process tests miss and challenge semantic contracts such as citations, streaming order, routing metadata, typed failures, dependencies, and cleanup. It is not a deployment system, account/proxy provider, CAPTCHA solver, load tester, or autonomous maintainer. Its AI analysis is a hypothesis, not proof.
| TestSprite can establish | It cannot establish by itself |
|---|---|
| What the observed target returned to its client | That the target serves the checkout's commit |
| Whether saved assertions held for a pinned execution | That a suggested root cause is correct |
| Request, response, backend output, trace, and dependency evidence that the service returns | That every backend evidence surface is immutable or run-scoped |
| A fresh external pass after deployment | Native correctness, load tolerance, or security completeness |
Essential rules
- Read the nearest repository instructions before touching code or TestSprite state.
- Prove the target revision. A green run against old or unknown code is not release proof.
- Keep both credential planes out of code, arguments, logs, artifacts, and git.
- Call every backend
test_*function. Backend Python executes top-to-bottom; pytest discovery is not the execution model. - Use explicit reviewed
BASE_URLvalues. Do not trust--target-urlorTARGET_URLto retarget backend Python; verify the observed host. - Keep tests bounded, deterministic, and reversible. Timeouts and the auditor's synchronous zero-argument subset are this skill's safety policy, not TestSprite platform requirements.
- Treat saved tests as durable contracts. Validate AI suggestions; refine and reuse rather than regenerate by default.
- Freeze exact expected test IDs, graph closure, assertions, and cleanup obligations before any run. Never reclassify scope after a failure to manufacture green.
- Require one fail-closed authorization record before any remote write, billable run, credential change, deployment, cancellation, or application side effect. Read-only discovery is the default.
- Do not report a batch or rerun green until every expected member and dependency-closure result is terminal and passed.
When to use this skill
Use it to bootstrap or deepen TestSprite backend coverage, turn API contracts into cloud tests, debug backend results, verify streaming/auth/metadata/error behavior, add deployed-revision CI checks, or prove a fix after deployment.
Do not use it for frontend browser plans, native unit tests alone, generic API-client implementation, load testing, fuzzing, security scanning, or destructive production probing. Hand frontend TestSprite journeys to the canonical run-testsprite-frontend skill. For a public target, the CLI workflow is sufficient and MCP is optional. For a localhost-only target, use TestSprite's documented MCP route; testsprite agent install --target ... installs local guidance and is not MCP.
Before spending runs, require a material consumer-visible contract, a deployed-boundary signal unavailable to native tests, or evidence for a meaningful external integration. A duplicate 200-only check has little value.
Reference router
Load only the references needed for the current phase.
| Situation | Read |
|---|---|
| Unfamiliar repo, project conventions, MCP/public reachability, or uncertain deployed revision | references/repo-discovery.md |
| New suite, coverage gaps, dependencies, streaming, or production-safety design | references/suite-design.md |
| Writing or statically auditing TestSprite Python | references/backend-test-authoring.md |
| Exact 0.4.0 commands, profiles, outputs, batch checks, cancellation, exit codes, or official sources | references/cli-reference.md |
| Failed, blocked, flaky, deferred, partial, or contradictory results | references/failure-loop.md |
| API keys, application credentials, CI, evidence handling, deployment proof, and final release gate | references/release-security.md |
Common reading sets:
- First suite: repo discovery + suite design + authoring + release/security.
- Existing-test verification: repo discovery + CLI reference + failure loop.
- Authenticated CI: CLI reference + release/security.
- Streaming/provider failure: authoring + failure loop + release/security.
Workflow
1. Frame the verification goal
Record:
- behavior or contract to prove;
- target environment and expected deployed revision;
- allowed mutations and cleanup path;
- relevant endpoints and authentication mode;
- exact expected test IDs and producer/consumer/teardown closure, frozen before execution;
- assertion and cleanup obligations, frozen before execution; and
- completion condition: a fresh
test run, terminal verdict, exact membership, and revision evidence.
Predeclare end states: verified, product defect, test defect, deployment drift, runtime/provider gate, or TestSprite execution failure. An account shortage, CAPTCHA, proxy failure, or upstream outage is not code evidence and is not a pass. Do not narrow, rename, or reclassify the frozen scope after a failure.
2. Establish the fail-closed authorization gate
Read-only repository and TestSprite discovery is the default. Before any remote write, billable run, credential operation, test create/update/code/metadata/cancel, deployment, or application side effect, require one explicit authorization record naming:
- TestSprite account/tenant, project ID, and exact test IDs;
- exact application target, environment, account/tenant, and credential environment;
- intended effect, including remote state and billable/provider side effects;
- maximum client polling concurrency and the separately bounded server/application concurrency plan;
- cleanup owner, exact cleanup operation, and success assertion; and
- rollback/recovery path for test state, credentials, deployment, and application data.
Fail closed if any field is absent or scope changes. The gate authorizes only the named operations and does not convert generated plans, a prior run, or general repository access into permission for writes or execution.
3. Discover repository truth
Read repository instructions, manifests, routes/controllers, schemas, product docs, native tests, live harnesses, deployment workflow, and version/health surfaces. Rank conflicting evidence:
- observed behavior of the intended deployed revision;
- executable server code and tests;
- published API schema;
- maintained product documentation;
- old examples and generated prose.
Record contradictions instead of silently normalizing them. Follow repository discovery for the evidence brief.
4. Install and preflight CLI 0.4.0 without persisting secrets
TestSprite CLI 0.4.0 requires Node ^20.19.0 || ^22.13.0 || >=24 and provides the testsprite binary:
npm install --global @testsprite/testsprite-cli@0.4.0
testsprite --version
testsprite --output json auth status
testsprite --output json doctor
Before any CLI call that could send a TestSprite key, reject an inherited TESTSPRITE_API_URL unless the authorization gate names that exact TLS endpoint and confirms the key/profile belongs to its credential environment. Treat explicit --endpoint-url the same way; it is a credential-routing control, not a harmless connectivity flag. Never disclose the inherited value while checking it.
Prefer TESTSPRITE_API_KEY inherited from the environment. It overrides a stored profile even when --profile NAME is explicit. Never auto-run setup, setup --from-env, create .env/.envrc, or replay a key into a profile. setup --from-env persists the environment key. If persistent credential setup is explicitly authorized, use the credential-only path:
testsprite setup --no-agent
To use a stored profile key, unset the inherited key for every invocation, pass the profile explicitly, and verify auth status identity and scopes before any write or run:
env -u TESTSPRITE_API_KEY testsprite --profile NAME --output json auth status
Released wrappers ignore TESTSPRITE_PROFILE. Global flags may appear in other accepted positions; this skill uses one consistent style without claiming the order is required.
Run usage only as an informational probe. Treat a balance as known only when balance fields are actually returned; Portal Billing is authoritative.
5. Resolve project and suite state
Use an explicitly supplied project ID first. TESTSPRITE_PROJECT_ID and .testsprite/config.json may be repository or skill conventions, but they are not general CLI defaults. Use them only when the repository documents them; otherwise resolve an unambiguous project from the API.
testsprite --output json project list --max-items 100
testsprite --output json project get "$PROJECT_ID"
testsprite --output json test list --project "$PROJECT_ID" --type backend --max-items 100
testsprite --output json test list --project "$PROJECT_ID" --type backend --status failed,blocked
Compare project type, names, saved code, recent targets, and repository docs. Create a backend project only when no correct project exists and external creation is authorized.
6. Prove the backend target
The released 0.4.0 implementation and scaffold use explicit code BASE_URL; --target-url does not reliably retarget backend code. Bundled vendor guidance also mentions TARGET_URL, so the official surfaces conflict. Use an explicit reviewed public BASE_URL, distrust overrides, and verify the observed target from result/failure evidence and application logs.
Capture before and after execution:
- base URL, environment, lane/tenant, and revision or image digest;
- TestSprite test ID and saved code version;
- application request ID and timestamps when available.
Matching pre/post revision strings are necessary but insufficient on a mutable target: traffic may have observed A→B→A between probes. Final proof requires an immutable revision URL/release, or request-correlated application and deployment evidence showing that the pinned TestSprite requests reached the intended revision and excluding mid-run drift.
If the service is localhost-only, route to the documented TestSprite MCP workflow rather than pretending the cloud CLI can reach it. Native agent install remains local guidance only.
7. Configure application authentication safely
The TestSprite API key authenticates the CLI. The TestSprite project credential authenticates backend Python to the application. Do not mix them.
Use protected credential files for project auth; avoid inline secret flags:
testsprite --output json project credential "$PROJECT_ID" \
--type "Bearer token" --credential-file "$CREDENTIAL_FILE"
testsprite --output json project credential "$PROJECT_ID" --type public
Vendor-shipped backend contracts include __AUTH_HEADERS__, __AUTH_CREDENTIAL__, and __AUTH_TYPE__. Bind managed headers to one explicitly approved origin: exact scheme, canonical host, and effective port. Managed requests must use allow_redirects=False unless the authorization gate separately approves redirects, and must not place credentials in URL userinfo or query targets. Copy __AUTH_HEADERS__ into authenticated requests. Never access, print, serialize, or expose __AUTH_CREDENTIAL__; do not needlessly expose __AUTH_TYPE__. Read release and security before changing credentials or handling historical leaks.
8. Design the suite
Build a traceable matrix from repository contracts. Cover relevant health/version, auth failures, validation, semantic success, headers/metadata, streaming order, state transitions, provider failure typing, and cleanup. Use stable invariants rather than exact model prose, timestamps, generated IDs, latency, or provider wording unless guaranteed.
Name the independent deployed-boundary signal each scenario adds. Keep non-streaming and streaming cases separate when their mapping paths differ. See suite design.
9. Author and audit executable Python
Start from the local scaffold in an unpredictable restricted directory:
umask 077
TESTSPRITE_WORK_DIR="$(mktemp -d "${TMPDIR:-/tmp}/testsprite-backend.XXXXXXXX")"
testsprite test scaffold --type backend --out "$TESTSPRITE_WORK_DIR/profile.py"
Replace its target and assertions. The runner supports Python stdlib plus vendor-documented requests, pytest, numpy, and scipy; the Python runtime version is undocumented. Use explicit top-to-bottom calls:
import requests
BASE_URL = "https://api.example.com"
AUTH_HEADERS = dict(__AUTH_HEADERS__)
def test_profile_contract() -> None:
response = requests.get(
f"{BASE_URL}/v1/profile",
headers=AUTH_HEADERS,
timeout=(10, 60),
allow_redirects=False,
)
assert response.status_code == 200, f"expected 200, got {response.status_code}"
body = response.json()
assert isinstance(body.get("id"), str) and body["id"]
assert body.get("status") in {"active", "limited"}
test_profile_contract()
Replace the example URL before upload. Before upload, create an assertion ledger. Map every material contract and cleanup obligation to the exact saved-code assert expression, its contract source, and expected test ID. Reject a test with no material assertion, unasserted cleanup, pytest.skip, swallowed assertion/exception paths, or an assertion deleted, weakened, or widened after failure. Freeze this ledger before execution; an auditor pass is not assertion proof.
Resolve TESTSPRITE_SKILL_DIR to this loaded skill directory and run its conservative static policy check. Authenticated audits require the exact approved HTTPS origin:
python3 "$TESTSPRITE_SKILL_DIR/scripts/audit_backend_test.py" \
--auth-required --allowed-origin "https://api.example.com" \
"$TESTSPRITE_WORK_DIR/profile.py"
The auditor rejects syntax errors, uncalled/request-free tests, tests without a reachable non-static assert, imports outside its allowlist, missing bounded request timeouts, private/placeholder targets, unsafe dynamic shapes, obvious skip/exception swallowing, likely embedded credentials, and managed-header origin/redirect violations without printing secret values. It intentionally accepts a narrower synchronous zero-argument subset than the platform may support. It cannot prove assertion strength or cleanup coverage; preserve and manually reconcile the assertion ledger. Read backend authoring.
10. Preview, then create and run one narrow test
--dry-run validates command shape only; it does not execute Python or prove auth, target, deployment, or behavior.
testsprite --dry-run --output json test create \
--type backend --project "$PROJECT_ID" --name "profile contract" \
--code-file "$TESTSPRITE_WORK_DIR/profile.py" --run --wait --timeout 600
testsprite --output json test create \
--type backend --project "$PROJECT_ID" --name "profile contract" \
--code-file "$TESTSPRITE_WORK_DIR/profile.py" --run --wait --timeout 600
For chained test create --run --wait, parse the nested 0.4.0 JSON fields: .testId, .codeVersion, .run.runId, and .run.status. Do not look for run fields at the top level. Capture those fields, observed target, timestamps, dashboard URL, and exit code. Exit 7 covers timeout, unsupported operation, and deferred cases; call test wait only when output contains a real run ID. Never invent or infer one.
11. Diagnose pinned evidence honestly
For backend failures, the released surfaces conflict: shipped code points to latest test failure get TEST_ID, while public docs advertise test artifact get RUN_ID. Do not promise immutable backend artifacts. Capture result/step/code/trace/artifact material only under umask 077 in an unpredictable restricted directory; do not print raw application payloads into the transcript. Share only allowlisted correlation metadata and sanitized content.
Attempt the documented/run-oriented surface when supported, but correlate every retrieved result, failure bundle, apiOutput, and trace to the pinned run ID, code version, target, request IDs, and timestamps. If backend wait uses a legacy/test-level fallback, final proof additionally requires runIdIfAvailable == RUN_ID. Missing, null, mismatched, or unparseable run correlation is diagnostic only and must be classified as unresolved TestSprite execution failure, never a pass.
umask 077
TESTSPRITE_EVIDENCE_DIR="$(mktemp -d "${TMPDIR:-/tmp}/testsprite-evidence.XXXXXXXX")"
testsprite --output json test steps "$TEST_ID" --run-id "$RUN_ID" --max-items 100 \
> "$TESTSPRITE_EVIDENCE_DIR/steps.json"
testsprite --output json test result "$TEST_ID" --include-analysis \
> "$TESTSPRITE_EVIDENCE_DIR/result.json"
testsprite --output json test failure get "$TEST_ID" \
--out "$TESTSPRITE_EVIDENCE_DIR/failure"
Use test artifact get "$RUN_ID" only when installed help and actual backend output support it; do not silently substitute latest failure data for the pinned run. Classify product code, test code, deployment, configuration, auth/capacity, edge, provider, TestSprite runner, dependency starvation, or nondeterminism before editing. Follow failure loop.
12. Fix the demonstrated layer
- Product defect: add a native regression, fix code, pass exact-revision CI, deploy that artifact, and rerun.
- Test defect: correct target/setup/parser/assertion without weakening the contract.
- Deployment defect: correct target/config/revision and prove it live before rerunning.
- Runtime/provider gate: preserve evidence and retry only after observable resource state changes.
- TestSprite defect: preserve run/request IDs and bounded diagnostics; do not patch product code without a faithful response.
- Dependency starvation: fix failed producers/teardowns before judging consumers.
Update code optimistically:
testsprite test code get "$TEST_ID" --out "$TESTSPRITE_WORK_DIR/current.py"
testsprite --output json test code get "$TEST_ID" \
> "$TESTSPRITE_WORK_DIR/current-code.json"
testsprite --output json test code put "$TEST_ID" \
--code-file "$TESTSPRITE_WORK_DIR/fixed.py" --expected-version "$CODE_VERSION"
After any code or metadata update, export the final saved code again, record the returned final codeVersion, and reconcile every ledger assertion and cleanup obligation against that exact version before another run. Do not accept deleted, weakened, widened, skipped, or exception-swallowed assertions.
TestSprite 0.4.0 also permits dependency metadata changes through test update, including --produces, --needs, and --category; inspect testsprite test update --help, review the current graph, and update in place instead of delete/recreate. Do not use forceful overwrites unless a competing edit is proven disposable.
13. Expand safely and prove exact outcomes
For batch execution, use the exact test IDs and graph closure frozen before any run. Exit 0 is insufficient: require no conflicts, deferred members, or skipped arrays; require returned membership to equal the expected set; and require every accepted member to be terminal passed. Missing, extra, queued, blocked, cancelled, or unknown members keep the batch non-green. Never drop a failed member or reclassify the closure after execution.
For backend test run --all --wait, --max-concurrency limits client polling only. It does not limit server execution, traffic, dispatch, shared-state overlap, or provider pressure; serialize or wave-order server work explicitly.
Backend rerun may execute a producer/teardown closure while the process exit reflects only the selected test. Inspect closureFailures[] and every closure member; any failed producer or teardown keeps the rerun non-green. --skip-dependencies is valid only for a truly self-contained test. Rerun and flaky checks are feedback, never final release proof.
Ctrl-C detaches local polling. Stop server-side execution explicitly when authorized:
testsprite --output json test cancel "$RUN_ID"
test cancel exits 0 when the run is cancelled or already cancelled, 4 when not found, 6 when the run is already terminal and cancellation conflicts, and may return 1 for a multi-error outcome. A subsequent test wait on a cancelled run exits 1. Cancellation does not refund credits; query and record final server state. Exit 14 is CLIENT_TOO_OLD and is never cancellation.
After deploying the exact verified revision and confirming the saved semantic contract and target are unchanged, finish with a fresh run:
testsprite --output json test run "$TEST_ID" --wait --timeout 600
An individual fresh run is final proof only for a self-contained test with no dependency or teardown relationship. Graph-backed release proof requires one fresh wave-ordered batch whose frozen membership is the exact producer/consumer/teardown closure and whose every member is terminal passed with no conflict, deferred, skipped, missing, or extra member.
Dry-run, rerun, flaky, queued, deferred, auto-attached, stale-target, or uncorrelated legacy fallback activity is not final proof. Re-probe the deployed revision after the run, exclude mutable-target A→B→A drift with immutable or request-correlated deployment evidence, and recheck the final saved code version against the assertion ledger.
Completion record
Report each planned test separately; never collapse environmental or partial states into “code green.”
| Field | Required evidence |
|---|---|
| Project/test | Project ID/name and exact expected test IDs |
| Target | Explicit saved base URL, observed host, environment/lane |
| Revision | Expected SHA plus immutable release URL or request-correlated deploy/app proof excluding A→B→A drift |
| Run | Run ID, fresh/rerun, terminal status, exit code; legacy fallback runIdIfAvailable equality |
| Membership | Frozen exact set; no conflicts/deferred/skips; all terminal passed |
| Closure | Frozen producer/consumer/teardown wave and closureFailures[] |
| Test code | Final saved code version and assertion-ledger reconciliation |
| Assertions | Every material contract and cleanup obligation mapped to exact saved-code assertions |
| Evidence | Correlation basis; unresolved TestSprite execution failure if run scope is missing/unparseable |
| Fix | Native regression/commit or environment action |
| Final gate | Fresh run after deploy, or explicit unpassed external gate |
Claim only the rung reached. “Native CI is green” and “TestSprite verified the deployed contract” are different facts.
Pitfalls
| Pitfall | Corrective action |
|---|---|
| API key is persisted automatically | Prefer inherited TESTSPRITE_API_KEY; never auto-run setup or create env files |
TESTSPRITE_PROFILE is assumed active |
Pass explicit --profile; unset TESTSPRITE_API_KEY, then verify identity/scopes |
| Endpoint override is inherited casually | Reject TESTSPRITE_API_URL/--endpoint-url unless exact TLS credential routing is authorized |
| Auditor pass is called assertion proof | Keep the assertion ledger; reject missing/weakened/widened assertions and unasserted cleanup |
| Test function is defined but not called | Call it at module bottom; run the auditor |
| Backend target override is trusted | Review explicit BASE_URL; verify observed host and revision |
| Latest failure is treated as pinned evidence | Correlate run/code/target/timestamps or report unresolved |
| Exit 7 always triggers wait | Wait only when output contains a real run ID |
| Batch exit 0 is called green | Check exact IDs, conflicts, deferred/skipped arrays, and terminal statuses |
| Selected rerun passed but producer failed | Inspect all closure members and closureFailures[] |
| Ctrl-C is treated as cancellation | Use authorized test cancel RUN_ID; parse 0/4/6/1 and remember later wait exits 1 |
| Exit 14 is treated as cancellation | Exit 14 is CLIENT_TOO_OLD; upgrade the client |
| Rerun is reported as release proof | Finish with a fresh test run after exact-revision deploy |
| LLM recommendation is applied directly | Reconcile TestSprite, runtime, repository, and deployment evidence |
Trigger calibration
Should trigger:
- “Create a TestSprite backend suite from this OpenAPI file.”
- “Run the API tests against staging and debug the failures.”
- “Test our SSE endpoint and citation URLs with TestSprite.”
- “Move these TestSprite tests to managed credentials.”
- “Add TestSprite backend checks to CI and gate the deployed SHA.”
- “Why did this TestSprite batch exit zero with skipped tests?”
- “Cancel this TestSprite backend run on the server.”
- “Did this API suite find a regression or only an environment gate?”
Should not trigger:
- “Write Vitest unit tests for this parser.”
- “Create a TestSprite frontend browser journey.”
- “Benchmark this endpoint at 5,000 requests per second.”
- “Implement an HTTP client SDK.”
- “Fuzz this API for security vulnerabilities.”
- “Validate OpenAPI formatting without calling a deployment.”
- “Mock the database in pytest.”