Kb Review

When to invoke

Use when: "knowledge base review", "rag review", "bedrock knowledge base", "chunking strategy", "retrieval quality", "recall at k", "vector store review", "why does the bot not find the document".

Preamble

eval "$(~/.vibestack/bin/vibe-slug 2>/dev/null)" 2>/dev/null || SLUG="unknown"
_LEARN_FILE="${VIBESTACK_HOME:-$HOME/.vibestack}/projects/${SLUG:-unknown}/learnings.jsonl"
if [ -f "$_LEARN_FILE" ]; then
  _LEARN_COUNT=$(wc -l < "$_LEARN_FILE" 2>/dev/null | tr -d ' ')
  echo "LEARNINGS: $_LEARN_COUNT entries loaded"
  if [ "$_LEARN_COUNT" -gt 5 ] 2>/dev/null; then
    ~/.vibestack/bin/vibe-learnings-search --limit 5 2>/dev/null || true
  fi
else
  echo "LEARNINGS: none yet"
fi

{{include lib/snippets/session-host.md}}

{{include lib/snippets/decision-brief.md}}

{{include lib/snippets/working-protocols.md}}

{{include lib/snippets/state-protocols.md}}

User-invocable

When the user types /kb-review, run this skill.

Step 1: Inventory

Find out what the knowledge base is made of before judging any of it. Two places to look: infrastructure code in the repo, and the live account.

Repo first. Grep for the Terraform resources and for the SDK calls a hand-built pipeline would use:

grep -rnE 'aws_bedrockagent_(knowledge_base|data_source)|aws_opensearchserverless_collection|pgvector|s3_vectors|S3_VECTORS' --include='*.tf' --include='*.py' --include='*.ts' --include='*.yaml' --include='*.yml' . 2>/dev/null | head -40
grep -rnE 'RecursiveCharacterTextSplitter|SentenceSplitter|chunk_size|chunk_overlap|embed_documents|Pinecone|Qdrant|Weaviate|pgvector' --include='*.py' --include='*.ts' . 2>/dev/null | head -40

If the first grep hits, this is a Bedrock Knowledge Base and the Terraform is the source of truth for configuration. If only the second hits, it is a hand-built pipeline: read the ingestion and query code and map each of the following steps onto it. If neither hits, ask where the knowledge base lives.

Live account, when the CLI works. Check credentials once and stop with a setup note if they fail (profile tried, error text, aws configure or aws sso login). Then list what exists:

aws sts get-caller-identity --output json
aws bedrock-agent list-knowledge-bases --output json
aws bedrock-agent get-knowledge-base --knowledge-base-id <kb-id> --output json
aws bedrock-agent list-data-sources --knowledge-base-id <kb-id> --output json
aws bedrock-agent get-data-source --knowledge-base-id <kb-id> --data-source-id <data-source-id> --output json

Every angle-bracket value in this skill — <kb-id>, <data-source-id>, <bucket>, <prefix> and the rest — is substituted by hand from the output of the preceding command before the line is run. They are placeholders, not shell variables.

Read the knowledge base type first. knowledgeBaseConfiguration.type is one of VECTOR, MANAGED, KENDRA or SQL, and it decides which fields exist and which retrieval request shape is valid in Steps 3 and 6:

• VECTOR — a knowledge base backed by a vector store you own. Embedding model ARN and dimensions come from knowledgeBaseConfiguration.vectorKnowledgeBaseConfiguration; the store comes from storageConfiguration.type (OpenSearch Serverless, Aurora with pgvector, S3 Vectors, Pinecone, or another). Query with retrievalConfiguration.vectorSearchConfiguration.
• MANAGED — Amazon Bedrock owns the vector store. Read knowledgeBaseConfiguration.managedKnowledgeBaseConfiguration: embeddingModelType is MANAGED (service-chosen model) or CUSTOM (with embeddingModelArn). storageConfiguration is optional on the KnowledgeBase object and is absent here — record the store as service-managed rather than reporting a missing field. Query with retrievalConfiguration.managedSearchConfiguration.
• KENDRA — an Amazon Kendra GenAI index holds the content, so there is no chunking, embedding or vector-store configuration to review. Query it with Retrieve as you would a standard knowledge base, vectorSearchConfiguration and all; the Kendra fields come back as metadata attributes and metadata filtering works on them. Say so, skip Steps 2, 4 and 7's store item, and run Steps 3, 5 and 6 unchanged.
• SQL — a structured data store answers by generated SQL, not by embeddings, so there is nothing chunked, embedded or stored as vectors to review. Retrieve returns the rows the generated query produced, and none of retrievalConfiguration applies: numberOfResults, search type, metadata filtering and reranking are all documented as applying to unstructured data sources only. Send the query on its own, with no --retrieval-configuration at all. Skip Steps 2, 4 and 7's store item. Step 3 keeps its question but changes its evidence — there is no metadata filter to check, so read the query engine's table and column inclusions and exclusions and the database grants on the role the knowledge base queries with. Steps 5 and 6 run with the row-based reading noted in each.

Sending a vectorSearchConfiguration to a managed knowledge base, or a managedSearchConfiguration to a VECTOR one, is an invalid request — a failed call there is a mistake in this review, not a finding about the knowledge base. So is sending either one to a SQL knowledge base. Set the key once and reuse it:

SEARCH_KEY=vectorSearchConfiguration    # managedSearchConfiguration on a MANAGED KB
                                        # SQL: send no retrieval configuration at all

Then fill in the KB summary table: knowledge base id, name and type; the store and embedding model from whichever branch above applies; and per data source its type (S3, Confluence, SharePoint, OneDrive, Google Drive, Salesforce, Box, web crawler, custom — record whatever dataSourceConfiguration.type reports, the connector set grows), location, chunking configuration, parsing configuration and dataDeletionPolicy.

Sources. For every S3 data source, size it without downloading it:

aws s3 ls s3://<bucket>/<prefix> --recursive --summarize | tail -2
aws s3 ls s3://<bucket>/<prefix> --recursive | awk '{print $4}' | sed -E 's/.*\.([A-Za-z0-9]+)$/\1/' | sort | uniq -c | sort -rn

Record object count, total size, and the format mix (PDF, DOCX, HTML, Markdown, CSV, JSON). Count the image objects (.jpeg, .png) on their own line: Step 7 prices a configured parser per page and per image, and neither number falls out of the object count. When any data source sets a parsingConfiguration, carry a page estimate into Step 7 as well — read the page count of each PDF sampled in Step 2 (pdfinfo, or mdls -name kMDItemNumberOfPages on macOS), take the mean, multiply by the PDF object count, and record the sample size beside it so the number is read as the estimate it is. Note the update cadence from the newest and oldest LastModified values and from whatever writes to the prefix (a pipeline, a person, a Confluence export). For Confluence and web sources, record the space keys or seed URLs and the crawl scope.

Findings at this step: INFO for every data source, MEDIUM if a data source points at a bucket prefix that also holds unrelated files (logs, exports, backups) that will be embedded along with the documents.

Step 2: Chunking

Read vectorIngestionConfiguration.chunkingConfiguration from each data source (or the splitter call in a hand-built pipeline) and compare it against the shape of the documents. Sample three to five documents per format into the scratchpad with aws s3 cp; look at them, do not paste them into the report.

Check:

• Strategy versus document shape. Fixed-size chunks suit short, flat documents. Hierarchical chunking (parent and child levels) suits long manuals and policies where a paragraph only makes sense with its section heading. Semantic chunking suits prose with topic shifts. NONE means each document is treated as a single source chunk; the service applies its own chunk handling from there, and a document that violates a limit surfaces in the ingestion job, not as a quiet truncation. Do not infer truncation from the embedding model's input limit — check it. Read numberOfDocumentsFailed and failureReasons on the last job (Step 5), then run one retrieve for a phrase that appears only near the end of the largest sampled document and read content.text on the results. HIGH if documents failed ingestion, or if that tail phrase is in no returned chunk while the same document answers other questions. MEDIUM if the strategy is NONE on long documents and neither check could be run — the risk is real, the evidence is not there yet. Note also that NONE gives up page-number citations and the x-amz-bedrock-kb-document-page-number filter attribute.
• Chunk size and overlap. Read maxTokens and overlapPercentage (fixed) or the level maxTokens and overlapTokens (hierarchical). MEDIUM if chunks are under 200 tokens on documents with long paragraphs (answers get split across chunks) or over 1000 tokens on FAQ-style content (one chunk carries several unrelated questions). LOW if overlap is 0 on fixed-size chunking.
• Tables and code. Open a sampled PDF or DOCX that contains a table and find the corresponding chunk text with a retrieve call for a value from the table. MEDIUM if table rows come back as a run of numbers with no header context and no parsing configuration is set (parsingConfiguration with a foundation model or Bedrock Data Automation keeps table structure). Same check for code blocks in Markdown.
• Metadata files. Each document may carry a sidecar <name>.metadata.json with a metadataAttributes object. Count them: aws s3 ls s3://<bucket>/<prefix> --recursive | grep -c '\.metadata\.json$'. Compare with the document count. MEDIUM if fewer than 90% of documents have one and Step 3 relies on filtering.

Step 3: Metadata and isolation

This is the step that decides whether one customer can read another customer's documents. Treat every gap here as at least HIGH.

• What is in the metadata. Read three sampled .metadata.json files. The attributes that matter are tenant (or customer, account), department, and classification (public, internal, confidential). Record which exist and whether values are consistent (tenant versus tenantId versus customer across files is a finding).
• Which mechanism is in play. Two are supported and they are checked differently, so decide before judging anything. Metadata filtering: documents carry a tenant attribute and every query passes a filter. ACL awareness, on a managed knowledge base: the connector crawls document permissions (allowed and denied users and groups) at ingestion, and the query passes a userContext naming the user, and the crawled permissions filter the candidates before retrieval. Some connectors add a second stage, a live call to the source per query that catches permission changes made since the last sync. Which ones do is a per-connector property and the connector set grows, so read it from that connector's own ACL page rather than from a list held in your head. Today: SharePoint, OneDrive, Google Drive, Confluence and Box document both stages; S3 and custom document pre-retrieval filtering only, because their ACLs come from a file the customer writes and there is nothing live to re-check; the web crawler supports no ACLs at all. For any connector you have not checked, record its real-time support as unknown and go read the page before a severity leans on it. Read each data source's configuration for ACL awareness, and read the call sites for which of filter and userContext they send. A knowledge base may use one, both, or neither. An ACL-enabled data source returning nothing when userContext is absent is documented behaviour, not a retrieval bug; non-ACL data sources in the same knowledge base still return to everyone.
• Filter applied on every retrieve. Find every call site of retrieve, retrieve_and_generate, RetrieveCommand, or the agent's knowledge base configuration: grep -rnE 'retrieve(_and_generate|AndGenerate)?\(|RetrieveCommand|RetrieveAndGenerateCommand' --include='*.py' --include='*.ts' --include='*.js' . On a metadata-filtered knowledge base, check that retrievalConfiguration.$SEARCH_KEY.filter is set from the caller's authenticated session, not from a request parameter the caller controls. CRITICAL if a call site queries a multi-tenant knowledge base with neither a filter nor a user context. HIGH if the filter value comes from the request body, a header or a query parameter.
• User context derived from a verified identity. On an ACL-aware knowledge base the same question applies to userContext.userId. The service does not authenticate end users and cannot verify the identity it is handed — it filters on whatever you send — so that email must come from the verified session (the token subject or an ID-token claim). CRITICAL if userId is caller-supplied: one edited request reads another tenant's documents. HIGH if the address passed is not the one the connected source knows the user by; matching is on the email with no alias resolution, so a mismatch returns nothing and invites someone to "fix" it by dropping the context. MEDIUM if group membership is the only thing standing between tenants and the connector behind that data source has no real-time check — memberships are resolved from the last crawl, so a removal takes effect only at the next sync. On a connector that re-verifies per query the live check corrects that at query time: record it INFO. If which of the two the connector is was never established, say the real-time support is unknown and leave the severity off rather than guessing at one.
• Prove the leakage path is closed. Run one retrieve as tenant A with a question only tenant B's documents answer. Substitute <kb-id> and use the branch that matches the mechanism:

eval "$(~/.vibestack/bin/vibe-slug 2>/dev/null)" 2>/dev/null || SLUG="unknown"
EVAL_DIR="${VIBESTACK_HOME:-$HOME/.vibestack}/projects/${SLUG:-unknown}"
mkdir -p "$EVAL_DIR"
OUT="$EVAL_DIR/kb-isolation-$(date +%Y-%m-%d).json"
SEARCH_KEY=vectorSearchConfiguration    # managedSearchConfiguration on a MANAGED KB
CFG=$(jq -nc --arg k "$SEARCH_KEY" '{($k):{numberOfResults:5,filter:{equals:{key:"tenant",value:"tenant-a"}}}}')
ERR=$(aws bedrock-agent-runtime retrieve --knowledge-base-id <kb-id> \
  --retrieval-query "$(jq -nc --arg text '<question only tenant B can answer>' '{text:$text}')" \
  --retrieval-configuration "$CFG" \
  --output json 2>&1 > "$OUT")
rc=$?
if [ "$rc" -ne 0 ]; then
  printf 'ISOLATION: N-A (exit %s)\n%s\n' "$rc" "$ERR"
else
  jq -r '.retrievalResults[] | [(.location.type // "?"), (.documentId // "-"), (.score|tostring)] | @tsv' "$OUT"
  printf 'ISOLATION: %s result(s)\n' "$(jq '.retrievalResults | length' "$OUT")"
fi

On an ACL-aware knowledge base, drop the filter from CFG and add --user-context '{"userId":"user-a@example.com"}' with an address that belongs to tenant A.

On a SQL knowledge base there is no filter and no user context to send: drop --retrieval-configuration and the CFG line with it, run the probe as the caller the application would use for tenant A, and read whether the returned rows belong to tenant A. What holds tenants apart there is the query engine's table and column inclusions and exclusions plus the grants on the database role, so review those alongside the probe — a generated query that can reach another tenant's rows is CRITICAL on the same terms as a missing filter.

Pass the query as JSON, not as the text=... shorthand: the shorthand parser splits on commas, so any question containing one is read as a list and the call fails with Invalid type for parameter retrievalQuery.text. That kills German questions in particular. Without jq, write the object out by hand: --retrieval-query '{"text":"Wie lange ist die Frist, wenn ich kuendige?"}'.

Read the exit status before the results. An empty list is a pass only when rc is 0; a denied bedrock:Retrieve, a throttle and a genuinely empty result all print nothing, and only the status tells them apart, so report N-A on a non-zero exit rather than a clean bill. On a zero exit, every returned document must belong to tenant A. A tenant B document is CRITICAL, and the fix is at ingestion (missing or wrong metadata, or permissions the connector did not crawl), not in the query.

• Source access control mirrored. If the S3 bucket, Confluence space, or SharePoint site restricts who can read what, that restriction has to be reproduced as metadata plus a filter, because the knowledge base reads everything with its service role. HIGH if a restricted source feeds the knowledge base and no metadata carries the restriction.

Step 4: Embeddings

• Model and dimension. From Step 1, record the embedding model id (for example amazon.titan-embed-text-v2:0 or cohere.embed-multilingual-v3) and the configured dimensions. Confirm the vector index in the store was created with the same dimension: for OpenSearch Serverless read the index mapping, for pgvector read the column type (vector(1024)). A mismatch fails at ingestion and shows up as every job failing; HIGH.
• Language coverage. Ask which languages the documents and the users are in. The usual case is German plus English. HIGH if the corpus or the questions include German and the model is English-only (cohere.embed-english-v3). Titan Text Embeddings v2 and Cohere Embed Multilingual cover both. Verify with the golden set in Step 6: include at least five German questions against English passages and five the other way round, and report their recall separately.
• Re-embedding plan. An embedding model cannot be changed on an existing Bedrock knowledge base; a model change means a new knowledge base, a full ingestion, and a cutover. MEDIUM if nothing in the repo or runbooks describes how that is done, and note the full ingestion cost from Step 7 next to the finding.

Step 5: Sync

The knowledge base only knows what the last ingestion job saw.

aws bedrock-agent list-ingestion-jobs --knowledge-base-id <kb-id> --data-source-id <data-source-id> --max-results 20 --output json

For each data source, record the last job's status, start time, and its statistics (documents scanned, new, modified, deleted, failed) plus failureReasons when present. Then check:

• Schedule. Find what starts the job: an EventBridge rule, a Step Functions state machine, a CI step, or a person. grep -rnE 'start_ingestion_job|StartIngestionJob|start-ingestion-job' . and aws events list-rules --output json | grep -i ingest. MEDIUM if the only trigger is manual and the source changes weekly or faster. HIGH if the last successful job is older than two update cycles of the source.
• Failure alerts. MEDIUM if a job can fail with nobody told; look for a CloudWatch alarm or a notification on the job status. numberOfDocumentsFailed above zero on the last job with no alert is the same finding.
• Deletions propagate. dataDeletionPolicy on the data source: DELETE removes vectors when a document leaves the source, RETAIN keeps them. HIGH if RETAIN is set and documents are ever removed for legal or contractual reasons; the knowledge base will keep answering from them.
• Source versioning. LOW if the S3 bucket has versioning off and nobody can say what the knowledge base was built from a month ago. INFO if it is on.

For a hand-built pipeline, the same four questions apply to whatever loop upserts into the vector store.

Step 6: Retrieval quality

Numbers, not opinions. Build a golden set, run retrieve-only against it, then check a sample of generated answers.

Golden set. Write 20 to 40 question and expected-passage pairs. Draw questions from the sampled documents in Step 2, from support tickets or chat logs if the user can share them, and from the user directly. Each line names the source document that answers it and a short quote of the passage. Include the language split from Step 4 and at least three questions that the corpus does not answer (the right result is nothing above the score threshold). Write it here:

eval "$(~/.vibestack/bin/vibe-slug 2>/dev/null)" 2>/dev/null || SLUG="unknown"
EVAL_DIR="${VIBESTACK_HOME:-$HOME/.vibestack}/projects/${SLUG:-unknown}"
mkdir -p "$EVAL_DIR"
EVAL_SET="$EVAL_DIR/kb-eval-$(date +%Y-%m-%d).jsonl"
echo "EVAL_SET: $EVAL_SET"

One JSON object per line:

{"id":"q01","question":"Wie lange ist die Kündigungsfrist?","lang":"de","tenant":"tenant-a","expected_uri":"s3://docs/tenant-a/contract-terms.pdf","expected_passage":"Die Kündigungsfrist beträgt drei Monate zum Quartalsende.","answerable":true}

The tenant field belongs there only when Step 3 found a tenant attribute in the document metadata. On a knowledge base with no such attribute, leave it out — every line then carries the same fields minus tenant.

Retrieve-only run. For every line, call retrieve with numberOfResults 5, filtered by tenant when the line has one, and record the ranked hits: the document the hit came from, its documentId, its content.type, its content.text, its content.row and its score. Keep the text — the metrics below cannot be computed without it — and keep the row with it: a structured result carries its payload in content.row, an array of columnName, columnValue and type, with content.type set to ROW and content.text empty. documentId identifies the document, not the chunk, and a retrieval result carries no per-chunk identifier, so the passage judgement below is made from the text and nothing else can stand in for it. The filter clause is dropped for a line with no tenant field: filtering on a metadata key the documents do not carry returns nothing for every question, which scores 0.00 and manufactures a HIGH finding against a knowledge base that is fine.

Result locations come in ten shapes, one per source type, and only one is populated per hit; location can also be absent. Reading three of them and calling the rest null turns every Salesforce, SharePoint, OneDrive, Google Drive, Kendra, custom or structured hit into a miss. Normalise all of them, and fall back to documentId.

With jq present the loop is this, with <kb-id> substituted before it runs (it re-derives the eval-set path, since each block runs in its own shell):

set -o pipefail
eval "$(~/.vibestack/bin/vibe-slug 2>/dev/null)" 2>/dev/null || SLUG="unknown"
EVAL_DIR="${VIBESTACK_HOME:-$HOME/.vibestack}/projects/${SLUG:-unknown}"
EVAL_SET="$EVAL_DIR/kb-eval-$(date +%Y-%m-%d).jsonl"
RESULTS="$EVAL_DIR/kb-eval-results-$(date +%Y-%m-%d).jsonl"
SEARCH_KEY=vectorSearchConfiguration    # managedSearchConfiguration on a MANAGED KB
: > "$RESULTS"
while IFS= read -r line; do
  id=$(printf '%s' "$line" | jq -r .id)
  q=$(printf '%s' "$line" | jq -r .question)
  t=$(printf '%s' "$line" | jq -r '.tenant // empty')
  if [ -n "$t" ]; then
    cfg=$(jq -nc --arg k "$SEARCH_KEY" --arg t "$t" '{($k):{numberOfResults:5,filter:{equals:{key:"tenant",value:$t}}}}')
  else
    cfg=$(jq -nc --arg k "$SEARCH_KEY" '{($k):{numberOfResults:5}}')
  fi
  raw=$(aws bedrock-agent-runtime retrieve --knowledge-base-id <kb-id> \
    --retrieval-query "$(jq -nc --arg text "$q" '{text:$text}')" \
    --retrieval-configuration "$cfg" \
    --output json 2>&1)
  rc=$?
  if [ "$rc" -ne 0 ]; then
    jq -nc --arg id "$id" --argjson rc "$rc" --arg err "$raw" \
      '{id:$id, status:"error", exit:$rc, error:$err, hits:[]}' >> "$RESULTS"
    continue
  fi
  printf '%s' "$raw" | jq -c --argjson g "$line" '
    {id:$g.id, status:"ok", expected:$g.expected_uri,
     hits:[.retrievalResults[] | {
       doc: (.location.s3Location.uri // .location.webLocation.url
             // .location.confluenceLocation.url // .location.salesforceLocation.url
             // .location.sharePointLocation.url // .location.oneDriveLocation.url
             // .location.googleDriveLocation.url // .location.kendraDocumentLocation.uri
             // .location.customDocumentLocation.id // .location.sqlLocation.query
             // .documentId // null),
       loc_type: (.location.type // "UNKNOWN"),
       doc_id: (.documentId // null),
       content_type: (.content.type // "UNKNOWN"),
       text: (.content.text // ""),
       row: (.content.row // []),
       score: .score}]}' >> "$RESULTS"
done < "$EVAL_SET"

Same reason for the query object as in Step 3: text=$q is shorthand and splits on the first comma, so a question with a comma in it fails the call and silently drops that line from the score.

On a SQL knowledge base drop --retrieval-configuration and both cfg branches — the settings they carry apply to unstructured sources only — and judge each answer against the expected value the golden line names rather than against a passage. The judgement reads row, not text — text is empty on every structured hit, and the returned value is the columnValue of the row entry whose columnName is the one the golden line expects. There are no chunks to rank, so report the share of questions answered with the right value and leave the passage and MRR columns blank.

Check the denominator before scoring anything. A throttle or a denied bedrock:Retrieve costs one row, and a metric computed over the survivors reads better than the truth:

This block runs in its own shell, so it re-derives the paths the same way Steps 3, 6 and 7 do. It exits non-zero on anything it cannot establish — an unwritable id file prints an all-clear otherwise, which is the failure it exists to catch:

set -euo pipefail
eval "$(~/.vibestack/bin/vibe-slug 2>/dev/null)" 2>/dev/null || SLUG="unknown"
EVAL_DIR="${VIBESTACK_HOME:-$HOME/.vibestack}/projects/${SLUG:-unknown}"
EVAL_SET="$EVAL_DIR/kb-eval-$(date +%Y-%m-%d).jsonl"
RESULTS="$EVAL_DIR/kb-eval-results-$(date +%Y-%m-%d).jsonl"
ASKED="$EVAL_DIR/.ids-asked"
GOT="$EVAL_DIR/.ids-got"
for f in "$EVAL_SET" "$RESULTS"; do
  [ -s "$f" ] || { printf 'DENOMINATOR: N-A (missing or empty %s)\n' "$f"; exit 1; }
done
jq -r .id "$EVAL_SET" | sort > "$ASKED" || { printf 'DENOMINATOR: N-A (cannot write %s)\n' "$ASKED"; exit 1; }
jq -r .id "$RESULTS"  | sort > "$GOT"   || { printf 'DENOMINATOR: N-A (cannot write %s)\n' "$GOT"; exit 1; }
missing=$(comm -23 "$ASKED" "$GOT")
dupes=$(uniq -d < "$GOT")
errs=$(jq -r 'select(.status=="error") | .id' "$RESULTS" | tr '\n' ' ')
printf 'asked=%s got=%s\n' "$(wc -l < "$ASKED" | tr -d ' ')" "$(wc -l < "$GOT" | tr -d ' ')"
printf 'missing=[%s] duplicate=[%s] errors=[%s]\n' "$missing" "$dupes" "$errs"
[ -z "${missing}${dupes}${errs}" ] || { printf 'DENOMINATOR: FAIL — do not score this run\n'; exit 1; }
printf 'DENOMINATOR: OK\n'

A non-zero exit, or anything but DENOMINATOR: OK on the last line, stops the scoring. Fix the cause and re-run the affected questions — pause and retry on a throttle, get the permission for an authorization error — or, if it cannot be fixed, report the metrics as N-A for that subset and state how many questions were dropped and why. Do not score a partial set as if it were whole.

Without jq, run the calls one at a time and record the ranks by hand; the set is small enough.

Metrics. Two different measurements, and the difference decides what a number is allowed to claim:

• Document hit rate@5 — the share of answerable questions whose expected_uri matches the doc of any of the five hits. It says the right file came back. It says nothing about the passage: on a long manual, a chunk about an unrelated section scores exactly as a hit, so this number alone cannot support a claim about retrieval quality and cannot feed a rank-based metric.
• Passage hit rate@5 and MRR — read the text of the hits in rank order and mark the first one that actually answers the question. expected_passage is the reference; an equally correct passage elsewhere counts, a chunk from the right document that does not answer it does not. Passage hit rate@5 is the share with such a hit among the five; MRR is the mean of 1/rank of that hit, 0 when there is none. Record the rank you assigned per question alongside its row so the next review can see the judgement.

Report both, for the whole set and per language. If the passages were not judged, print the document number under its own name, leave the passage and MRR columns blank, and write "passage relevance not judged" under the table — do not carry the document-level rank into the MRR column.

For the unanswerable questions, report how many returned a top score above the threshold the application uses (a false positive).

Findings, all read off the passage numbers: HIGH if passage hit rate@5 is under 0.7, MEDIUM under 0.85, LOW if MRR is under 0.5 while passage hit rate@5 is fine (the right chunk is there but ranked low; a reranker or smaller chunks are the usual fix). MEDIUM if a language subset trails the other by more than 0.15. When only the document number was measured, cap every one of these at MEDIUM and say so — that evidence cannot tell a right passage from a wrong one in the right file.

Generation and faithfulness. Pick 10 answerable questions and run aws bedrock-agent-runtime retrieve-and-generate (or the application's own answer endpoint) with citations on. For each answer check two things by reading the cited passages: every factual claim in the answer appears in a cited passage, and the citation points at the expected document. Score each answer faithful, partly (a claim with no support), or wrong. HIGH if two or more of ten are wrong. Record the model id and the prompt template used, since both move the result.

These calls cost tokens. Say the count before running (40 retrieves plus 10 generations is the default) and do not loop beyond the golden set.

Step 7: Cost

The bill has more parts than the embeddings. Walk the list, mark the items this knowledge base actually uses, and drop the rest with a line saying it is not configured. Pull current unit prices from the billing MCP pricing tool when the session has it, otherwise from the public pricing page for each service, and write the date the price was read next to it.

1. Vector store. Read the live configuration; there is no fixed floor to assume. OpenSearch Serverless capacity is a minimum and a maximum OCU count for indexing and for search, set per collection group — or at account level for a Classic collection that belongs to no group. The minimum can be 0, in which case an idle group needs no OCUs at all, and collections inside a group share OCUs. So the idle cost is whatever the configured minimums are, divided across every collection sharing them, not two or four per collection. Read it and measure it:

   aws opensearchserverless batch-get-collection --names <collection-name> --output json
   aws opensearchserverless list-collection-groups --output json
   aws opensearchserverless batch-get-collection-group --ids <collection-group-id> --output json
   aws opensearchserverless get-account-settings --output json
   aws cloudwatch get-metric-statistics --namespace AWS/AOSS --metric-name SearchOCU \
     --start-time <30 days ago> --end-time <now> --period 3600 \
     --statistics Average Maximum --output json
   

   Record the configured minimums, the measured SearchOCU and IndexingOCU averages, and how many collections share the group; repeat the metric call for IndexingOCU. Aurora Serverless v2 bills ACU-hours plus storage — read the cluster's minimum capacity and the ServerlessDatabaseCapacity metric the same way. S3 Vectors bills storage plus requests, so it has no idle floor and its cost tracks query and put volume.

2. Embedding tokens at ingestion. Corpus tokens are roughly the bytes of extracted text divided by 4. Overlap repeats text: with fixed-size chunking at maxTokens M and overlapPercentage p as a fraction, consecutive chunks advance by M(1-p), so the corpus is embedded about 1/(1-p) times over — 1.25x at 20%, 2x at 50%. It is not 1+p, and the gap widens exactly where it costs most. Hierarchical chunking embeds both levels, so count parent and child separately. Multiply by the embedding price per 1,000 tokens. This is the cost of one full sync and of any re-embedding.

3. Semantic chunking, when configured. A chunkingStrategy of SEMANTIC calls a foundation model during ingestion and is charged for that on top of the embeddings, scaling with corpus size. Zero for every other strategy.

4. Parsing, when configured. Neither parser is charged per document, so the object count from Step 1 cannot price this line. BEDROCK_DATA_AUTOMATION is priced per page of the document and per image processed: multiply the page estimate and the image count recorded in Step 1 by the per-page and per-image rates. BEDROCK_FOUNDATION_MODEL is priced on the input and output tokens the parser model processes: estimate input tokens from the extracted text of the pages it parses, output tokens from the extraction it returns, and multiply by that model's input and output prices. On a large scanned-PDF corpus this can be the biggest single ingestion line, and per-document pricing understates a 200-page PDF by two orders of magnitude. Either parser, once set, is applied to every .pdf in the data source, text-only ones included, so the whole PDF count is in scope. The default parser adds nothing.

5. Embedding tokens per query. One query embedding per retrieve; small but nonzero.

6. Reranking, when configured. A rerankingConfiguration on the search configuration sends the candidate chunks through a reranker model on every query, which is charged per query and can rival the generation cost on a chatty application. Zero when rerankingModelType is NONE or no reranking is set.

7. Vector store request and data-processing charges. Separate from item 1's capacity or storage: S3 Vectors charges per query and per put, and the store's own data-processing charges scale with query volume. On a high-volume, small-corpus knowledge base these can exceed the embedding cost, which is why they cannot be left out of a per-query figure.

8. Generation tokens per query. Prompt tokens are the template plus the five chunks (chunk maxTokens times 5) plus the question; output tokens from the Step 6 answers. Multiply by the model's input and output prices.

Getting the store's real monthly figure. Filter Cost Explorer down to this store. A service-only filter returns the whole account's spend for that service, which on any shared account is not this knowledge base's bill:

eval "$(~/.vibestack/bin/vibe-slug 2>/dev/null)" 2>/dev/null || SLUG="unknown"
EVAL_DIR="${VIBESTACK_HOME:-$HOME/.vibestack}/projects/${SLUG:-unknown}"
mkdir -p "$EVAL_DIR"
CE_OUT="$EVAL_DIR/kb-cost-$(date +%Y-%m-%d).json"
CE_ERR=$(aws ce get-cost-and-usage \
  --time-period Start=<first of last month>,End=<first of this month> \
  --granularity MONTHLY --metrics UnblendedCost \
  --filter '{"And":[{"Dimensions":{"Key":"SERVICE","Values":["Amazon OpenSearch Service"]}},{"Tags":{"Key":"<cost-allocation-tag>","Values":["<value>"]}}]}' \
  --group-by Type=DIMENSION,Key=USAGE_TYPE \
  --output json 2>&1 > "$CE_OUT")
rc=$?
[ "$rc" -ne 0 ] && printf 'COST: N-A (exit %s)\n%s\n' "$rc" "$CE_ERR"

--filter takes Dimensions, Tags and CostCategories combined with And, Or and Not, so swap the tag clause for a cost category when the account uses one. The tag has to be activated as a cost allocation tag and only covers usage from its activation date. Cost Explorer is a separate permission from the rest of this review: on a non-zero exit, or when the store carries no tag or category that isolates it, report the store cost as N-A and price it instead from the measured usage in item 1 against published unit rates. Say in the report which of the two routes produced the number. Never print an unfiltered service total as this store's cost.

Report a per-1000-queries figure: fixed monthly cost of the store divided by the observed or expected monthly query volume, plus the per-query items. Findings: MEDIUM if the store's idle floor is more than half of the monthly bill at the current query volume (S3 Vectors or pgvector on an existing database would cost less) — this one needs both the configured minimum capacity and a month of OCU or ACU metrics showing usage sitting at that minimum; with only one of the two it is LOW and written as a question for the user, not a verdict. LOW if prompt tokens per query are over 4000 because of large chunks times numberOfResults.

Output

eval "$(~/.vibestack/bin/vibe-slug 2>/dev/null)" 2>/dev/null || SLUG="unknown"
EVAL_DIR="${VIBESTACK_HOME:-$HOME/.vibestack}/projects/${SLUG:-unknown}"
mkdir -p "$EVAL_DIR"
REPORT="$EVAL_DIR/kb-review-$(date +%Y-%m-%d).md"
echo "REPORT: $REPORT"

Write the report to $REPORT (overwrite today's if it exists) and print it in full.

KNOWLEDGE BASE REVIEW
=====================

Account:   <alias or last 4 digits> (profile: <name>)     Region: <region>
Date:      2026-08-14
Mode:      Bedrock knowledge base (type VECTOR) | hand-built pipeline

KB SUMMARY
Field              Value
-----------------  ----------------------------------------------
Knowledge base     kb-docs-prod (ABCD1234), type VECTOR
Vector store       OpenSearch Serverless, group min 2 index / 2 search OCU shared
                   with 3 collections, 1024-dim index
Embedding model    amazon.titan-embed-text-v2:0, 1024 dims
Data sources       s3://docs/tenant-*/ (1,240 PDF, 380 DOCX, 4.1 GB), confluence ENG space
Chunking           hierarchical, parent 1500 / child 300 tokens, overlap 60
Parsing            none (default text extraction)
Isolation          metadata filter on tenant (no ACL awareness configured)
Metadata           tenant, classification on 1,580 of 1,620 documents
Deletion policy    RETAIN
Last sync          2026-08-12 03:10 UTC, 0 failed, EventBridge nightly
Languages          de 70% / en 30%

FINDINGS
#  Severity  Area        Evidence                                              Fix
-  --------  ----------  ----------------------------------------------------  --------------------------------------------
1  CRITICAL  isolation   api/search.py:88 retrieve() has no filter             derive filter from the session tenant
2  HIGH      sync        dataDeletionPolicy RETAIN; legal deletes monthly      set DELETE, re-run sync, verify vector count
3  MEDIUM    chunking    table values in pricing.pdf split across 3 chunks     parsingConfiguration with a foundation model
4  MEDIUM    metadata    40 documents without .metadata.json                   generate sidecars in the export job
5  MEDIUM    cost        min 2 OCU idle = 68% of tagged spend; SearchOCU avg 2.0  evaluate S3 Vectors for this volume

RETRIEVAL METRICS (golden set: 32 questions, 5 unanswerable, 0 retrieval errors)
Subset      n    doc hit@5   passage hit@5   MRR    false positives
----------  ---  ---------   -------------   ----   ---------------
all         27   0.81        0.70            0.55   1 of 5
de          19   0.79        0.68            0.52   -
en           8   0.88        0.75            0.63   -
Faithfulness (10 answers, <model id>): 7 faithful, 2 partly, 1 wrong

COST (prices read 2026-08-14; store cost from Cost Explorer, tag kb=docs-prod)
Item                            Monthly      Per 1,000 queries
------------------------------

…(truncated)