Sap LLM Engineering

SAP LLM Engineering — Eval-First Patterns for SAP AI

Adapts Karpathy's LLM engineering methodology (build eval first, iterate on metrics not vibes) to SAP ABAP/BTP/CPI contexts. The eval IS the product — prompts are just configuration.

Prerequisites

• abaplint installed and pinned to a specific version (rule sets evolve)
• SAP ADT or abaplint CLI for syntax validation
• LLM API access (for generation + optional embedding)
• sap-crew-analysis skill loaded (companion for code-quality scoring)
• 20% of training data held out as eval set — never evaluate on training prompts

Core Principle

Build eval → Measure baseline → Improve prompt → Re-measure → Repeat
  (NOT: Write prompt → Hope it works → Manual review)

Steps

1. Build Evaluation Harness (Before Any Prompt Work)

Define what "good ABAP" means across 7 dimensions:

EVAL_DIMENSIONS = {
    'abaplint_pass':    'abaplint --format json → zero errors',
    'syntax_ok':        'ADT syntax check passes',
    'unit_test_pass':   'ABAP Unit test methods all green',
    'clean_abap_score': 'Clean ABAP compliance ≥ 70/100',
    'bapi_correctness': 'Import/export params match BAPI signature',
    'security_ok':      'No SQL injection, no auth-check gaps',
    'performance_ok':   'No SELECT *, no SELECT-in-loop',
}

Run eval on every LLM output:

def evaluate(generated_code: str, spec: dict) -> dict:
    results = {
        'abaplint': run_abaplint(generated_code),
        'syntax': run_adt_syntax_check(generated_code),
        'security': audit_security(generated_code),
        'performance': audit_performance(generated_code),
    }
    if spec.get('bapi_name'):
        results['bapi'] = verify_bapi_signature(generated_code, spec['bapi_name'])
    return {
        'pass': all(r['pass'] for r in results.values()),
        'score': sum(r['score'] for r in results.values()) / len(results),
        'details': results,
    }

2. Establish Baseline

# Run current prompt against eval set, record scores
python eval_abap_llm.py --prompt baseline.json --eval-set holdout_20pct.json
# Output: baseline_score=0.42 — security dimension failing 80% of cases

3. Optimize Prompt (Iterate on Failures)

# prompt_optimizer.py — fix prompt, not output
PROMPT_BAPI = """
You are an expert ABAP developer. Write a method calling {bapi_name}.
Requirements:
- Modern ABAP (inline declarations, NEW constructor)
- Always call BAPI_TRANSACTION_COMMIT with WAIT = 'X'
- Check BAPIRET2 tables, not just RETURN
- Handle errors with CX_STATIC_CHECK
- Include ABAP Unit test class with {min_tests} methods
ABAP version: {version}  Package: {package}
"""

def optimize(template, spec, eval_fn, max_iter=10):
    best_score, best_prompt = 0, None
    for i in range(max_iter):
        code = call_llm(template.format(**spec))
        result = eval_fn(code, spec)
        if result['score'] > best_score:
            best_score, best_prompt = result['score'], template
        if result['pass']:
            break
        spec = inject_failure_feedback(spec, result)  # feed failures back
    return best_prompt, best_score

4. Curate Training Data (For Fine-Tuning)

Extract (instruction, signature, body) triples from existing ABAP repos:

QUALITY_FILTERS = [
    'abaplint_pass',           # must pass static analysis
    'syntax_check_pass',        # must compile
    'unit_test_pass',           # tests must pass
    'not_generated_code',       # exclude SAP standard generated code
    'clean_abap_score >= 70',   # quality threshold
]
# Sources: method implementations, BAPI wrappers, RAP behaviors, CDS views, unit tests

5. Build RAG Pipeline for SAP Knowledge

Chunk SAP docs by entity type for better retrieval:

CHUNK_STRATEGIES = {
    'bapi':       {'split_on': 'FUNCTION ',    'max': 4000, 'meta': ['BAPI_NAME', 'MODULE']},
    'cds_view':   {'split_on': 'DEFINE VIEW ', 'max': 2000, 'meta': ['ENTITY_NAME']},
    'sap_note':   {'split_on': '## Solution',  'max': 8000, 'meta': ['NOTE_NUMBER', 'VERSION']},
    'abap_class': {'split_on': 'METHOD ',      'max': 1500, 'meta': ['CLASS_NAME', 'METHOD_NAME']},
}

Query with context:

def generate_with_rag(question: str) -> str:
    context = vector_store.similarity_search(question, k=5)
    return call_llm(build_rag_prompt(question, context))

6. Evaluate Agent (Not Just Code)

AGENT_EVAL_CASES = [
    {'task': 'Create ZCL_MATERIAL_HELPER with BAPI_MATERIAL_SAVEDATA wrapper',
     'criteria': ['syntax_valid', 'abaplint_pass', 'bapi_params_correct',
                  'has_error_handling', 'has_unit_test', 'clean_abap_70plus']},
    {'task': 'Debug ST22 dump SYNTAX_ERROR in ZREPORT_MM line 245',
     'criteria': ['reads_dump', 'identifies_root_cause', 'proposes_fix']},
]
# Run 3× per case, average scores, compute weighted total

7. Integration with sap-crew-analysis

sap-llm-engineering optimizes the PROMPT; sap-crew-analysis evaluates the CODE. Loop: optimize prompt → generate → evaluate code → feed back → iterate.

Pitfalls

• Eval data leakage inflates scores

  • Cause: Eval prompts overlap with training/few-shot examples in the prompt.
  • Solution: Hold out 20% of cases in a separate file. Never include them in few-shot examples.

• LLM hallucinates non-existent BAPI names

  • Cause: Generic LLMs trained on web data invent plausible-sounding BAPI names.
  • Solution: Validate every BAPI name against the SAP system via mcp_sap_bapi_call or RFC_FUNCTION_SEARCH before accepting generated code.

• abaplint rule changes break eval reproducibility

  • Cause: abaplint updates change rule severity between versions.
  • Solution: Pin abaplint version in requirements.txt. Re-baseline when upgrading.

• Security dimension silently skipped

  • Cause: Security checks are slower and teams deprioritize them.
  • Solution: Make security_ok a hard gate — eval fails if security check doesn't run. LLMs can introduce SQL injection in ABAP via dynamic WHERE clauses.

• Token budget exhausted before convergence

  • Cause: Full eval harness burns ~50K tokens per iteration × 10+ iterations.
  • Solution: Budget 15-20 iterations. Use cheaper model for initial screening, expensive model for final candidates.

• Generic code LLMs score poorly on ABAP

  • Cause: ABAP syntax is underrepresented in general training data.
  • Solution: Use extensive few-shot examples (5+ ABAP snippets). Consider fine-tuned models or RAG-augmented generation.

Verification

• Eval harness covers all 7 dimensions (abaplint, syntax, unit test, clean ABAP, BAPI, security, performance)
• Baseline score recorded before any prompt optimization
• Eval set is separate from training/few-shot data (20% holdout)
• abaplint version pinned in project dependencies
• Every generated BAPI name validated against live SAP system
• Security dimension is a hard gate (not optional)
• Agent eval run 3× per case with averaged scores
• Prompt revision history tracked (iteration, score, failures)
• RAG chunks tagged with entity-type metadata for filtering