Moodle Coderunner

CodeRunner Question Generator (Minimal Pipeline)

Generate Moodle CodeRunner questions. Two modes depending on your environment.

Mode Detection

Before doing anything else, determine your mode:

Run this command:

echo "jobe-check"

• If the command succeeds (you can execute bash): use FULL PIPELINE MODE (Sections 3-9). Jobe computes all expected output via curl. This is the preferred mode.
• If the command fails (no bash access, e.g. Claude.ai or ChatGPT): use FALLBACK MODE (see box below). You must mentally trace expected output yourself.

FALLBACK MODE (no bash access)

When you cannot run bash commands (Claude.ai, ChatGPT, API without tools):

1. Generate the same JSON structure as Section 6, but you MUST include "expected" in each test case by carefully tracing the solution through the test code.
2. Trace character-by-character. Pay special attention to: trailing spaces, newlines, floating-point precision, array formatting.
3. Apply all per-language rules from Section 2 and auto-fix rules from Section 8 manually.
4. Wrap in XML using the template from Section 7.
5. Add this warning to the user: "These questions were generated WITHOUT Jobe server validation. Before using them, import into Moodle and ensure Validate on save is enabled (it is by default). Moodle will run the solution against all test cases on import and flag any mismatches."
6. All other sections (question design, type rules, XML template, checklist) still apply.

Full Pipeline mode is strongly preferred. It eliminates the #1 source of errors (hallucinated expected output, which causes 35% of all failures).

1. Question Type Reference

Choose the type FIRST -- it determines everything else.

Default choice: Prefer java_class > java_method > java_program. Prefer function types over program types -- they avoid stdin/EOF problems entirely.

2. Per-Language Rules

These rules are derived from 1000+ validated questions. Every rule exists because questions failed without it.

Java

Python

C

C++

Node.js

3. Jobe Server Configuration

Servers

Try the primary server first. If it fails (timeout or HTTP error), fall back to the backup.

Language IDs for Jobe

Calling Jobe via curl

Use this exact curl pattern for every Jobe call:

curl -s -w "\n%{http_code}" \
  https://jobe2.cosc.canterbury.ac.nz/jobe/index.php/restapi/runs \
  -H "Content-Type: application/json" \
  -d '{
    "run_spec": {
      "language_id": "LANGUAGE_ID",
      "sourcecode": "FULL_SOURCE_CODE",
      "input": "STDIN_OR_EMPTY",
      "sourcefilename": "FILENAME",
      "parameters": {"cputime": 10, "memorylimit": 256000}
    }
  }'

Response format:

{"outcome": 15, "cmpinfo": "", "stdout": "42\n", "stderr": ""}

Outcome codes:

• 15 = success. Use stdout as the expected output.
• 11 = compilation error. Read cmpinfo for the error.
• 12 = runtime error. Read stderr for the error.
• 13 = time limit exceeded.
• 17 = memory limit exceeded.
• Any other outcome = failure.

Important: Stdin MUST end with \n. If the input does not end with a newline, append one before sending.

Important: Expected output: strip the trailing newline from stdout before using it as expected output. CodeRunner strips trailing newlines before comparing, so the <expected> tag should NOT include a trailing newline.

Escaping for curl -d

The sourcecode and input fields are inside a JSON string inside a bash command. You MUST properly escape:

• Backslashes: \ becomes \\
• Double quotes: " becomes \"
• Newlines: actual newlines become \n
• Tabs: actual tabs become \t
• Dollar signs in bash: use single-quoted heredoc to avoid shell expansion

Recommended pattern: Write the JSON payload to a temp file, then curl with --data @file:

# Write payload to temp file (avoids all escaping issues)
cat > /tmp/jobe_payload.json << 'PAYLOAD'
{
  "run_spec": {
    "language_id": "python3",
    "sourcecode": "def add(a, b):\n    return a + b\n\nprint(add(2, 3))",
    "input": "",
    "sourcefilename": "test.py",
    "parameters": {"cputime": 10, "memorylimit": 256000}
  }
}
PAYLOAD

curl -s https://jobe2.cosc.canterbury.ac.nz/jobe/index.php/restapi/runs \
  -H "Content-Type: application/json" \
  --data @/tmp/jobe_payload.json

For complex source code (multi-line, quotes, special chars), ALWAYS use the temp file approach.

4. Building Source Code

Before calling Jobe, assemble the full source code from the solution and test code. The assembly rules depend on the question type.

Assembly Rules by Type

java_method

Wrap the solution methods inside a class, add main() with the test code:

public class Answer {
    SOLUTION_METHODS

    public static void main(String[] args) throws Exception {
        TESTCODE
    }
}

• Filename: Answer.java
• Stdin: empty

java_class

Strip public from the student's class declaration, append a tester class:

class ClassName {
    // solution with 'public class' changed to 'class'
}

public class __Tester__ {
    public static void main(String[] args) throws Exception {
        TESTCODE
    }
}

• Filename: __Tester__.java
• Stdin: empty
• Important: Test code must use the class name qualifier: ClassName.method() or ClassName obj = new ClassName();

java_program

Use the solution as-is. Extract the public class name for the filename.

• Filename: {ClassName}.java (extracted from public class X, default Answer)
• Stdin: test input
• Test code: empty

python3

Concatenate solution and test code:

SOLUTION

TESTCODE

• Filename: test.py
• Stdin: empty

python3_w_input

Use solution as-is, pass stdin separately.

• Filename: test.py
• Stdin: test input
• Test code: empty

c_function / cpp_function

Concatenate solution and test code:

SOLUTION

TESTCODE

• Filename: test.c or test.cpp
• Stdin: empty
• Important: Test code must include #include directives and int main().

c_program / cpp_program

Use solution as-is, pass stdin separately.

• Filename: test.c or test.cpp
• Stdin: test input
• Test code: empty

nodejs

Concatenate solution and test code:

SOLUTION

TESTCODE

• Filename: test.js
• Stdin: empty

5. Question Design

Every question MUST include:

1. name -- Micro-Assessment - Descriptive Title (unique, not "Question 1")
2. question_text -- Clear HTML instructions: method names, parameter types, return types, example usage. Include penalty regime text (see below).
3. solution -- Complete, correct model solution (NEVER empty)
4. preload -- Compilable skeleton with comments (do not give away the solution structure)
5. feedback -- Explanation of correct approach (shown after quiz closes)
6. Test cases -- 3+ test cases: 1 visible, 2+ hidden, include edge cases
7. Self-contained -- never reference lectures or slides

Penalty Regime Text (include in every question_text)

<p><strong>Penalty Regime:</strong></p>
<ul>
<li>You have a total of <strong>3 attempts</strong> without any penalties.</li>
<li>Starting from the <strong>4th attempt onwards</strong>, a penalty of <strong>10%</strong> will be applied.</li>
</ul>

Test Case Requirements

• Minimum 3 test cases
• First test case: visible (useasexample="1", display SHOW)
• Remaining test cases: hidden (useasexample="0", display HIDE)
• Each test MUST produce DIFFERENT output (prevents hard-coding)
• Include at least one edge case (empty input, zero, negative, boundary)
• For program types with stdin: include empty stdin test case
• For partial credit (allornothing="0"): marks MUST sum to 1.0
• Do NOT include expected output in the JSON -- Jobe will compute it

6. Workflow (Full Pipeline Mode)

This workflow assumes you passed the mode detection check and can run bash commands. If you cannot, follow Fallback Mode described at the top.

Step 1: Gather Requirements

Ask the user for:

• Language: Java, Python, C, C++, Node.js (and specific type if they have a preference)
• Topic: What the questions should be about
• Count: How many questions (default: 5)
• Difficulty: easy / medium / hard
• Grading mode: all-or-nothing (default) or partial credit

Step 2: Generate JSON for All Questions

For each question, generate a JSON object with this structure:

{
  "name": "Micro-Assessment - Descriptive Title",
  "qtype": "java_class",
  "question_text": "<h4>Title</h4><p>Instructions with examples...</p><p><strong>Penalty Regime:</strong></p><ul><li>You have a total of <strong>3 attempts</strong> without any penalties.</li><li>Starting from the <strong>4th attempt onwards</strong>, a penalty of <strong>10%</strong> will be applied.</li></ul>",
  "feedback": "<p>Explanation of the correct approach.</p>",
  "solution": "complete model solution code",
  "preload": "skeleton code for students",
  "test_cases": [
    {"testcode": "System.out.println(obj.method(arg));", "stdin": "", "visible": true},
    {"testcode": "System.out.println(obj.method(arg2));", "stdin": "", "visible": false},
    {"testcode": "System.out.println(obj.method(edge));", "stdin": "", "visible": false}
  ]
}

Do NOT include expected in test_cases. Jobe will compute it.

For program types (java_program, python3_w_input, c_program, cpp_program):

• testcode is empty string ""
• stdin contains the input data

Step 3: Validate Each Test Case on Jobe

For each question, for each test case:

1. Build the full source code using the assembly rules from Section 4
2. Determine the language_id from Section 3
3. Determine the filename from Section 4
4. Write the Jobe payload to a temp file
5. Call Jobe via curl
6. Check the outcome:
   • If outcome is 15: strip the trailing newline from stdout and save it as the expected output for this test case
   • If outcome is NOT 15: the test case failed -- see Step 4

Example -- validating a python3 question:

# Question: def add(a, b): return a + b
# Test case: print(add(2, 3))
# Built source: "def add(a, b):\n    return a + b\n\nprint(add(2, 3))"

cat > /tmp/jobe_payload.json << 'PAYLOAD'
{
  "run_spec": {
    "language_id": "python3",
    "sourcecode": "def add(a, b):\n    return a + b\n\nprint(add(2, 3))",
    "input": "",
    "sourcefilename": "test.py",
    "parameters": {"cputime": 10, "memorylimit": 256000}
  }
}
PAYLOAD

curl -s https://jobe2.cosc.canterbury.ac.nz/jobe/index.php/restapi/runs \
  -H "Content-Type: application/json" \
  --data @/tmp/jobe_payload.json

Response: {"outcome":15,"cmpinfo":"","stdout":"5\n","stderr":""}

Expected output for this test case: 5 (stdout with trailing newline stripped).

Example -- validating a java_class question:

# Solution: public class Calculator { public int add(int a, int b) { return a + b; } }
# Test case: Calculator c = new Calculator(); System.out.println(c.add(2, 3));
# Built source (java_class assembly):
#   class Calculator { public int add(int a, int b) { return a + b; } }
#   public class __Tester__ {
#       public static void main(String[] args) throws Exception {
#           Calculator c = new Calculator(); System.out.println(c.add(2, 3));
#       }
#   }

cat > /tmp/jobe_payload.json << 'PAYLOAD'
{
  "run_spec": {
    "language_id": "java",
    "sourcecode": "class Calculator { public int add(int a, int b) { return a + b; } }\n\npublic class __Tester__ {\n    public static void main(String[] args) throws Exception {\n        Calculator c = new Calculator(); System.out.println(c.add(2, 3));\n    }\n}",
    "input": "",
    "sourcefilename": "__Tester__.java",
    "parameters": {"cputime": 10, "memorylimit": 256000}
  }
}
PAYLOAD

curl -s https://jobe2.cosc.canterbury.ac.nz/jobe/index.php/restapi/runs \
  -H "Content-Type: application/json" \
  --data @/tmp/jobe_payload.json

Step 4: Fix and Retry on Failure

If Jobe returns a non-15 outcome for any test case:

1. Read the error from cmpinfo (compile errors) or stderr (runtime errors)
2. Diagnose the issue using the common fixes table below
3. Fix the solution or test code
4. Rebuild the source and call Jobe again
5. Repeat up to 3 times per question

If a question still fails after 3 retries, report it to the user and move on.

Common Fixes

Step 5: Assemble the XML

Once ALL test cases for ALL questions have passed Jobe validation, wrap everything in the Moodle XML template from Section 7.

For each test case, the <expected> value is the stdout captured from Jobe (with trailing newline stripped).

Step 6: Write and Confirm

1. Write the XML to a file named {module}_{topic}_coderunner.xml
2. Report to the user:
   • Number of questions generated
   • Number that passed validation
   • Any that failed and were excluded
   • The file path

7. XML Template

<?xml version="1.0" encoding="UTF-8"?>
<quiz>
  <question type="category">
    <category><text>$course$/Category Name</text></category>
  </question>

  <!-- Repeat this block for each question -->
  <question type="coderunner">
    <name><text>QUESTION_NAME</text></name>
    <questiontext format="html"><text><![CDATA[QUESTION_HTML]]></text></questiontext>
    <generalfeedback format="html"><text><![CDATA[FEEDBACK_HTML]]></text></generalfeedback>
    <defaultgrade>1</defaultgrade>
    <penalty>0</penalty>
    <hidden>0</hidden>
    <idnumber></idnumber>
    <coderunnertype>QTYPE</coderunnertype>
    <prototypetype>0</prototypetype>
    <allornothing>1</allornothing>
    <penaltyregime>0, 0, 0, 10, 20, ...</penaltyregime>
    <precheck>0</precheck>
    <hidecheck>0</hidecheck>
    <showsource>0</showsource>
    <answerboxlines>18</answerboxlines>
    <answerboxcolumns>100</answerboxcolumns>
    <answerpreload><![CDATA[PRELOAD_CODE]]></answerpreload>
    <globalextra></globalextra>
    <useace></useace>
    <resultcolumns></resultcolumns>
    <template></template>
    <iscombinatortemplate></iscombinatortemplate>
    <allowmultiplestdins></allowmultiplestdins>
    <answer><![CDATA[MODEL_SOLUTION]]></answer>
    <validateonsave>1</validateonsave>
    <testsplitterre></testsplitterre>
    <language></language>
    <acelang></acelang>
    <sandbox></sandbox>
    <grader></grader>
    <cputimelimitsecs></cputimelimitsecs>
    <memlimitmb></memlimitmb>
    <sandboxparams></sandboxparams>
    <templateparams></templateparams>
    <hoisttemplateparams>1</hoisttemplateparams>
    <extractcodefromjson>1</extractcodefromjson>
    <templateparamslang>None</templateparamslang>
    <templateparamsevalpertry>0</templateparamsevalpertry>
    <templateparamsevald>{}</templateparamsevald>
    <twigall>0</twigall>
    <uiplugin></uiplugin>
    <uiparameters><![CDATA[{"live_autocompletion": true}]]></uiparameters>
    <attachments>0</attachments>
    <attachmentsrequired>0</attachmentsrequired>
    <maxfilesize>10240</maxfilesize>
    <filenamesregex></filenamesregex>
    <filenamesexplain></filenamesexplain>
    <displayfeedback>1</displayfeedback>
    <giveupallowed>0</giveupallowed>
    <prototypeextra></prototypeextra>
    <testcases>
      <!-- Repeat for each test case -->
      <testcase testtype="0" useasexample="USE_AS_EXAMPLE" hiderestiffail="0" mark="MARK_VALUE">
        <testcode><text>TESTCODE</text></testcode>
        <stdin><text>STDIN</text></stdin>
        <expected><text>EXPECTED_FROM_JOBE</text></expected>
        <extra><text></text></extra>
        <display><text>DISPLAY_VALUE</text></display>
      </testcase>
    </testcases>
  </question>
</quiz>

Field Values per Test Case

For partial credit: distribute marks so they sum to 1.0 (e.g., 4 tests = 0.2500000 each).

Optional Feature Tags

8. Auto-Fix Rules to Apply Before Jobe

Before sending code to Jobe, apply these fixes to the solution and test code. These catch the most common AI generation mistakes.

Solution Fixes

Test Code Fixes

Apply these fixes programmatically before building the source and calling Jobe.

9. Complete Worked Example

Here is a complete end-to-end example generating one Python question.

9a. Generate JSON

{
  "name": "Micro-Assessment - Sum of Even Numbers",
  "qtype": "python3",
  "question_text": "<h4>Sum of Even Numbers</h4><p>Write a function <code>sum_evens(nums)</code> that takes a list of integers and returns the sum of all even numbers in the list.</p><h5>Example</h5><pre>sum_evens([1, 2, 3, 4, 5, 6]) -> 12</pre><p><strong>Penalty Regime:</strong></p><ul><li>You have a total of <strong>3 attempts</strong> without any penalties.</li><li>Starting from the <strong>4th attempt onwards</strong>, a penalty of <strong>10%</strong> will be applied.</li></ul>",
  "feedback": "<p>Use a list comprehension or loop to filter even numbers (n % 2 == 0) and sum them. The built-in <code>sum()</code> function works well with a generator expression.</p>",
  "solution": "def sum_evens(nums):\n    return sum(n for n in nums if n % 2 == 0)",
  "preload": "def sum_evens(nums):\n    # Write your code here\n    pass",
  "test_cases": [
    {"testcode": "print(sum_evens([1, 2, 3, 4, 5, 6]))", "stdin": "", "visible": true},
    {"testcode": "print(sum_evens([]))", "stdin": "", "visible": false},
    {"testcode": "print(sum_evens([1, 3, 5]))", "stdin": "", "visible": false},
    {"testcode": "print(sum_evens([-2, -1, 0, 1, 2]))", "stdin": "", "visible": false}
  ]
}

9b. Validate Test Case 1 on Jobe

Build source (python3 = solution + "\n\n" + testcode):

def sum_evens(nums):
    return sum(n for n in nums if n % 2 == 0)

print(sum_evens([1, 2, 3, 4, 5, 6]))

cat > /tmp/jobe_payload.json << 'PAYLOAD'
{
  "run_spec": {
    "language_id": "python3",
    "sourcecode": "def sum_evens(nums):\n    return sum(n for n in nums if n % 2 == 0)\n\nprint(sum_evens([1, 2, 3, 4, 5, 6]))",
    "input": "",
    "sourcefilename": "test.py",
    "parameters": {"cputime": 10, "memorylimit": 256000}
  }
}
PAYLOAD

curl -s https://jobe2.cosc.canterbury.ac.nz/jobe/index.php/restapi/runs \
  -H "Content-Type: application/json" \
  --data @/tmp/jobe_payload.json

Result: {"outcome":15,"stdout":"12\n",...} -- expected output: 12

9c. Repeat for All Test Cases

• Test 2: sum_evens([]) -- stdout 0\n -- expected: 0
• Test 3: sum_evens([1, 3, 5]) -- stdout 0\n -- expected: 0
• Test 4: sum_evens([-2, -1, 0, 1, 2]) -- stdout 0\n -- expected: 0

9d. Wrap in XML

Fill in the XML template from Section 7:

• QUESTION_NAME = Micro-Assessment - Sum of Even Numbers
• QTYPE = python3
• QUESTION_HTML = the question_text HTML
• FEEDBACK_HTML = the feedback HTML
• MODEL_SOLUTION = the solution code
• PRELOAD_CODE = the preload code
• For each test case: fill TESTCODE, STDIN, EXPECTED_FROM_JOBE, USE_AS_EXAMPLE, DISPLAY_VALUE

10. Final Checklist

Before delivery, every question must satisfy:

• <name> tag uses descriptive title starting with Micro-Assessment -
• Correct <coderunnertype> for the task
• <answer> contains complete, compilable model solution
• <answer> for java_method is raw method(s) only (no class wrapper)
• <answerpreload> provides compilable skeleton
• <generalfeedback> explains correct approach
• <validateonsave>1</validateonsave>
• <penaltyregime>0, 0, 0, 10, 20, ...</penaltyregime>
• Penalty regime text in <questiontext>
• Self-contained (no lecture references)
• 3+ test cases: 1 visible, 2+ hidden, edge cases included
• Each test produces different output
• No floating-point exact comparisons
• Marks sum to 1.0 (if partial credit)
• Language-specific rules from Section 2 followed
• Auto-fix rules from Section 8 applied before Jobe calls
• ALL test cases returned outcome 15 from Jobe (Full Pipeline mode)
• ALL expected values are Jobe stdout (never hand-written) (Full Pipeline mode)
• If Fallback Mode: warned user to validate on import with Validate on save enabled

11. File Naming

• Questions: {module}_{topic}_coderunner.xml
• Example: oop1_polymorphism_coderunner.xml

12. Moodle Import

1. Course > Question bank > Import
2. Format: Moodle XML
3. Upload .xml file
4. Import and review