# Dabstep Dataset Metadata Business Rules

> Solve questions about the dabstep payment dataset's metadata, schema, column definitions, and business rules. Use this skill when answering questions about column names/meanings, fee structures, how factors affect fees (is_credit, intracountry, monthly_fraud_level, monthly_volume, capture_delay), fee thresholds, volume tiers, or any business rule from the manual. Trigger on any dabstep question involving "column", "field", "fee", "factor", "cheaper", "threshold", "volume tier", "boolean", "rule", or asking about dataset structure/documentation.

- Skill: `zjunlp/dabstep-dataset-metadata-business-rules-2` (Agent Skill)
- Install (CLI): `npx skillmds@latest add zjunlp/dabstep-dataset-metadata-business-rules-2`
- Raw SKILL.md: https://api.skillmd.com/api/skills/zjunlp/dabstep-dataset-metadata-business-rules-2/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: zjunlp (https://skillmd.com/u/zjunlp)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/zjunlp/dabstep-dataset-metadata-business-rules-2

---


# Dataset Metadata and Business Rules — DABSTEP

## Dataset Files

| File | Purpose |
|------|---------|
| `payments.csv` | Transaction records (21 columns) |
| `fees.json` | Fee rules (~1000 rules, JSON array) |
| `manual.md` | Business rules and field definitions |
| `payments-readme.md` | Column descriptions for payments.csv |
| `merchant_data.json` | Merchant-level information |
| `merchant_category_codes.csv` | MCC code lookup |
| `acquirer_countries.csv` | Acquirer country codes |

## Core Fee Formula

```
fee = fixed_amount + rate * transaction_value / 10000
```

`fees.json` rules contain these fields:

| Field | Type | Description |
|-------|------|-------------|
| `fixed_amount` | float | Flat fee per transaction (euros) |
| `rate` | int | Variable rate × transaction_value / 10000 |
| `is_credit` | bool | True = credit card (typically more expensive) |
| `intracountry` | bool | True = domestic (same issuer/acquirer country; cheaper) |
| `monthly_fraud_level` | str | Fraud ratio tier: `<7.2%`, `7.2%-7.7%`, `7.7%-8.3%`, `>8.3%` |
| `monthly_volume` | str | Volume tier: `<100k`, `100k-1m`, `1m-5m`, `>5m` (euros) |
| `capture_delay` | str | `immediate`, `<3`, `3-5`, `>5`, `manual` |
| `card_scheme` | str | MasterCard, Visa, Amex, Other |
| `account_type` | list | R, D, H, F, S, O |
| `merchant_category_code` | list | MCC integers |
| `aci` | list | Authorization Characteristics Indicator values |
| `ID` | int | Rule identifier |

**Null values** in any field mean the rule applies to ALL values of that field.

## payments.csv Key Columns

| Column | Type | Description |
|--------|------|-------------|
| `psp_reference` | ID | Unique payment identifier |
| `has_fraudulent_dispute` | Boolean | Fraudulent dispute flagged by issuing bank |
| `is_refused_by_adyen` | Boolean | Adyen refusal indicator |
| `is_credit` | Categorical | Credit or debit card |
| `eur_amount` | Numeric | Payment amount in euros |
| `card_scheme` | Categorical | MasterCard, Visa, Amex, Other |
| `aci` | Categorical | Authorization Characteristics Indicator |
| `acquirer_country` | Categorical | Acquiring bank country |
| `issuing_country` | Categorical | Card-issuing country |
| `merchant` | Categorical | Merchant name |

## Factor Directionality (from manual.md)

These are the definitive business rule statements for how each factor affects fees:

| Factor | Manual Statement | Cheaper When |
|--------|-----------------|--------------|
| `is_credit` | "credit transactions are more expensive" | `is_credit = False` (debit) |
| `intracountry` | "international transactions are typically more expensive" | `intracountry = True` (domestic) |
| `monthly_fraud_level` | "more expensive as fraud rate increases" | Decrease fraud level |
| `monthly_volume` | "higher volume → cheaper fees" | Increase volume |
| `capture_delay` | "faster capture is more expensive" | Increase delay (slower) |

**Key rule**: When asked "which boolean factor, when set to True, leads to cheaper fees?" — only `intracountry=True` is cheaper (domestic). `is_credit=True` is more expensive.

**Key rule**: When asked "which factors, when DECREASED, lead to cheaper fees?" — `monthly_fraud_level` (lower fraud = cheaper) and `is_credit` (False < True in terms of cost, so decreasing/switching to False is cheaper).

## Strategy by Question Type

### Type 1: Column/Field Identification
*"What column indicates X?" / "What field stores Y?"*

1. Read `payments-readme.md` for payments.csv columns
2. Read `manual.md` Section 5 for fees.json field definitions
3. Match the described concept to the exact column name

```python
# Load and inspect
import pandas as pd, json
df = pd.read_csv('payments.csv', nrows=3)
print(df.columns.tolist())

with open('fees.json') as f:
    fees = json.load(f)
print(list(fees[0].keys()))
```

### Type 2: Fee/Rule Existence Questions
*"Does fee X exist?" / "How much is fee Y?"*

Search strategy:
1. Search `manual.md` for the concept (e.g., grep for "retry", "refund", "chargeback")
2. Inspect `fees.json` field names — the schema defines what fee types are modeled
3. If the manual mentions a behavior causally (e.g., "excessive retrying causes downgrades") but no numeric fee is defined → answer **"Not Applicable"**
4. If the concept is absent entirely → answer **"Not Applicable"**

```python
import json
with open('fees.json') as f:
    fees = json.load(f)
# Check available fields
print(list(fees[0].keys()))
# Fields present: ID, card_scheme, account_type, capture_delay,
# monthly_fraud_level, monthly_volume, merchant_category_code,
# is_credit, aci, fixed_amount, rate, intracountry
# No "retry_fee", "refund_fee", etc.
```

### Type 3: Boolean Factor Analysis
*"Which boolean factors contribute to cheaper fees when True/False?"*

**Always trust manual.md over empirical data averages.**

Empirical mean/median comparisons across all rules are unreliable because rules for different boolean values serve different contexts. Use the manual's explicit directional statements.

```python
import json, pandas as pd
with open('fees.json') as f:
    fees = json.load(f)
df = pd.DataFrame(fees)

# For a proper comparison, find rule PAIRS differing only in the boolean
# Use a representative transaction amount for fee calculation
amount = 100  # euros
df['sample_fee'] = df['fixed_amount'] + df['rate'] * amount / 10000

# Compare within matched groups
non_bool_cols = ['card_scheme', 'account_type', 'capture_delay',
                 'monthly_fraud_level', 'monthly_volume',
                 'merchant_category_code', 'aci']
# Look at mean by boolean value as a sanity check, but defer to manual
for col in ['is_credit', 'intracountry']:
    print(f"\n{col}:")
    print(df.groupby(col)['sample_fee'].mean())
```

**Expected results per manual:**
- `is_credit=True` → higher fees; `is_credit=False` → cheaper
- `intracountry=True` → cheaper (domestic); `intracountry=False` → expensive (international)

### Type 4: Volume/Threshold Analysis
*"At what volume do fees become cheaper?" / "Highest volume where fees do NOT become cheaper?"*

Volume tier order (ascending): `<100k` → `100k-1m` → `1m-5m` → `>5m`

Manual rule: higher volume → cheaper fees (economies of scale). The pricing curve **flattens out** at higher volumes — meaning at the highest tier (`>5m`), fees cannot decrease further.

**Answering volume-boundary questions:**

- "At what volume do fees become cheaper?" — the tier(s) where lower rates apply; in practice `>5m` is the cheapest tier.
- "What is the highest volume at which fees do NOT become cheaper?" — Answer: **`>5m`**
  - Reasoning: `>5m` is the maximum volume tier. There is no higher tier to transition to, so fees cannot become cheaper at this volume level. The pricing curve has flattened at this ceiling.
  - Do NOT answer `1m-5m` — that is the tier below the cheapest, not the highest tier.

```python
import json
with open('fees.json') as f:
    fees = json.load(f)

# Enumerate all distinct monthly_volume tier values in logical order
tier_order = {'<100k': 0, '100k-1m': 1, '1m-5m': 2, '>5m': 3}
volumes = sorted({r['monthly_volume'] for r in fees if r.get('monthly_volume')},
                 key=lambda x: tier_order[x])
print(volumes)  # ['<100k', '100k-1m', '1m-5m', '>5m']
# Highest tier = volumes[-1] = '>5m'
```

### Type 5: Multi-Factor Direction Analysis
*"Which factors lead to cheaper fees when decreased?"*

For each candidate factor, determine if its value range has a monotonic direction:

```python
import json, pandas as pd
with open('fees.json') as f:
    fees = json.load(f)
df = pd.DataFrame(fees)

amount = 100
df['sample_fee'] = df['fixed_amount'] + df['rate'] * amount / 10000

# Define ordered tiers for categorical factors
vol_order = {'<100k': 0, '100k-1m': 1, '1m-5m': 2, '>5m': 3}
fraud_order = {'<7.2%': 0, '7.2%-7.7%': 1, '7.7%-8.3%': 2, '>8.3%': 3}
delay_order = {'immediate': 0, '<3': 1, '3-5': 2, '>5': 3, 'manual': 4}

for factor, order_map in [('monthly_volume', vol_order),
                           ('monthly_fraud_level', fraud_order),
                           ('capture_delay', delay_order)]:
    subset = df[df[factor].notna()].copy()
    subset['rank'] = subset[factor].map(order_map)
    print(f"\n{factor}:")
    print(subset.groupby('rank')['sample_fee'].mean().sort_index())
```

## Reading manual.md Efficiently

The manual is organized as:
1. Account types (Section 2)
2. MCC codes (Section 3)
3. ACI values (Section 4)
4. **Fee fields and formulas (Section 5)** ← most relevant
5. PIN limits, fraud management, reporting (Sections 6-8)
6. Glossary (Appendix)

For any question about a fee factor's meaning or directionality, go directly to **Section 5** which contains the authoritative definitions.

## Common Errors to Avoid

1. **Raw statistical analysis vs. manual definitions**: Computing average fees across all True vs. all False rows gives misleading results. Rules apply to different merchant segments — the manual's directional statements are authoritative.

2. **"Highest volume where fees do NOT become cheaper"**: This asks for the maximum/ceiling tier — the answer is `>5m`. Reasoning: at `>5m` there is no higher volume tier, so fees cannot decrease further. The pricing curve flattens at this ceiling. Do NOT interpret this as the tier just before the cheapest tier (which would be `1m-5m`).

3. **Volume tier string ordering**: String sort puts `>5m` before `<100k` alphabetically. Always use the logical numeric order: `<100k` < `100k-1m` < `1m-5m` < `>5m`.

4. **Null fields**: A null/NaN value in a fee rule field means the rule applies universally for that field — not that data is missing.

5. **Absence evidence**: When a fee concept is mentioned causally in the manual (e.g., "retrying may cause downgrades") but no fee amount or rule field exists for it, the correct answer is "Not Applicable" — not 0 or unknown.

6. **is_credit direction**: The column exists in both `payments.csv` (transaction-level) and `fees.json` (rule-level). Credit = more expensive. Debit (False) = cheaper. Don't conflate "more expensive" with "True".