Dabstep Dataset Metadata Business Rules

Dataset Metadata and Business Rules — DABSTEP

Dataset Files

Core Fee Formula

fee = fixed_amount + rate * transaction_value / 10000

fees.json rules contain these fields:

Null values in any field mean the rule applies to ALL values of that field.

payments.csv Key Columns

Factor Directionality (from manual.md)

These are the definitive business rule statements for how each factor affects fees:

Key rule: When asked "which boolean factor, when set to True, leads to cheaper fees?" — only intracountry=True is cheaper (domestic). is_credit=True is more expensive.

Key rule: When asked "which factors, when DECREASED, lead to cheaper fees?" — monthly_fraud_level (lower fraud = cheaper) and is_credit (False < True in terms of cost, so decreasing/switching to False is cheaper).

Strategy by Question Type

Type 1: Column/Field Identification

"What column indicates X?" / "What field stores Y?"

1. Read payments-readme.md for payments.csv columns
2. Read manual.md Section 5 for fees.json field definitions
3. Match the described concept to the exact column name

# Load and inspect
import pandas as pd, json
df = pd.read_csv('payments.csv', nrows=3)
print(df.columns.tolist())

with open('fees.json') as f:
    fees = json.load(f)
print(list(fees[0].keys()))

Type 2: Fee/Rule Existence Questions

"Does fee X exist?" / "How much is fee Y?"

Search strategy:

1. Search manual.md for the concept (e.g., grep for "retry", "refund", "chargeback")
2. Inspect fees.json field names — the schema defines what fee types are modeled
3. If the manual mentions a behavior causally (e.g., "excessive retrying causes downgrades") but no numeric fee is defined → answer "Not Applicable"
4. If the concept is absent entirely → answer "Not Applicable"

import json
with open('fees.json') as f:
    fees = json.load(f)
# Check available fields
print(list(fees[0].keys()))
# Fields present: ID, card_scheme, account_type, capture_delay,
# monthly_fraud_level, monthly_volume, merchant_category_code,
# is_credit, aci, fixed_amount, rate, intracountry
# No "retry_fee", "refund_fee", etc.

Type 3: Boolean Factor Analysis

"Which boolean factors contribute to cheaper fees when True/False?"

Always trust manual.md over empirical data averages.

Empirical mean/median comparisons across all rules are unreliable because rules for different boolean values serve different contexts. Use the manual's explicit directional statements.

import json, pandas as pd
with open('fees.json') as f:
    fees = json.load(f)
df = pd.DataFrame(fees)

# For a proper comparison, find rule PAIRS differing only in the boolean
# Use a representative transaction amount for fee calculation
amount = 100  # euros
df['sample_fee'] = df['fixed_amount'] + df['rate'] * amount / 10000

# Compare within matched groups
non_bool_cols = ['card_scheme', 'account_type', 'capture_delay',
                 'monthly_fraud_level', 'monthly_volume',
                 'merchant_category_code', 'aci']
# Look at mean by boolean value as a sanity check, but defer to manual
for col in ['is_credit', 'intracountry']:
    print(f"\n{col}:")
    print(df.groupby(col)['sample_fee'].mean())

Expected results per manual:

• is_credit=True → higher fees; is_credit=False → cheaper
• intracountry=True → cheaper (domestic); intracountry=False → expensive (international)

Type 4: Volume/Threshold Analysis

"At what volume do fees become cheaper?" / "Highest volume where fees do NOT become cheaper?"

Volume tier order (ascending): <100k → 100k-1m → 1m-5m → >5m

Manual rule: higher volume → cheaper fees (economies of scale). The pricing curve flattens out at higher volumes — meaning at the highest tier (>5m), fees cannot decrease further.

Answering volume-boundary questions:

• "At what volume do fees become cheaper?" — the tier(s) where lower rates apply; in practice >5m is the cheapest tier.
• "What is the highest volume at which fees do NOT become cheaper?" — Answer: >5m
  • Reasoning: >5m is the maximum volume tier. There is no higher tier to transition to, so fees cannot become cheaper at this volume level. The pricing curve has flattened at this ceiling.
  • Do NOT answer 1m-5m — that is the tier below the cheapest, not the highest tier.

import json
with open('fees.json') as f:
    fees = json.load(f)

# Enumerate all distinct monthly_volume tier values in logical order
tier_order = {'<100k': 0, '100k-1m': 1, '1m-5m': 2, '>5m': 3}
volumes = sorted({r['monthly_volume'] for r in fees if r.get('monthly_volume')},
                 key=lambda x: tier_order[x])
print(volumes)  # ['<100k', '100k-1m', '1m-5m', '>5m']
# Highest tier = volumes[-1] = '>5m'

Type 5: Multi-Factor Direction Analysis

"Which factors lead to cheaper fees when decreased?"

For each candidate factor, determine if its value range has a monotonic direction:

import json, pandas as pd
with open('fees.json') as f:
    fees = json.load(f)
df = pd.DataFrame(fees)

amount = 100
df['sample_fee'] = df['fixed_amount'] + df['rate'] * amount / 10000

# Define ordered tiers for categorical factors
vol_order = {'<100k': 0, '100k-1m': 1, '1m-5m': 2, '>5m': 3}
fraud_order = {'<7.2%': 0, '7.2%-7.7%': 1, '7.7%-8.3%': 2, '>8.3%': 3}
delay_order = {'immediate': 0, '<3': 1, '3-5': 2, '>5': 3, 'manual': 4}

for factor, order_map in [('monthly_volume', vol_order),
                           ('monthly_fraud_level', fraud_order),
                           ('capture_delay', delay_order)]:
    subset = df[df[factor].notna()].copy()
    subset['rank'] = subset[factor].map(order_map)
    print(f"\n{factor}:")
    print(subset.groupby('rank')['sample_fee'].mean().sort_index())

Reading manual.md Efficiently

The manual is organized as:

1. Account types (Section 2)
2. MCC codes (Section 3)
3. ACI values (Section 4)
4. Fee fields and formulas (Section 5) ← most relevant
5. PIN limits, fraud management, reporting (Sections 6-8)
6. Glossary (Appendix)

For any question about a fee factor's meaning or directionality, go directly to Section 5 which contains the authoritative definitions.

Common Errors to Avoid

1. Raw statistical analysis vs. manual definitions: Computing average fees across all True vs. all False rows gives misleading results. Rules apply to different merchant segments — the manual's directional statements are authoritative.

2. "Highest volume where fees do NOT become cheaper": This asks for the maximum/ceiling tier — the answer is >5m. Reasoning: at >5m there is no higher volume tier, so fees cannot decrease further. The pricing curve flattens at this ceiling. Do NOT interpret this as the tier just before the cheapest tier (which would be 1m-5m).

3. Volume tier string ordering: String sort puts >5m before <100k alphabetically. Always use the logical numeric order: <100k < 100k-1m < 1m-5m < >5m.

4. Null fields: A null/NaN value in a fee rule field means the rule applies universally for that field — not that data is missing.

5. Absence evidence: When a fee concept is mentioned causally in the manual (e.g., "retrying may cause downgrades") but no fee amount or rule field exists for it, the correct answer is "Not Applicable" — not 0 or unknown.

6. is_credit direction: The column exists in both payments.csv (transaction-level) and fees.json (rule-level). Credit = more expensive. Debit (False) = cheaper. Don't conflate "more expensive" with "True".