Dataset_Metadata_and_Business_Rules

Dataset_Metadata_and_Business_Rules Skill

Dataset Overview

The dabstep dataset consists of payment processing data for a merchant payment processor:

Step 1: Always Read Documentation First

Before analyzing any data, read both reference documents:

with open('data/manual.md', 'r') as f:
    manual = f.read()
print(manual)

with open('data/payments-readme.md', 'r') as f:
    readme = f.read()
print(readme)

These files are the authoritative source for all definitions and business rules. Many questions can be answered directly from them without loading fee data.

payments.csv Column Reference

Key columns (from payments-readme.md):

Critical: has_fraudulent_dispute is THE fraud column. "Fraud" = this column.

fees.json Structure and Business Rules

Fee Rule Fields

import json
with open('data/fees.json', 'r') as f:
    fees = json.load(f)
# Each rule has: ID, card_scheme, account_type, capture_delay,
# monthly_fraud_level, monthly_volume, merchant_category_code,
# is_credit, aci, fixed_amount, rate, intracountry

Fee formula: fee = fixed_amount + rate * transaction_value / 10000

Null means "applies to all": A null value in any field means that fee rule applies regardless of that field's value.

Boolean Fee Fields (Critical for Analysis)

From manual.md (authoritative):

• is_credit: "Typically credit transactions are more expensive (higher fee)"
• intracountry: "False are for international transactions...typically are more expensive"

Therefore:

• Factors cheaper when True: intracountry only
• Factors cheaper when False: is_credit only
• Factors cheaper when decreased: monthly_fraud_level (less fraud → cheaper), is_credit (True→False = cheaper)

Ordered/Tiered Fee Fields

monthly_volume (higher volume → cheaper fees per manual):

• '<100k' → '100k-1m' → '1m-5m' → '>5m'

monthly_fraud_level (lower fraud → cheaper fees):

• '<7.2%' → '7.2%-7.7%' → '7.7%-8.3%' → '>8.3%'

capture_delay (faster = MORE expensive, not cheaper):

• 'immediate' (most expensive) → '<3' → '3-5' → '>5' → 'manual'

Authorization Characteristics Indicator (ACI)

Account Types

Task-Specific Strategies

"What column indicates X?"

→ Look in payments-readme.md first. For fraud: has_fraudulent_dispute.

"Does a specific fee (e.g., retry fee, penalty fee) exist?"

→ Check fees.json field names and manual.md. Fee rules only contain: fixed_amount and rate. If no such fee type exists in the data, answer "Not Applicable".

# Check available fields
df = pd.DataFrame(fees)
print(df.columns.tolist())
# Check for specific concept in manual
import re
matches = re.findall(r'.*retry.*', manual, re.IGNORECASE)
print(matches)

Note: manual.md mentions "excessive retrying causes downgrades" but there is NO explicit retry fee amount in fees.json → "Not Applicable".

"What boolean factors lead to cheaper fees if set to True/False?"

→ Use manual.md's explicit statements, not empirical data analysis (confounding variables make raw data misleading).

• True → cheaper: intracountry (domestic is cheaper)
• False → cheaper: is_credit (debit is cheaper than credit)

"Which factors lead to cheaper fees when decreased?"

Use manual.md rules:

• monthly_fraud_level ↓ → cheaper ✓ (less fraud = less risk = lower fees)
• is_credit ↓ (True→False) → cheaper ✓ (debit < credit cost)
• monthly_volume ↓ → more expensive ✗ (higher volume = better rates)
• capture_delay ↓ (faster) → more expensive ✗ ("faster capture = more expensive")
• intracountry is boolean (True/False), not meaningfully "decreased"

"What volume tier marks a threshold for cheaper fees?"

→ Analyze fees.json statistically by monthly_volume tier:

import pandas as pd
import json

with open('data/fees.json') as f:
    fees = json.load(f)
df = pd.DataFrame(fees)

volume_order = ['<100k', '100k-1m', '1m-5m', '>5m']
for vol in volume_order:
    sub = df[df['monthly_volume'] == vol]
    print(f"{vol}: count={len(sub)}, median_rate={sub['rate'].median():.1f}, median_fixed={sub['fixed_amount'].median():.4f}")

Compare median rate and fixed_amount across tiers to find where fees stop improving.

Example pattern (from actual data):

• <100k: median_rate=50, median_fixed=0.05
• 100k-1m: median_rate=49.5, median_fixed=0.07
• 1m-5m: median_rate=52, median_fixed=0.08 ← fees got WORSE
• >5m: median_rate=59, median_fixed=0.07 ← rate still not cheaper

So "the highest volume at which fees do NOT become cheaper" = >5m (the highest tier where fees are still not lower than all prior tiers).

"What are the possible values for field X?"

import json, pandas as pd
with open('data/fees.json') as f:
    fees = json.load(f)
df = pd.DataFrame(fees)
print(df['monthly_fraud_level'].unique())  # or any other field
print(df['capture_delay'].unique())
print(df['monthly_volume'].unique())

Key Business Rules (from manual.md)

1. Fee formula: fee = fixed_amount + rate * transaction_value / 10000
2. Fraud definition: ratio of fraudulent volume over total volume (monthly)
3. PIN limit: 3 consecutive incorrect PIN attempts → card temporarily blocked
4. PCI DSS non-compliance penalty: EUR 5,000 to EUR 100,000 per month
5. No excessive retry fee defined in fees.json (downgrades mentioned, not fees)
6. Monthly volumes computed in natural months (day 1 to last day of month)
7. Local acquiring (intracountry=True) → reduced fees and friction

Answer Format Guidelines

• Single column name → just the column name (e.g., has_fraudulent_dispute)
• List of factors → comma-separated (e.g., monthly_fraud_level, is_credit)
• No applicable answer → Not Applicable
• Volume/tier value → use exact string from data (e.g., >5m, 1m-5m)
• Dollar amount → number only if clearly defined; otherwise Not Applicable