Trade The Event Eval

trade-the-event-eval

Trade the Event: Corporate Events Detection for News-Based Event-Driven Trading — Zhou et al. (2021) (arXiv:2105.12825, 2021)

What this evaluates

This evaluation probes a model's ability to detect objective corporate events in financial news and translate those detections into actionable, timely trading signals. It measures how effectively the detected events predict short-term stock price movements and generate excess returns compared to a market benchmark.

Datasets

• EDT — total ?; splits: (unstated); repo https://github.com/Zhihan1996/TradeTheEvent

Metrics

• Winning Rate (primary) — range: percent
  • Calculated per transaction: return = (P_sell - P_buy) / P_buy for long or (P_sell - P_buy) / P_sell for short. Winning Rate is the percentage of all transactions where return ≥ 0%. Big Win Rate uses return ≥ 1%. Average Return is the mean return per transaction. Excess Returns equal total model returns minus the S&P 500 ETF benchmark return.

Input / output format

Input: News article text and associated minute-level historical stock price data for the detected ticker at the time of publication.

Output: Trading signals specifying buy or short positions for specific tickers, executed at the minute of news publication.

Scoring recipe

def evaluate(signals, prices, market_prices):
    returns = []
    for sig in signals:
        p_buy = prices[sig.ticker][sig.time]
        p_sell = prices[sig.ticker][sig.time + 1_day]
        ret = (p_sell - p_buy) / p_buy if sig.dir == 'long' else (p_sell - p_buy) / p_sell
        returns.append(ret * 0.997)  # 0.3% commission fee
    win_rate = sum(1 for r in returns if r >= 0) / len(returns)
    avg_ret = sum(returns) / len(returns)
    market_ret = (market_prices[-1] - market_prices[0]) / market_prices[0]
    excess_ret = sum(returns) - market_ret
    return win_rate, avg_ret, excess_ret

Common pitfalls

• Ignoring the 0.3% commission fee on each transaction when calculating returns.
• Assuming zero-latency execution; trading at the 'Close' price of the publication minute instead of the 'Open' price drastically reduces profitability.
• Overlooking liquidity constraints; the protocol assumes infinite liquidity, which may not hold for larger investment scales.

Evidence (verbatim from paper)

For a "buy" transaction, we define its return as $\frac{P_{sell} - P_{buy}}{P_{buy}}%$ , while for a "short-selling", we define its return as $\frac{P_{sell} - P_{buy}}{P_{sell}}%$ . Here, $P$ stands for the price. If a transaction's return is greater than or equal to 0, we call it a "win". If a transaction's return is greater than or equal to 1%, we call it a "big win". For each model, we calculate its winning rate, big win rate (rate of big wins among all the transactions) and average return on each transaction. We also evaluate the models' excess returns over the market, where we consider the S&P 500 index as the benchmark of the market performance.

Citation

@misc{zhou2021trade,
  title={Trade the Event: Corporate Events Detection for News-Based Event-Driven Trading},
  author={Zhou et al. (2021)},
  year={2021},
  note={arXiv:2105.12825}
}

• arXiv: 2105.12825