Tushare Duckdb

tushare-duckdb

Local DuckDB cache layer for Tushare Pro data with integrated concept sector data from jquant_data_sync.

Quick Start

Read Data (no network, concurrent-safe)

from tushare_db import DataReader

reader = DataReader(db_path="tushare.db")

# Single stock with forward adjustment
df = reader.get_stock_daily('000001.SZ', '20240101', adj='qfq')

# Multiple stocks
df = reader.get_multiple_stocks_daily(['000001.SZ', '600519.SH'], '20240101')

# Concept sector constituents (from jquant_data_sync)
df = reader.get_concept_stocks('20240115', concept_name='人工智能')

# Stock's concept sectors
df = reader.get_stock_concepts('20240115', ts_code='000001.SZ')

# Trade calendar
df = reader.get_trade_calendar('20240101', '20241231')

# Stock list
df = reader.get_stock_basic(list_status='L')

# Custom SQL
df = reader.query("SELECT * FROM daily WHERE ts_code = ?", ['000001.SZ'])

reader.close()

Download Data (requires TUSHARE_TOKEN env var)

from tushare_db import DataDownloader

downloader = DataDownloader(db_path="tushare.db", rate_limit_profile="standard")

# Basic data
downloader.download_trade_calendar()
downloader.download_stock_basic()

# Daily data for all stocks
downloader.download_all_stocks_daily(start_date='20200101')

# Daily update (single date)
downloader.download_daily_data_by_date('20241218')

downloader.close()

Concept Sector Data (jquant_data_sync)

External concept sector data integrated via DataReader:

from tushare_db import DataReader

reader = DataReader(db_path="tushare.db")

# Get concept constituents for a specific date (PIT)
df = reader.get_concept_stocks('20240115', concept_name='人工智能')
# Returns: ts_code, concept_name, in_date, out_date

# Get all concept sectors for a stock
df = reader.get_stock_concepts('20240115', ts_code='000001.SZ')
# Returns: concept_code, concept_name, in_date, out_date

# Get full cross-section for a date
df = reader.get_concept_cross_section('20240115')
# Returns all concept-stock relationships for the date

# Search concept sectors
df = reader.search_concepts('芯片')  # Fuzzy search

# List all concept sectors
df = reader.get_all_concepts()  # concept_code, concept_name

# Force refresh data from GitHub
reader.refresh_concept_data()

# Check cache status
info = reader.get_concept_cache_info()
reader.close()

Data Characteristics:

• Source: jquant_data_sync GitHub Releases
• Format: SCD (Slowly Changing Dimension) with in_date/out_date
• Update: Daily auto-download with local caching
• Cache Location: .concept_cache/all_concepts_pit_scd.csv

Point-in-Time (PIT) Queries

For backtesting, always use PIT queries to avoid look-ahead bias:

from tushare_db import DataReader

reader = DataReader(db_path="tushare.db")

# ❌ Wrong: Current snapshot (causes look-ahead bias)
df = reader.get_index_member_all(l1_code='801010.SI', is_new='Y')

# ✅ Correct: PIT query as of historical date
df = reader.get_index_member_all(
    l1_code='801010.SI',
    trade_date='20230115'  # Only stocks in sector on this date
)

# ✅ Correct: Concept sector PIT query
df = reader.get_concept_stocks('20230115', concept_name='人工智能')

Correct SQL for PIT joins:

-- For index_member_all
SELECT *
FROM my_data d
JOIN index_member_all im ON d.ts_code = im.ts_code
WHERE im.in_date <= d.trade_date
  AND (im.out_date IS NULL OR im.out_date > d.trade_date)

-- For concept sectors (from jquant_data_sync)
SELECT *
FROM my_data d
JOIN concept_data c ON d.ts_code = c.ts_code
WHERE c.in_date <= d.trade_date
  AND c.out_date >= d.trade_date

Manual PIT data backfill:

# If database lacks historical out_date data
python scripts/backfill_index_member_pit.py

Key Notes

• Date format: YYYYMMDD string (e.g., '20240101')
• Adjustment types:
  • adj_factor table stores raw Tushare adjustment factors
  • qfq (forward): price * (factor / latest_factor) — latest price = market price
  • hfq (backward): price * factor — historical prices stay fixed
  • DataReader handles calculation automatically via adj='qfq' or adj='hfq'
• Rate limit profiles: 'trial', 'standard', 'pro'

Implemented Tables

External Data Sources

基础信息表（静态数据）

日频时间序列表

财务数据表（季度更新）

基金数据表

沪深港通数据表

同花顺板块说明

ths_index 表包含以下类型的板块指数，本项目仅实现部分类型：

For column details and parameter meanings, invoke /tushare-finance <api_name>.

Special Notes

Update Frequency

Date Field Notes

• Daily data tables: Use trade_date field for queries
• Financial data tables: Use end_date field (reporting period), not announcement date
• Fund NAV tables: Use nav_date field
• Membership tables (hs_const, index_member_all): Contains in_date/out_date fields for tracking historical changes
• Concept sectors: Use PIT queries with in_date/out_date range

Important Query Tips

1. index_weight: Only query for month-end dates (e.g., 20240131, 20240229); daily data is not available
2. Financial reports: When querying quarterly data, use the period-end date (e.g., 20240331 for Q1 2024)
3. trade_cal: Use is_open='1' to filter trading days only
4. hs_const: Contains historical records; filter by in_date and out_date to get valid constituents for a specific date
5. Concept sectors: Always use PIT queries (e.g., get_concept_stocks('20240115', concept_name='AI')) to avoid look-ahead bias

Converted and distributed by TomeVault — claim your Tome and manage your conversions.