Nlb21 Eval

nlb21-eval

Neural Latents Benchmark '21: Evaluating latent variable models of neural population activity — Pei et al. (2021) (arXiv:2109.04463, 2021)

What this evaluates

Evaluates latent variable models for their ability to infer neural population dynamics and rates from spiking data. It probes model fidelity in predicting held-out spiking activity, decoding behavioral variables, and capturing autonomous forward dynamics without relying on external labels.

Datasets

• MC_Maze — total ?; splits: full (-1); repo https://github.com/neurallatents/nlb_tools
• MC_Maze-L — total ?; splits: full (-1); repo https://github.com/neurallatents/nlb_tools
• MC_Maze-M — total ?; splits: full (-1); repo https://github.com/neurallatents/nlb_tools
• MC_Maze-S — total ?; splits: full (-1); repo https://github.com/neurallatents/nlb_tools
• MC_RTT — total ?; splits: full (-1); repo https://github.com/neurallatents/nlb_tools
• Area2_Bump — total ?; splits: full (-1); repo https://github.com/neurallatents/nlb_tools
• DMFC_RSG — total ?; splits: full (-1); repo https://github.com/neurallatents/nlb_tools

Metrics

• Co-smoothing bps (primary) — range: other
  • Measures the improvement in log-likelihood of held-out spiking activity when using the model's inferred rates compared to a null model, normalized per spike. Reported in bits/spike.
• Behavior decoding — range: [0, 1]
  • Performance of linear decoding of hand velocities or behavioral variables based on inferred neural rates.
• PSTH R^2 — range: [0, 1]
  • Coefficient of determination between inferred single-trial rates and peri-stimulus time histogram (PSTH) averages across repeated conditions.
• Forward pred bps — range: other
  • Bits per spike measuring the model's ability to predict future spiking activity using its learned autonomous dynamics.

Input / output format

Input: Neural population spiking activity recorded from specific brain regions (e.g., motor cortex, area 2, DMFC) across multiple trials and conditions.

Output: Inferred neural firing rates/dynamics and predicted spiking activity for held-out trials.

Scoring recipe

def compute_co_smoothing_bps(observed_spikes, predicted_rates, null_rates):
    # Calculate log-likelihood of observed spikes under predicted rates
    ll_model = sum(observed_spikes * log(predicted_rates) - predicted_rates)
    # Calculate log-likelihood under null (average firing rate)
    ll_null = sum(observed_spikes * log(null_rates) - null_rates)
    # Convert to bits/spike
    bits_per_spike = (ll_model - ll_null) / (log(2) * sum(observed_spikes))
    return bits_per_spike

# Report mean ± SEM across datasets/trials

Common pitfalls

• Absolute co-smoothing scores are not easily comparable across datasets due to varying neuron counts and firing rates.
• Unsupervised co-smoothing and supervised behavioral decoding can conflict; post-processing (e.g., smoothing) may improve decoding while decreasing co-smoothing scores.

Evidence (verbatim from paper)

We report co-smoothing bits/spike ± standard error of the mean across the datasets in NLB. While exact rankings and performance gaps vary per dataset, more expressive deep neural networks tend to perform the best. Note that absolute co-smoothing performance is not easily compared across datasets.

Citation

@misc{pei2021nlb21,
  title={Neural Latents Benchmark '21: Evaluating latent variable models of neural population activity},
  author={Pei et al. (2021)},
  year={2021},
  note={arXiv:2109.04463}
}

• arXiv: 2109.04463