Long Context

Extend context windows of transformer models using RoPE, YaRN, ALiBi, and position interpolation techniques. Use when processing long documents (32k-128k+ tokens), extending pre-trained models beyond original context limits, or implementing efficient positional encodings. Covers rotary embeddings, attention biases, interpolation methods, and extrapolation strategies for LLMs.

synthetic-sciences Updated 4.4k repo stars
An OpenScience session comparing EGFR inhibitor potency: the trace shows the search, the written files, the analysis run and the answer with a results table

File contents

synthetic-sciences/openscience commit d2b16e631d

Frequently asked questions

npx get-skill add synthetic-sciences/long-context