tools-only Research & Survey
Contents
- Large Language Model: Landscape
- Prompt Engineering and Visual Prompts
- Finetuning
- Large Language Model: Challenges and Solutions
- Survey and Reference
Large Language Model: Landscape
- The best NLP papers from 2015 to now
- In 2023: As abilities emerge only at scale, we must unlearn outdated intuitions, scale Transformers via massive distributed matrix multiplications, and discover the inductive bias needed to push ~10,000× beyond GPT-4. 🗣️ / 📺 / ✍️ [6 Oct 2023]
Large Language Model Comparison
- AI Model Review: Compare 75 AI Models on 200+ Prompts Side By Side.
- Artificial Analysis:💡Independent analysis of AI models and API providers.
- Inside language models (from GPT to Olympus)
- LiveBench: a benchmark for LLMs designed with test set contamination.
- LLMArena:💡Chatbot Arena (formerly LMSYS): Free AI Chat to Compare & Test Best AI Chatbots
- LLMprices.dev: Compare prices for models like GPT-4, Claude Sonnet 3.5, Llama 3.1 405b and many more.
- LLM Pre-training and Post-training Paradigms [17 Aug 2024]
The Big LLM Architecture Comparison (in 2025)
The Big LLM Architecture Comparison✍️:💡 [19 Jul 2025]
Model Parameters Attention Type MoE Norm Positional Encoding Notable Features DeepSeek V3 / R1 671B Multi-Head Latent Attention (MLA) Yes, 256 experts (37B active) Pre-normalization RoPE KV compression via MLA, shared expert, high inference efficiency OLMo 2 32B Multi-Head Attention (MHA) No Post-normalization + QK norm (RMSNorm) RoPE RMSNorm scaling after attention & FF, training stability Gemma 3 / 3n 27B / 4B Sliding Window + Grouped-Query Attention No Pre + Post RMSNorm RoPE Sliding window attention, Gemma 3n: Per-Layer Embedding (PLE), MatFormer slices Mistral Small 3.1 24B Grouped-Query Attention No Pre-normalization RoPE Optimized for low latency, simpler than Gemma 3 Llama 4 Maverick 400B Grouped-Query Attention Yes, fewer & larger experts Pre-normalization RoPE Alternating MoE & dense layers, 17B active parameters Qwen3 (Dense) 0.6–32B Grouped-Query Attention No Pre-normalization RoPE Deep architecture, small memory footprint Qwen3 (MoE) 30B–235B Grouped-Query Attention Yes, no shared expert Pre-normalization RoPE Sparse MoE, optimized for large-scale inference SmolLM3 3B Grouped-Query Attention No Pre-normalization NoPE (No Positional Embedding) Good small-scale performance, improved length generalization Kimi K2 1T MLA Yes, more experts than DeepSeek Pre-normalization RoPE Muon optimizer, very high modeling performance, open-weight gpt-oss 20B / 120B Grouped-Query + Sliding Window Yes, few large experts Pre-normalization RoPE Wider architecture, attention sinks, bias units Grok 2.5 70B Grouped-Query Attention Yes Pre-normalization RoPE Standard large-scale architecture GLM-4.5 130B Grouped-Query Attention Yes Pre-normalization RoPE Standard architecture with high performance Qwen3-Next - Grouped-Query Attention Yes Pre-normalization RoPE Expert size & number tuned, Gated DeltaNet + Gated Attention Hybrid, Multi-Token Prediction Beyond Standard LLMs✍️:💡Linear Attention Hybrids, Text Diffusion, Code World Models, and Small Recursive Transformers [04 Nov 2025]
Architecture Type Key Models Attention Mechanism Main Advantage Main Limitation Use Case Standard Transformer GPT-5, DeepSeek V3/R1, Llama 4, Qwen3, Gemini 2.5, MiniMax-M2 Quadratic O(n²) scaled-dot-product Proven, SOTA performance, mature tooling Expensive training & inference, quadratic complexity General-purpose LLM tasks Linear Attention Hybrids Qwen3-Next, Kimi Linear, MiniMax-M1, DeepSeek V3.2 Gated DeltaNet + Full Attention (3:1 ratio) 75% KV cache reduction, 6× decoding throughput, linear O(n) Trades accuracy for efficiency, added complexity Long-context tasks, resource-constrained environments Text Diffusion LLaDA, Gemini Diffusion Bidirectional (no causal mask) Parallel token generation, faster responses Can't stream, tricky tool-calling, quality degradation with fewer steps Fast inference, on-device LLMs Code World Models CWM (32B) Standard sliding-window attention Simulates code execution, improves reasoning Limited to code domain, added latency from execution traces Code generation, debugging, test-time scaling Small Recursive Transformers TRM (7M), HRM (28M) Standard attention with recursive refinement Very small (7M params), strong puzzle solving, <$500 training cost Special-purpose, limited to structured tasks (Sudoku, ARC, Maze) Domain-specific reasoning, tool-calling modules
GPT-2 vs gpt-oss
| Feature | GPT-2 | GPT-OSS |
|---|---|---|
| Release & Size | 2019, up to 1.5B params | 2025, 20B & 120B params (MoE) |
| Architecture | Dense transformer decoder | Mixture-of-Experts (MoE) decoder |
| Activation & Dropout | Swish activation, uses dropout | GELU (or optimized), no dropout |
| Parameter Efficiency | All params active per token | Sparse activation of experts |
| Deployment & License | MIT license | Open-weight local runs, Apache 2.0 |
| Reasoning & Tools | Basic generation | Built-in chain-of-thought & tool use |
Evolutionary Tree of Large Language Models
- Evolutionary Graph of LLaMA Family
- LLM evolutionary tree
- Timeline of SLMs
- A Comprehensive Survey of Small Language Models in the Era of Large Language Models📑 / ✨ [4 Nov 2024]
- LLM evolutionary tree📑: A curated list of practical guide resources of LLMs (LLMs Tree, Examples, Papers) ✨ [26 Apr 2023]
- A Survey of Large Language Models📑: /✨ [31 Mar 2023] contd.
A Taxonomy of Natural Language Processing
- An overview of different fields of study and recent developments in NLP. 🗄️ / ✍️ [24 Sep 2023] Exploring the Landscape of Natural Language Processing Research ref📑 [20 Jul 2023]
- NLP taxonomy
Large Language Model Collection
- Ai2 (Allen Institute for AI)
- Founded by Paul Allen, the co-founder of Microsoft, in Sep 2024.
- DR Tulu✨: 8B. Deep Research (DR) model trained for long-form DR tasks. [Nov 2025]
- OLMo📑:💡Truly open language model and framework to build, study, and advance LMs, along with the training data, training and evaluation code, intermediate model checkpoints, and training logs. ✨ [Feb 2024]
- OLMo 2 [26 Nov 2024]
- OLMo 3✍️: Fully open models including the entire flow. [20 Nov 2025]
- OLMoE✨: fully-open LLM leverages sparse Mixture-of-Experts [Sep 2024]
- TÜLU 3📑:💡Pushing Frontiers in Open Language Model Post-Training ✨ / demo:✍️ [22 Nov 2024]
- Alibaba
- Qwen (通义千问: Universal Intelligence that can answer a thousand questions) ✨ Flagship Models✍️
- Qwen model family: Qwen first model released in [April 2023]
- Qwen-Image-Edit [18 Aug 2025]
- Qwen3-Max: over 1 trillion parameters. 256K tokens. [5 Sep 2025]
- Amazon
- Amazon Nova Foundation Models: Text only - Micro, Multimodal - Light, Pro [3 Dec 2024]
- The Amazon Nova Family of Models: Technical Report and Model Card📑 [17 Mar 2025]
- Anthrophic
- Claude 3✍️, the largest version of the new LLM, outperforms rivals GPT-4 and Google’s Gemini 1.0 Ultra. Three variants: Opus, Sonnet, and Haiku. [Mar 2024]
- Claude 3.7 Sonnet and Claude Code✍️: the first hybrid reasoning model. ✍️ [25 Feb 2025]
- Claude 4✍️: Claude Opus 4 (72.5% on SWE-bench), Claude Sonnet 4 (72.7% on SWE-bench). Extended Thinking Mode (Beta). Parallel Tool Use & Memory. Claude Code SDK. AI agents: code execution, MCP connector, Files API, and 1-hour prompt caching. [23 May 2025]
- Claude 4.5✍️: Major upgrades in autonomous coding, tool use, context handling, memory, and long-horizon reasoning; supports over 30 hours of continuous operation. [30 Sep 2025]
- Claude Opus 4.5✍️: SWE-bench Verified (80.9%). $5/$25 per million tokens [25 Nov 2025]
- anthropic/cookbook✨
- Apple
- OpenELM: Apple released a Transformer-based language model. Four sizes of the model: 270M, 450M, 1.1B, and 3B parameters. [April 2024]
- Apple Intelligence Foundation Language Models: 1. A 3B on-device model used for language tasks like summarization and Writing Tools. 2. A large Server model used for language tasks too complex to do on-device. [10 Jun 2024]
- Baidu
- ERNIE Bot's official website: ERNIE X1 (deep-thinking reasoning) and ERNIE 4.5 (multimodal) [16 Mar 2025]
- A list of models & libraries: ✨
- Chatbot Arena🤗
- Chatbot Arena🤗: Benchmarking LLMs in the Wild with Elo Ratings
- Cohere
- Founded in 2019. Canadian multinational tech.
- Command R+🤗: The performant model for RAG capabilities, multilingual support, and tool use. [Aug 2024]
- An Overview of Cohere’s Models | Playground
- Databricks
- Deepseek
- Founded in 2023, is a Chinese company dedicated to AGI.
- DeepSeek-V3✨: Mixture-of-Experts (MoE) with 671B. [26 Dec 2024]
- DeepSeek-R1✨:💡an open source reasoning model. Group Relative Policy Optimization (GRPO). Base -> RL -> SFT -> RL -> SFT -> RL [20 Jan 2025] ref📑: A Review of DeepSeek Models' Key Innovative Techniques [14 Mar 2025]
- Janus✨: Multimodal understanding and visual generation. [28 Jan 2025]
- DeepSeek-V3🤗: 671B. Top-tier performance in coding and reasoning tasks [25 Mar 2025]
- DeepSeek-Prover-V2✨: Mathematical reasoning [30 Apr 2025]
- DeepSeek-v3.1🤗: Think/Non‑Think hybrid reasoning. 128K and MoE. Agent abilities. [19 Aug 2025]
- DeepSeek-V3.2📑: DeepSeek Sparse Attention (DSA) cuts complexity from O(L²) to O(Lk). [12 Dec 2025]
- DeepSeek-V3.2-Exp✨ [Sep 2025]
- DeepSeek-OCR✨: Convert long text into an image, compresses it into visual tokens, and sends those to the LLM — cutting cost and expanding context capacity. [Oct 2025]
- DeepSeekMath-V2✨: a Self-Verifiable Mathematical Reasoning model [27 Nov 2025]
- mHC (Manifold-Constrained Hyper-Connections)📑 [31 Dec 2025]
Controlled layer updates for stable deep models.
next state = current state + constrained update
(vs. residuals: F(x) + x -> Hyper-Connections: unconstrained -> mHC: constrained) - Engram (Conditional Memory Module)✨
Adds a native memory lookup alongside neural computation, letting frequent patterns be retrieved in constant time.
output = compute(x) + memory lookup(x)
(vs. attention: recomputing patterns every time -> Engram) - A list of models: ✨
- EleutherAI
- Founded in July 2020. United States tech. GPT-Neo, GPT-J, GPT-NeoX, and The Pile dataset.
- Pythia📑: How do large language models (LLMs) develop and evolve over the course of training and change as models scale? A suite of decoder-only autoregressive language models ranging from 70M to 12B parameters ✨ [Apr 2023]
- Google
- Foundation Models: Gemini, Veo, Gemma etc.
- Gemma: Open weights LLM from Google DeepMind. ✨ / Pytorch ✨ [Feb 2024]
- Gemma 2 2B, 9B, 27B ref: releases [Jun 2024]
- Gemma 3: Single GPU. Context length of 128K tokens, SigLIP encoder, Reasoning ✍️ [12 Mar 2025]
- Gemini: Rebranding: Bard -> Gemini [8 Feb 2024]
- Gemini 1.5✍️: 1 million token context window, 1 hour of video, 11 hours of audio, codebases with over 30,000 lines of code or over 700,000 words. [Feb 2024]
- Gemini 2 Flash✍️: Multimodal LLM with multilingual inputs/outputs, real-time capabilities (Project Astra), complex task handling (Project Mariner), and developer tools (Jules) [11 Dec 2024]
- Gemini 2.0 Flash Thinking Experimental [19 Dec 2024]
- Gemini 2.5✍️: strong reasoning and code. 1 million token context [25 Mar 2025] -> I/O 2025✍️ Deep Think, 1M-token context, Native audio output, Project Mariner: AI-powered computer control. [20 May 2025] Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities.📑
- Gemma 3n: The next generation of Gemini Nano. Gemma 3n uses DeepMind’s Per-Layer Embeddings (PLE) to run 5B/8B models at 2GB/3GB RAM. [20 May 2025]
- gemini/cookbook✨
- Gemini 3 Pro✍️: Deep Think reasoning, Advanced multimodal understanding, spatial reasoning, and agentic capabilities up 30% from 2.5 Pro — reaching 37.5% on Humanity’s Last Exam (41% in Deep Think mode). [18 Nov 2025]
- Groq
- Founded in 2016. low-latency AI inference H/W. American tech.
- Llama-3-Groq-Tool-Use: a model optimized for function calling [Jul 2024]
- Huggingface
- Open R1✨: A fully open reproduction of DeepSeek-R1. [25 Jan 2025]
- Huggingface Open LLM Learboard🤗
- IBM
- Granite Guardian✨: a collection of models designed to detect risks in prompts and responses [10 Dec 2024]
- Jamba: AI21's SSM-Transformer Model. Mamba + Transformer + MoE [28 Mar 2024]
- KoAlpaca✨: Alpaca for korean [Mar 2023]
- Llama variants emerged in 2023
- Falcon LLM Apache 2.0 license [Mar 2023]
- Alpaca: Fine-tuned from the LLaMA 7B model [Mar 2023]
- vicuna: 90% ChatGPT Quality [Mar 2023]
- dolly: Databricks [Mar 2023]
- Cerebras-GPT: 7 GPT models ranging from 111m to 13b parameters. [Mar 2023]
- Koala: Focus on dialogue data gathered from the web. [Apr 2023]
- StableVicuna First Open Source RLHF LLM Chatbot [Apr 2023]
- Upstage's 70B Language Model Outperforms GPT-3.5: ✍️ [1 Aug 2023]
- LLM Collection: promptingguide.ai
- Meta
- Most OSS LLM models have been built on the Llama✨ / ✍️ / ✨
- Llama 2🤗: 1) 40% more data than Llama. 2)7B, 13B, and 70B. 3) Trained on over 1 million human annotations. 4) double the context length of Llama 1: 4K 5) Grouped Query Attention, KV Cache, and Rotary Positional Embedding were introduced in Llama 2 [18 Jul 2023] demo🤗
- Llama 3: 1) 7X more data than Llama 2. 2) 8B, 70B, and 400B. 3) 8K context length [18 Apr 2024]
- MEGALODON✨: Long Sequence Model. Unlimited context length. Outperforms Llama 2 model. [Apr 2024]
- Llama 3.1: 405B, context length to 128K, add support across eight languages. first OSS model outperforms GTP-4o. [23 Jul 2024]
- Llama 3.2: Multimodal. Include text-only models (1B, 3B) and text-image models (11B, 90B), with quantized versions of 1B and 3B [Sep 2024]
- NotebookLlama✨: An Open Source version of NotebookLM [28 Oct 2024]
- Llama 3.3: a text-only 70B instruction-tuned model. Llama 3.3 70B approaches the performance of Llama 3.1 405B. [6 Dec 2024]
- Llama 4: Mixture of Experts (MoE). Llama 4 Scout (actived 17b / total 109b, 10M Context, single GPU), Llama 4 Maverick (actived 17b / total 400b, 1M Context) ✨: Model Card [5 Apr 2025]
- ModernBERT📑: ModernBERT can handle sequences up to 8,192 tokens and utilizes sparse attention mechanisms to efficiently manage longer context lengths. [18 Dec 2024]
- Microsoft
- MAI-1✍️: MAI-Voice-1, MAI-1-preview. Microsoft in-house models. [28 Aug 2025]
- phi-series: cost-effective small language models (SLMs) ✍️ ✨: Cookbook
- Phi-1📑: Despite being small in size, phi-1 attained 50.6% on HumanEval and 55.5% on MBPP. Textbooks Are All You Need. ✍️ [20 Jun 2023]
- Phi-1.5📑: Textbooks Are All You Need II. Phi 1.5 is trained solely on synthetic data. Despite having a mere 1 billion parameters compared to Llama 7B's much larger model size, Phi 1.5 often performs better in benchmark tests. [11 Sep 2023]
- phi-2: open source, and 50% better at mathematical reasoning. ✨🤗 [Dec 2023]
- phi-3-vision (multimodal), phi-3-small, phi-3 (7b), phi-sillica (Copilot+PC designed for NPUs)
- Phi-3📑: Phi-3-mini, with 3.8 billion parameters, supports 4K and 128K context, instruction tuning, and hardware optimization. [22 Apr 2024] ✍️
- phi-3.5-MoE-instruct: 🤗 [Aug 2024]
- Phi-4📑: Specializing in Complex Reasoning ✍️ [12 Dec 2024]
- Phi-4-multimodal / mini🤗 5.6B. speech, vision, and text processing into a single, unified architecture. [26 Feb 2025]
- Phi-4-reasoning✍️: Phi-4-reasoning, Phi-4-reasoning-plus, Phi-4-mini-reasoning [30 Apr 2025]
- Phi-4-mini-flash-reasoning✍️: 3.8B, 64K context, Single GPU, Decoder-Hybrid-Decoder architecture [9 Jul 2025]
- MiniMaxAI
- Founded in Dec 2021. Shanghai, China.
- MiniMax-M2✨: Coding and Agent tasks, 230B (10B Active), MoE, a new high ahead of DeepSeek-V3.2 and Kimi K2
- Mistral
- Founded in April 2023. French tech.
- Model overview ✍️
- NeMo: 12B model with 128k context length that outperforms LLama 3 8B [18 Jul 2024]
- Mistral OCR: Precise text recognition with up to 99% accuracy. Multimodal. Browser based [6 Mar 2025]
- Moonshot AI
- Moonshot AI is a Beijing-based Chinese AI company founded in March 2023
- Kimi-K2✨: 1T parameter MoE model. MuonClip Optimizer. Agentic Intelligence. [11 Jul 2025]
- Kimi K2 Thinking✍️: The first open-source model beats GPT-5 in Agent benchmark. [7 Nov 2025]
- Kimi-K2.5✨: Open-source multimodal agentic model by Moonshot AI. [Jan 2026]
- NVIDIA
- Nemotron-4 340B: Synthetic Data Generation for Training Large Language Models [14 Jun 2024]
- ollam: ollama-supported models
- Open-Sora✨: Democratizing Efficient Video Production for All [Mar 2024]
- OpenAI
- gpt-oss✨:💡gpt-oss-120b and gpt-oss-20b are two open-weight language models by OpenAI. [Jun 2025]
- Qualcomm
- Qualcomm’s on-device AI models🤗: Bring generative AI to mobile devices [Feb 2024]
- Tencent
- Founded in 1998, Tencent is a Chinese company dedicated to various technology sectors, including social media, gaming, and AI development.
- Hunyuan-Large: An open-source MoE model with open weights. [4 Nov 2024] ✨
- Hunyuan-T1: Reasoning model [21 Mar 2025]
- A list of models: ✨
- The LLM Index: A list of large language models (LLMs)
- The mother of all spreadsheets for anyone into LLMs [17 Dec 2024]
- The Open Source AI Definition [28 Oct 2024]
- xAI
- xAI is an American AI company founded by Elon Musk in March 2023
- Grok: 314B parameter Mixture-of-Experts (MoE) model. Released under the Apache 2.0 license. Not includeded training code. Developed by JAX ✨ [17 Mar 2024]
- Grok-2 and Grok-2 mini [13 Aug 2024]
- Grok-2.5: Grok 2.5 Goes Open Source [24 Aug 2025]
- Grok-3: 200,000 GPUs to train. Grok 3 beats GPT-4o on AIME, GPQA. Grok 3 Reasoning and Grok 3 mini Reasoning. [17 Feb 2025]
- Grok-4: Humanity’s Last Exam, Grok 4 Heavy scored 44.4% [9 Jul 2025]
- Grok 4.1✍️ [17 Nov 2025]
- Xiaomi
- Founded in 2010, Xiaomi is a Chinese company known for its innovative consumer electronics and smart home products.
- Mimo✨: 7B. advanced reasoning for code and math [30 Apr 2025)
- Z.ai
LLM for Domain Specific
- AI for Scaling Legal Reform: Mapping and Redacting Racial Covenants in Santa Clara County📑: a fine-tuned open LLM to detect racial covenants in 24 million housing documents, cutting 86,500 hours of manual work. [12 Feb 2025]
- AlphaChip: Reinforcement learning-based model for designing physical chip layouts. [26 Sep 2024]
- AlphaFold3✨: Open source implementation of AlphaFold3 [Nov 2023] / OpenFold✨: PyTorch reproduction of AlphaFold 2 [Sep 2021]
- AlphaGenome: DeepMind’s advanced AI model, launched in June 2025, is designed to analyze the regulatory “dark matter” of the genome—specifically, the 98% of DNA that does not code for proteins but instead regulates when and how genes are expressed. [June 2025]
- BioGPT📑: Generative Pre-trained Transformer for Biomedical Text Generation and Mining ✨ [19 Oct 2022]
- BloombergGPT📑: A Large Language Model for Finance [30 Mar 2023]
- Chai-1✨: a multi-modal foundation model for molecular structure prediction [Sep 2024]
- Code Llama📑: Built on top of Llama 2, free for research and commercial use. ✍️ / ✨ [24 Aug 2023]
- DeepSeek-Coder-V2✨: Open-source Mixture-of-Experts (MoE) code language model [17 Jun 2024]
- Devin AI: Devin is an AI software engineer developed by Cognition AI [12 Mar 2024]
- EarthGPT📑: A Universal Multi-modal Large Language Model for Multi-sensor Image Comprehension in Remote Sensing Domain [30 Jan 2024]
- ESM3: A frontier language model for biology: Simulating 500 million years of evolution ✨ / ✍️ [31 Dec 2024]
- FrugalGPT📑: LLM with budget constraints, requests are cascaded from low-cost to high-cost LLMs. ✨ [9 May 2023]
- Galactica📑: A Large Language Model for Science [16 Nov 2022]
- Gemma series
- Gemma series in Huggingface🤗
- PaliGemma📑: a 3B VLM [10 Jul 2024]
- DataGemma✍️ [12 Sep 2024] / NotebookLM✍️: LLM-powered notebook. free to use, not open-source. [12 Jul 2023]
- PaliGemma 2📑: VLMs at 3 different sizes (3B, 10B, 28B) [4 Dec 2024]
- TxGemma: Therapeutics development [25 Mar 2025]
- Dolphin Gemma✍️: Decode dolphin communication [14 Apr 2025]
- MedGemma: Model fine-tuned for biomedical text and image understanding. [20 May 2025]
- SignGemma: Vision-language model for sign language recognition and translation. [27 May 2025)
- Huggingface StarCoder: A State-of-the-Art LLM for Code🤗: ✨🤗 [May 2023]
- MechGPT📑: Language Modeling Strategies for Mechanics and Materials ✨ [16 Oct 2023]
- MeshGPT: Generating Triangle Meshes with Decoder-Only Transformers [27 Nov 2023]
- OpenCoder✨: 1.5B and 8B base and open-source Code LLM, supporting both English and Chinese. [Oct 2024] 