# 2564 Models Research 0d1c711d
> Research & Survey
- Skill: `tools-only/2564-models-research-0d1c711d` (Agent Skill, multi-file: 3 files)
- Install (CLI): `npx skillmds@latest add tools-only/2564-models-research-0d1c711d`
- Raw SKILL.md: https://api.skillmd.com/api/skills/tools-only/2564-models-research-0d1c711d/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Research & Search
- Author: tools-only (https://skillmd.com/u/tools-only)
- Updated: 2026-09-22
- Page: https://skillmd.com/skills/tools-only/2564-models-research-0d1c711d
---
# Research & Survey
### **Contents**
- [Large Language Model: Landscape](#large-language-model-landscape)
- [Large Language Model Comparison](#large-language-model-comparison)
- [Evolutionary Tree of Large Language Models](#evolutionary-tree-of-large-language-models)
- [Large Language Model Collection](#large-language-model-collection)
- [Prompt Engineering and Visual Prompts](#prompt-engineering-and-visual-prompts)
- [Finetuning](#finetuning)
- [Quantization Techniques](#quantization-techniques)
- [Other Techniques and LLM Patterns](#other-techniques-and-llm-patterns)
- [Large Language Model: Challenges and Solutions](#large-language-model-challenges-and-solutions)
- [Context Constraints](#context-constraints)
- [Trustworthy, Safe and Secure LLM](#trustworthy-safe-and-secure-llm)
- [Large Language Model's Abilities](#large-language-model-is-abilities)
- [Reasoning](#reasoning)
- [OpenAI Roadmap](#openai-roadmap) | [OpenAI Products](#openais-products)
- [Anthropic AI Products](#anthropic-ai-products)
- [Google AI Products](#google-ai-products)
- [AGI Discussion and Social Impact](#agi-discussion-and-social-impact)
- [Survey and Reference](#survey-and-reference)
- [Survey on Large Language Models](#survey-on-large-language-models)
- [Additional Topics: A Survey of LLMs](#additional-topics-a-survey-of-llms)
- [LLM Research (Ranked by cite count ≥150)](#llm-research-ranked-by-cite-count-150)
- [Build an LLMs from Scratch](#build-an-llms-from-scratch-picogpt-and-lit-gpt)
- [Business Use Cases](#business-use-cases)
## **Large Language Model: Landscape**
1. [The best NLP papers from 2015 to now](https://thebestnlppapers.com/nlp/papers/all/)
1. In 2023: As abilities emerge only at scale, we must unlearn outdated intuitions, scale Transformers via massive distributed matrix multiplications, and discover the inductive bias needed to push ~10,000× beyond GPT-4. [🗣️](https://twitter.com/hwchung27/status/1710003293223821658) / [📺](https://t.co/vumzAtUvBl) / [✍️](https://t.co/IidLe4JfrC) [6 Oct 2023]
#### Large Language Model Comparison
- [AI Model Review](https://aimodelreview.com/): Compare 75 AI Models on 200+ Prompts Side By Side.
- [Artificial Analysis](https://artificialanalysis.ai/):💡Independent analysis of AI models and API providers.
- [Inside language models (from GPT to Olympus)](https://lifearchitect.ai/models/)
- [LiveBench](https://livebench.ai): a benchmark for LLMs designed with test set contamination.
- [LLMArena](https://lmarena.ai/):💡Chatbot Arena (formerly LMSYS): Free AI Chat to Compare & Test Best AI Chatbots
- [LLMprices.dev](https://llmprices.dev): Compare prices for models like GPT-4, Claude Sonnet 3.5, Llama 3.1 405b and many more.
- [LLM Pre-training and Post-training Paradigms](https://sebastianraschka.com/blog/2024/new-llm-pre-training-and-post-training.html) [17 Aug 2024]
#### The Big LLM Architecture Comparison (in 2025)
- [The Big LLM Architecture Comparison✍️](https://sebastianraschka.com/blog/2025/the-big-llm-architecture-comparison.html):💡 [19 Jul 2025]
| Model | Parameters | Attention Type | MoE | Norm | Positional Encoding | Notable Features |
| --------------------- | ---------- | ---------------------------------------- | ------------------------------- | ------------------------------- | ------------------------------ | ------------------------------------------------------------------------------------------- |
| **DeepSeek V3 / R1** | 671B | Multi-Head Latent Attention (MLA) | Yes, 256 experts (37B active) | Pre-normalization | RoPE | KV compression via MLA, shared expert, high inference efficiency |
| **OLMo 2** | 32B | Multi-Head Attention (MHA) | No | Post-normalization + QK norm (RMSNorm) | RoPE | RMSNorm scaling after attention & FF, training stability |
| **Gemma 3 / 3n** | 27B / 4B | Sliding Window + Grouped-Query Attention | No | Pre + Post RMSNorm | RoPE | Sliding window attention, Gemma 3n: Per-Layer Embedding (PLE), MatFormer slices |
| **Mistral Small 3.1** | 24B | Grouped-Query Attention | No | Pre-normalization | RoPE | Optimized for low latency, simpler than Gemma 3 |
| **Llama 4 Maverick** | 400B | Grouped-Query Attention | Yes, fewer & larger experts | Pre-normalization | RoPE | Alternating MoE & dense layers, 17B active parameters |
| **Qwen3 (Dense)** | 0.6–32B | Grouped-Query Attention | No | Pre-normalization | RoPE | Deep architecture, small memory footprint |
| **Qwen3 (MoE)** | 30B–235B | Grouped-Query Attention | Yes, no shared expert | Pre-normalization | RoPE | Sparse MoE, optimized for large-scale inference |
| **SmolLM3** | 3B | Grouped-Query Attention | No | Pre-normalization | NoPE (No Positional Embedding) | Good small-scale performance, improved length generalization |
| **Kimi K2** | 1T | MLA | Yes, more experts than DeepSeek | Pre-normalization | RoPE | Muon optimizer, very high modeling performance, open-weight |
| **gpt-oss** | 20B / 120B | Grouped-Query + Sliding Window | Yes, few large experts | Pre-normalization | RoPE | Wider architecture, attention sinks, bias units |
| **Grok 2.5** | 70B | Grouped-Query Attention | Yes | Pre-normalization | RoPE | Standard large-scale architecture |
| **GLM-4.5** | 130B | Grouped-Query Attention | Yes | Pre-normalization | RoPE | Standard architecture with high performance |
| **Qwen3-Next** | - | Grouped-Query Attention | Yes | Pre-normalization | RoPE | Expert size & number tuned, Gated DeltaNet + Gated Attention Hybrid, Multi-Token Prediction |
- [Beyond Standard LLMs✍️](https://magazine.sebastianraschka.com/p/beyond-standard-llms):💡Linear Attention Hybrids, Text Diffusion, Code World Models, and Small Recursive Transformers [04 Nov 2025]
| **Architecture Type** | **Key Models** | **Attention Mechanism** | **Main Advantage** | **Main Limitation** | **Use Case** |
| --- | --- | --- | --- | --- | --- |
| **Standard Transformer** | GPT-5, DeepSeek V3/R1, Llama 4, Qwen3, Gemini 2.5, MiniMax-M2 | Quadratic O(n²) scaled-dot-product | Proven, SOTA performance, mature tooling | Expensive training & inference, quadratic complexity | General-purpose LLM tasks |
| **Linear Attention Hybrids** | Qwen3-Next, Kimi Linear, MiniMax-M1, DeepSeek V3.2 | Gated DeltaNet + Full Attention (3:1 ratio) | 75% KV cache reduction, 6× decoding throughput, linear O(n) | Trades accuracy for efficiency, added complexity | Long-context tasks, resource-constrained environments |
| **Text Diffusion** | LLaDA, Gemini Diffusion | Bidirectional (no causal mask) | Parallel token generation, faster responses | Can't stream, tricky tool-calling, quality degradation with fewer steps | Fast inference, on-device LLMs |
| **Code World Models** | CWM (32B) | Standard sliding-window attention | Simulates code execution, improves reasoning | Limited to code domain, added latency from execution traces | Code generation, debugging, test-time scaling |
| **Small Recursive Transformers** | TRM (7M), HRM (28M) | Standard attention with recursive refinement | Very small (7M params), strong puzzle solving, <$500 training cost | Special-purpose, limited to structured tasks (Sudoku, ARC, Maze) | Domain-specific reasoning, tool-calling modules |
#### GPT-2 vs gpt-oss
- [From GPT-2 to gpt-oss: Analyzing the Architectural Advances✍️](https://magazine.sebastianraschka.com/p/from-gpt-2-to-gpt-oss-analyzing-the) [9 Aug 2025]
| Feature | GPT-2 | GPT-OSS |
| -------------------- | ------------------------------ | ------------------------------------ |
| Release & Size | 2019, up to 1.5B params | 2025, 20B & 120B params (MoE) |
| Architecture | Dense transformer decoder | Mixture-of-Experts (MoE) decoder |
| Activation & Dropout | Swish activation, uses dropout | GELU (or optimized), no dropout |
| Parameter Efficiency | All params active per token | Sparse activation of experts |
| Deployment & License | MIT license | Open-weight local runs, Apache 2.0 |
| Reasoning & Tools | Basic generation | Built-in chain-of-thought & tool use |
### **Evolutionary Tree of Large Language Models**
- Evolutionary Graph of LLaMA Family
- LLM evolutionary tree
- Timeline of SLMs
- [A Comprehensive Survey of Small Language Models in the Era of Large Language Models📑](https://arxiv.org/abs/2411.03350) / [✨](https://github.com/FairyFali/SLMs-Survey) [4 Nov 2024] 
- [LLM evolutionary tree📑](https://arxiv.org/abs/2304.13712): A curated list of practical guide resources of LLMs (LLMs Tree, Examples, Papers) [✨](https://github.com/Mooler0410/LLMsPracticalGuide) [26 Apr 2023]
- [A Survey of Large Language Models📑](https://arxiv.org/abs/2303.18223): /[✨](https://github.com/RUCAIBox/LLMSurvey) [31 Mar 2023] contd.
### **A Taxonomy of Natural Language Processing**
- An overview of different fields of study and recent developments in NLP. [🗄️](../files/taxonomy-nlp.pdf) / [✍️](https://towardsdatascience.com/a-taxonomy-of-natural-language-processing-dfc790cb4c01) [24 Sep 2023]
Exploring the Landscape of Natural Language Processing Research [ref📑](https://arxiv.org/abs/2307.10652) [20 Jul 2023]
- NLP taxonomy
Distribution of the number of papers by most popular fields of study from 2002 to 2022
### **Large Language Model Collection**
- Ai2 (Allen Institute for AI)
- Founded by Paul Allen, the co-founder of Microsoft, in Sep 2024.
- [DR Tulu✨](https://github.com/rlresearch/DR-Tulu): 8B. Deep Research (DR) model trained for long-form DR tasks. [Nov 2025]
- [OLMo📑](https://arxiv.org/abs/2402.00838):💡Truly open language model and framework to build, study, and advance LMs, along with the training data, training and evaluation code, intermediate model checkpoints, and training logs. [✨](https://github.com/allenai/OLMo) [Feb 2024]
- [OLMo 2](https://allenai.org/blog/olmo2) [26 Nov 2024]
 
- [OLMo 3✍️](https://allenai.org/blog/olmo3): Fully open models including the entire flow. [20 Nov 2025]
- [OLMoE✨](https://github.com/allenai/OLMoE): fully-open LLM leverages sparse Mixture-of-Experts [Sep 2024]
- [TÜLU 3📑](https://arxiv.org/abs/2411.15124):💡Pushing Frontiers in Open Language Model Post-Training [✨](https://github.com/allenai/open-instruct) / demo:[✍️](https://playground.allenai.org/) [22 Nov 2024] 
- Alibaba
- Qwen (通义千问: Universal Intelligence that can answer a thousand questions) [✨](https://github.com/QwenLM) [Flagship Models✍️](https://qwen.ai/)
- [Qwen model family](https://qwen4.net/qwen-model-family/): Qwen first model released in [April 2023]
- [Qwen-Image-Edit](https://qwen4.net/qwen-image-edit/) [18 Aug 2025]
- [Qwen3-Max](https://qwen4.net/qwen3-max-is-the-most-intelligent-non-reasoning-model/): over 1 trillion parameters. 256K tokens. [5 Sep 2025]
- Amazon
- [Amazon Nova Foundation Models](https://aws.amazon.com/de/ai/generative-ai/nova/): Text only - Micro, Multimodal - Light, Pro [3 Dec 2024]
- [The Amazon Nova Family of Models: Technical Report and Model Card📑](https://arxiv.org/abs/2506.12103) [17 Mar 2025]
- Anthrophic
- [Claude 3✍️](https://www.anthropic.com/news/claude-3-family), the largest version of the new LLM, outperforms rivals GPT-4 and Google’s Gemini 1.0 Ultra. Three variants: Opus, Sonnet, and Haiku. [Mar 2024]
- [Claude 3.7 Sonnet and Claude Code✍️](https://www.anthropic.com/news/claude-3-7-sonnet): the first hybrid reasoning model. [✍️](https://assets.anthropic.com/m/785e231869ea8b3b/original/claude-3-7-sonnet-system-card.pdf) [25 Feb 2025]
- [Claude 4✍️](https://www.anthropic.com/news/claude-4): Claude Opus 4 (72.5% on SWE-bench), Claude Sonnet 4 (72.7% on SWE-bench). Extended Thinking Mode (Beta). Parallel Tool Use & Memory. Claude Code SDK. AI agents: code execution, MCP connector, Files API, and 1-hour prompt caching. [23 May 2025]
- [Claude 4.5✍️](https://www.anthropic.com/news/claude-sonnet-4-5): Major upgrades in autonomous coding, tool use, context handling, memory, and long-horizon reasoning; supports over 30 hours of continuous operation. [30 Sep 2025]
- [Claude Opus 4.5✍️](https://www.anthropic.com/news/claude-opus-4-5): SWE-bench Verified (80.9%). $5/$25 per million tokens [25 Nov 2025]
- [anthropic/cookbook✨](https://github.com/anthropics/anthropic-cookbook)
- Apple
- [OpenELM](https://machinelearning.apple.com/research/openelm): Apple released a Transformer-based language model. Four sizes of the model: 270M, 450M, 1.1B, and 3B parameters. [April 2024]
- [Apple Intelligence Foundation Language Models](https://machinelearning.apple.com/research/apple-intelligence-foundation-language-models): 1. A 3B on-device model used for language tasks like summarization and Writing Tools. 2. A large Server model used for language tasks too complex to do on-device. [10 Jun 2024]
- Baidu
- [ERNIE Bot's official website](https://yiyan.baidu.com/): ERNIE X1 (deep-thinking reasoning) and ERNIE 4.5 (multimodal) [16 Mar 2025]
- A list of models & libraries: [✨](https://github.com/PaddlePaddle/ERNIE)
- Chatbot Arena🤗
- [Chatbot Arena🤗](https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard): Benchmarking LLMs in the Wild with Elo Ratings
- Cohere
- Founded in 2019. Canadian multinational tech.
- [Command R+🤗](https://huggingface.co/collections/CohereForAI/c4ai-command-r-plus-660ec4c34f7a69c50ce7f7b9): The performant model for RAG capabilities, multilingual support, and tool use. [Aug 2024]
- [An Overview of Cohere’s Models](https://docs.cohere.com/v2/docs/models) | [Playground](https://dashboard.cohere.com/playground)
- Databricks
- [DBRX](https://www.databricks.com/blog/introducing-dbrx-new-state-art-open-llm): MoE, open, general-purpose LLM created by Databricks. [✨](https://github.com/databricks/dbrx) [27 Mar 2024]
- Deepseek
- Founded in 2023, is a Chinese company dedicated to AGI.
- [DeepSeek-V3✨](https://github.com/deepseek-ai/DeepSeek-V3): Mixture-of-Experts (MoE) with 671B. [26 Dec 2024]
- [DeepSeek-R1✨](https://github.com/deepseek-ai/DeepSeek-R1):💡an open source reasoning model. Group Relative Policy Optimization (GRPO). Base -> RL -> SFT -> RL -> SFT -> RL [20 Jan 2025] [ref📑](https://arxiv.org/abs/2503.11486): A Review of DeepSeek Models' Key Innovative Techniques [14 Mar 2025]
- [Janus✨](https://github.com/deepseek-ai/Janus): Multimodal understanding and visual generation. [28 Jan 2025]
- [DeepSeek-V3🤗](https://huggingface.co/deepseek-ai/DeepSeek-V3): 671B. Top-tier performance in coding and reasoning tasks [25 Mar 2025]
- [DeepSeek-Prover-V2✨](https://github.com/deepseek-ai/DeepSeek-Prover-V2): Mathematical reasoning [30 Apr 2025]
- [DeepSeek-v3.1🤗](https://huggingface.co/deepseek-ai/DeepSeek-V3.1): Think/Non‑Think hybrid reasoning. 128K and MoE. Agent abilities. [19 Aug 2025]
- [DeepSeek-V3.2📑](https://arxiv.org/abs/2512.02556): DeepSeek Sparse Attention (DSA) cuts complexity from O(L²) to O(Lk). [12 Dec 2025]
- [DeepSeek-V3.2-Exp✨](https://github.com/deepseek-ai/DeepSeek-V3.2-Exp) [Sep 2025] 
- [DeepSeek-OCR✨](https://github.com/deepseek-ai/DeepSeek-OCR): Convert long text into an image, compresses it into visual tokens, and sends those to the LLM — cutting cost and expanding context capacity. [Oct 2025] 
- [DeepSeekMath-V2✨](https://github.com/deepseek-ai/DeepSeek-Math-V2/): a Self-Verifiable Mathematical Reasoning model [27 Nov 2025] 
- [mHC (Manifold-Constrained Hyper-Connections)📑](https://arxiv.org/abs/2512.24880) [31 Dec 2025]
Controlled layer updates for stable deep models. `next state = current state + constrained update`
(vs. residuals: F(x) + x -> Hyper-Connections: unconstrained -> mHC: constrained)
- [Engram (Conditional Memory Module)✨](https://github.com/deepseek-ai/Engram)
Adds a native memory lookup alongside neural computation, letting frequent patterns be retrieved in constant time. `output = compute(x) + memory lookup(x)`
(vs. attention: recomputing patterns every time -> Engram)
- A list of models: [✨](https://github.com/deepseek-ai)
- EleutherAI
- Founded in July 2020. United States tech. GPT-Neo, GPT-J, GPT-NeoX, and The Pile dataset.
- [Pythia📑](https://arxiv.org/abs/2304.01373): How do large language models (LLMs) develop and evolve over the course of training and change as models scale? A suite of decoder-only autoregressive language models ranging from 70M to 12B parameters [✨](https://github.com/EleutherAI/pythia) [Apr 2023]
- Google
- [Foundation Models](https://ai.google/discover/our-models/): Gemini, Veo, Gemma etc.
- [Gemma](http://ai.google.dev/gemma): Open weights LLM from Google DeepMind. [✨](https://github.com/google-deepmind/gemma) / Pytorch [✨](https://github.com/google/gemma_pytorch) [Feb 2024]
 
- [Gemma 2](https://www.kaggle.com/models/google/gemma-2/) 2B, 9B, 27B [ref: releases](https://ai.google.dev/gemma/docs/releases) [Jun 2024]
- [Gemma 3](https://developers.googleblog.com/en/introducing-gemma3/): Single GPU. Context
length of 128K tokens, SigLIP encoder, Reasoning [✍️](https://storage.googleapis.com/deepmind-media/gemma/Gemma3Report.pdf) [12 Mar 2025]
- [Gemini](https://gemini.google.com/app): Rebranding: Bard -> Gemini [8 Feb 2024]
- [Gemini 1.5✍️](https://blog.google/technology/ai/google-gemini-next-generation-model-february-2024): 1 million token context window, 1 hour of video, 11 hours of audio, codebases with over 30,000 lines of code or over 700,000 words. [Feb 2024]
- [Gemini 2 Flash✍️](https://blog.google/technology/google-deepmind/google-gemini-ai-update-december-2024/): Multimodal LLM with multilingual inputs/outputs, real-time capabilities (Project Astra), complex task handling (Project Mariner), and developer tools (Jules) [11 Dec 2024]
- Gemini 2.0 Flash Thinking Experimental [19 Dec 2024]
- [Gemini 2.5✍️](https://blog.google/technology/google-deepmind/gemini-model-thinking-updates-march-2025/): strong reasoning and code. 1 million token context [25 Mar 2025] -> [I/O 2025✍️](https://blog.google/technology/ai/io-2025-keynote) Deep Think, 1M-token context, Native audio output, Project Mariner: AI-powered computer control. [20 May 2025] [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities.📑](https://arxiv.org/abs/2507.06261)
- [Gemma 3n](https://developers.googleblog.com/en/introducing-gemma-3n/): The next generation of Gemini Nano. Gemma 3n uses DeepMind’s Per-Layer Embeddings (PLE) to run 5B/8B models at 2GB/3GB RAM. [20 May 2025]
- [gemini/cookbook✨](https://github.com/google-gemini/cookbook)
- [Gemini 3 Pro✍️](https://blog.google/products/gemini/gemini-3/): Deep Think reasoning, Advanced multimodal understanding, spatial reasoning, and agentic capabilities up 30% from 2.5 Pro — reaching 37.5% on Humanity’s Last Exam (41% in Deep Think mode). [18 Nov 2025]
- Groq
- Founded in 2016. low-latency AI inference H/W. American tech.
- [Llama-3-Groq-Tool-Use](https://wow.groq.com/introducing-llama-3-groq-tool-use-models/): a model optimized for function calling [Jul 2024]
- Huggingface
- [Open R1✨](https://github.com/huggingface/open-r1): A fully open reproduction of DeepSeek-R1. [25 Jan 2025]
- [Huggingface Open LLM Learboard🤗](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
- IBM
- [Granite Guardian✨](https://github.com/ibm-granite/granite-guardian): a collection of models designed to detect risks in prompts and responses [10 Dec 2024]
- [Jamba](https://www.ai21.com/blog/announcing-jamba): AI21's SSM-Transformer Model. Mamba + Transformer + MoE [28 Mar 2024]
- [KoAlpaca✨](https://github.com/Beomi/KoAlpaca): Alpaca for korean [Mar 2023]
- Llama variants emerged in 2023
- [Falcon LLM](https://falconllm.tii.ae/) Apache 2.0 license [Mar 2023]
- [Alpaca](https://crfm.stanford.edu/2023/03/13/alpaca.html): Fine-tuned from the LLaMA 7B model [Mar 2023]
- [vicuna](https://vicuna.lmsys.org/): 90% ChatGPT Quality [Mar 2023]
- [dolly](https://www.databricks.com/blog/2023/03/24/hello-dolly-democratizing-magic-chatgpt-open-models.html): Databricks [Mar 2023]
- [Cerebras-GPT](https://www.cerebras.net/blog/cerebras-gpt-a-family-of-open-compute-efficient-large-language-models/): 7 GPT models ranging from 111m to 13b parameters. [Mar 2023]
- [Koala](https://bair.berkeley.edu/blog/2023/04/03/koala/): Focus on dialogue data gathered from the web. [Apr 2023]
- [StableVicuna](https://stability.ai/blog/stablevicuna-open-source-rlhf-chatbot) First Open Source RLHF LLM Chatbot [Apr 2023]
- Upstage's 70B Language Model Outperforms GPT-3.5: [✍️](https://en.upstage.ai/newsroom/upstage-huggingface-llm-no1) [1 Aug 2023]
- [LLM Collection](https://www.promptingguide.ai/models/collection): promptingguide.ai
- Meta
- Most OSS LLM models have been built on the [Llama✨](https://github.com/facebookresearch/llama) / [✍️](https://ai.meta.com/llama) / [✨](https://github.com/meta-llama/llama-models)
 
- [Llama 2🤗](https://huggingface.co/blog/llama2): 1) 40% more data than Llama. 2)7B, 13B, and 70B. 3) Trained on over 1 million human annotations. 4) double the context length of Llama 1: 4K 5) Grouped Query Attention, KV Cache, and Rotary Positional Embedding were introduced in Llama 2 [18 Jul 2023] [demo🤗](https://huggingface.co/blog/llama2#demo)
- [Llama 3](https://llama.meta.com/llama3/): 1) 7X more data than Llama 2. 2) 8B, 70B, and 400B. 3) 8K context length [18 Apr 2024]
- [MEGALODON✨](https://github.com/XuezheMax/megalodon): Long Sequence Model. Unlimited context length. Outperforms Llama 2 model. [Apr 2024]
- [Llama 3.1](https://ai.meta.com/blog/meta-llama-3-1/): 405B, context length to 128K, add support across eight languages. first OSS model outperforms GTP-4o. [23 Jul 2024]
- [Llama 3.2](https://ai.meta.com/blog/llama-3-2-connect-2024-vision-edge-mobile-devices/): Multimodal. Include text-only models (1B, 3B) and text-image models (11B, 90B), with quantized versions of 1B and 3B [Sep 2024]
- [NotebookLlama✨](https://github.com/meta-llama/llama-recipes/tree/main/recipes/quickstart/NotebookLlama): An Open Source version of NotebookLM [28 Oct 2024]
- [Llama 3.3](https://www.llama.com/docs/model-cards-and-prompt-formats/llama3_3/): a text-only 70B instruction-tuned model. Llama 3.3 70B approaches the performance of Llama 3.1 405B. [6 Dec 2024]
- [Llama 4](https://www.llama.com/docs/model-cards-and-prompt-formats/llama4_omni/): Mixture of Experts (MoE). Llama 4 Scout (actived 17b / total 109b, 10M Context, single GPU), Llama 4 Maverick (actived 17b / total 400b, 1M Context) [✨](https://github.com/meta-llama/llama-models/tree/main/models/llama4): Model Card [5 Apr 2025]
- [ModernBERT📑](https://arxiv.org/abs/2412.13663): ModernBERT can handle sequences up to 8,192 tokens and utilizes sparse attention mechanisms to efficiently manage longer context lengths. [18 Dec 2024]
- Microsoft
- [MAI-1✍️](https://microsoft.ai/news/two-new-in-house-models/): MAI-Voice-1, MAI-1-preview. Microsoft in-house models. [28 Aug 2025]
- phi-series: cost-effective small language models (SLMs) [✍️](https://azure.microsoft.com/en-us/products/phi) [✨](https://aka.ms/Phicookbook): Cookbook
- [Phi-1📑](https://arxiv.org/abs/2306.11644): Despite being small in size, phi-1 attained 50.6% on HumanEval and 55.5% on MBPP. Textbooks Are All You Need. [✍️](https://analyticsindiamag.com/microsoft-releases-1-3-bn-parameter-language-model-outperforms-llama/) [20 Jun 2023]
- [Phi-1.5📑](https://arxiv.org/abs/2309.05463): Textbooks Are All You Need II. Phi 1.5 is trained solely on synthetic data. Despite having a mere 1 billion parameters compared to Llama 7B's much larger model size, Phi 1.5 often performs better in benchmark tests. [11 Sep 2023]
- phi-2: open source, and 50% better at mathematical reasoning. [✨🤗](https://huggingface.co/microsoft/phi-2) [Dec 2023]
- phi-3-vision (multimodal), phi-3-small, phi-3 (7b), phi-sillica (Copilot+PC designed for NPUs)
- [Phi-3📑](https://arxiv.org/abs/2404.14219): Phi-3-mini, with 3.8 billion parameters, supports 4K and 128K context, instruction tuning, and hardware optimization. [22 Apr 2024] [✍️](https://azure.microsoft.com/en-us/blog/introducing-phi-3-redefining-whats-possible-with-slms/)
- phi-3.5-MoE-instruct: [🤗](https://huggingface.co/microsoft/Phi-3.5-MoE-instruct) [Aug 2024]
- [Phi-4📑](https://arxiv.org/abs/2412.08905): Specializing in Complex Reasoning [✍️](https://techcommunity.microsoft.com/blog/aiplatformblog/introducing-phi-4-microsoft%E2%80%99s-newest-small-language-model-specializing-in-comple/4357090) [12 Dec 2024]
- [Phi-4-multimodal / mini🤗](https://huggingface.co/microsoft/Phi-4-multimodal-instruct/blob/main/phi_4_mm.tech_report.02252025.pdf) 5.6B. speech, vision, and text processing into a single, unified architecture. [26 Feb 2025]
- [Phi-4-reasoning✍️](https://azure.microsoft.com/en-us/blog/one-year-of-phi-small-language-models-making-big-leaps-in-ai/): Phi-4-reasoning, Phi-4-reasoning-plus, Phi-4-mini-reasoning [30 Apr 2025]
- [Phi-4-mini-flash-reasoning✍️](https://azure.microsoft.com/en-us/blog/reasoning-reimagined-introducing-phi-4-mini-flash-reasoning/): 3.8B, 64K context, Single GPU, Decoder-Hybrid-Decoder architecture [9 Jul 2025]
- MiniMaxAI
- Founded in Dec 2021. Shanghai, China.
- [MiniMax-M2✨](https://github.com/MiniMax-AI/MiniMax-M2): Coding and Agent tasks, 230B (10B Active), MoE, a new high ahead of DeepSeek-V3.2 and Kimi K2 
- Mistral
- Founded in April 2023. French tech.
- Model overview [✍️](https://docs.mistral.ai/getting-started/models/)
- [NeMo](https://mistral.ai/news/mistral-nemo/): 12B model with 128k context length that outperforms LLama 3 8B [18 Jul 2024]
- [Mistral OCR](https://mistral.ai/news/mistral-ocr): Precise text recognition with up to 99% accuracy. Multimodal. Browser based [6 Mar 2025]
- Moonshot AI
- Moonshot AI is a Beijing-based Chinese AI company founded in March 2023
- [Kimi-K2✨](https://github.com/MoonshotAI/Kimi-K2): 1T parameter MoE model. MuonClip Optimizer. Agentic Intelligence. [11 Jul 2025]
- [Kimi K2 Thinking✍️](https://moonshotai.github.io/Kimi-K2/thinking.html): The first open-source model beats GPT-5 in Agent benchmark. [7 Nov 2025]
- [Kimi-K2.5✨](https://github.com/MoonshotAI/Kimi-K2.5): Open-source multimodal agentic model by Moonshot AI. [Jan 2026] 
- NVIDIA
- [Nemotron-4 340B](https://research.nvidia.com/publication/2024-06_nemotron-4-340b): Synthetic Data Generation for Training Large Language Models [14 Jun 2024]
- [ollam](https://ollama.com/library?sort=popular): ollama-supported models
- [Open-Sora✨](https://github.com/hpcaitech/Open-Sora): Democratizing Efficient Video Production for All [Mar 2024]
- OpenAI
- [gpt-oss✨](https://github.com/openai/gpt-oss):💡**gpt-oss-120b** and **gpt-oss-20b** are two open-weight language models by OpenAI. [Jun 2025] 
- Qualcomm
- [Qualcomm’s on-device AI models🤗](https://huggingface.co/qualcomm): Bring generative AI to mobile devices [Feb 2024]
- Tencent
- Founded in 1998, Tencent is a Chinese company dedicated to various technology sectors, including social media, gaming, and AI development.
- [Hunyuan-Large](https://arxiv.org/pdf/2411.02265): An open-source MoE model with open weights. [4 Nov 2024] [✨](https://github.com/Tencent/Tencent-Hunyuan-Large) 
- [Hunyuan-T1](https://tencent.github.io/llm.hunyuan.T1/README_EN.html): Reasoning model [21 Mar 2025]
- A list of models: [✨](https://github.com/Tencent-Hunyuan)
- [The LLM Index](https://sapling.ai/llm/index): A list of large language models (LLMs)
- [The mother of all spreadsheets for anyone into LLMs](https://x.com/DataChaz/status/1868708625310699710) [17 Dec 2024]
- [The Open Source AI Definition](https://opensource.org/ai/open-source-ai-definition) [28 Oct 2024]
- xAI
- xAI is an American AI company founded by Elon Musk in March 2023
- [Grok](https://x.ai/blog/grok-os): 314B parameter Mixture-of-Experts (MoE) model. Released under the Apache 2.0 license. Not includeded training code. Developed by JAX [✨](https://github.com/xai-org/grok) [17 Mar 2024]
- [Grok-2 and Grok-2 mini](https://x.ai/blog/grok-2) [13 Aug 2024]
- [Grok-2.5](https://x.com/elonmusk/status/1959379349322313920): Grok 2.5 Goes Open Source [24 Aug 2025]
- [Grok-3](https://x.ai/grok): 200,000 GPUs to train. Grok 3 beats GPT-4o on AIME, GPQA. Grok 3 Reasoning and Grok 3 mini Reasoning. [17 Feb 2025]
- [Grok-4](https://x.ai/news/grok-4): Humanity’s Last Exam, Grok 4 Heavy scored 44.4% [9 Jul 2025]
- [Grok 4.1✍️](https://x.ai/news/grok-4-1) [17 Nov 2025]
- Xiaomi
- Founded in 2010, Xiaomi is a Chinese company known for its innovative consumer electronics and smart home products.
- [Mimo✨](https://github.com/XiaomiMiMo/MiMo): 7B. advanced reasoning for code and math [30 Apr 2025)
- Z.ai
- formerly Zhipu, Beijing-based Chinese AI company founded in March 2019
- [GLM-4.5✨](https://github.com/zai-org/GLM-4.5): An open-source large language model designed for intelligent agents
- [GLM-4.6✍️](https://z.ai/blog/glm-4.6): GLM-4.6: Advanced Agentic, Reasoning and Coding Capabilities [30 Sep 2025]
### **LLM for Domain Specific**
- [AI for Scaling Legal Reform: Mapping and Redacting Racial Covenants in Santa Clara County📑](https://arxiv.org/abs/2503.03888): a fine-tuned open LLM to detect racial covenants in 24 million housing documents, cutting 86,500 hours of manual work. [12 Feb 2025]
- [AlphaChip](https://deepmind.google/discover/blog/how-alphachip-transformed-computer-chip-design/): Reinforcement learning-based model for designing physical chip layouts. [26 Sep 2024]
- [AlphaFold3✨](https://github.com/Ligo-Biosciences/AlphaFold3): Open source implementation of AlphaFold3 [Nov 2023] / [OpenFold✨](https://github.com/aqlaboratory/openfold): PyTorch reproduction of AlphaFold 2 [Sep 2021]  
- [AlphaGenome](https://deepmind.google/discover/blog/alphagenome-ai-for-better-understanding-the-genome): DeepMind’s advanced AI model, launched in June 2025, is designed to analyze the regulatory “dark matter” of the genome—specifically, the 98% of DNA that does not code for proteins but instead regulates when and how genes are expressed. [June 2025]
- [BioGPT📑](https://arxiv.org/abs/2210.10341): Generative Pre-trained Transformer for Biomedical Text Generation and Mining [✨](https://github.com/microsoft/BioGPT) [19 Oct 2022] 
- [BloombergGPT📑](https://arxiv.org/abs/2303.17564): A Large Language Model for Finance [30 Mar 2023]
- [Chai-1✨](https://github.com/chaidiscovery/chai-lab): a multi-modal foundation model for molecular structure prediction [Sep 2024]
- [Code Llama📑](https://arxiv.org/abs/2308.12950): Built on top of Llama 2, free for research and commercial use. [✍️](https://ai.meta.com/blog/code-llama-large-language-model-coding/) / [✨](https://github.com/facebookresearch/codellama) [24 Aug 2023] 
- [DeepSeek-Coder-V2✨](https://github.com/deepseek-ai/DeepSeek-Coder-V2): Open-source Mixture-of-Experts (MoE) code language model [17 Jun 2024]
- [Devin AI](https://preview.devin.ai/): Devin is an AI software engineer developed by Cognition AI [12 Mar 2024]
- [EarthGPT📑](https://arxiv.org/abs/2401.16822): A Universal Multi-modal Large Language Model for Multi-sensor Image Comprehension in Remote Sensing Domain [30 Jan 2024]
- [ESM3: A frontier language model for biology](https://www.evolutionaryscale.ai/blog/esm3-release): Simulating 500 million years of evolution [✨](https://github.com/evolutionaryscale/esm) / [✍️](https://doi.org/10.1101/2024.07.01.600583) [31 Dec 2024] 
- [FrugalGPT📑](https://arxiv.org/abs/2305.05176): LLM with budget constraints, requests are cascaded from low-cost to high-cost LLMs. [✨](https://github.com/stanford-futuredata/FrugalGPT) [9 May 2023] 
- [Galactica📑](https://arxiv.org/abs/2211.09085): A Large Language Model for Science [16 Nov 2022]
- Gemma series
- [Gemma series in Huggingface🤗](https://huggingface.co/google)
- [PaliGemma📑](https://arxiv.org/abs/2407.07726): a 3B VLM [10 Jul 2024]
- [DataGemma✍️](https://blog.google/technology/ai/google-datagemma-ai-llm/) [12 Sep 2024] / [NotebookLM✍️](https://blog.google/technology/ai/notebooklm-audio-overviews/): LLM-powered notebook. free to use, not open-source. [12 Jul 2023]
- [PaliGemma 2📑](https://arxiv.org/abs/2412.03555): VLMs
at 3 different sizes (3B, 10B, 28B) [4 Dec 2024]
- [TxGemma](https://developers.googleblog.com/en/introducing-txgemma-open-models-improving-therapeutics-development/): Therapeutics development [25 Mar 2025]
- [Dolphin Gemma✍️](https://blog.google/technology/ai/dolphingemma/): Decode dolphin communication [14 Apr 2025]
- [MedGemma](https://deepmind.google/models/gemma/medgemma/): Model fine-tuned for biomedical text and image understanding. [20 May 2025]
- [SignGemma](https://x.com/GoogleDeepMind/status/1927375853551235160): Vision-language model for sign language recognition and translation. [27 May 2025)
- [Huggingface StarCoder: A State-of-the-Art LLM for Code🤗](https://huggingface.co/blog/starcoder): [✨🤗](https://huggingface.co/bigcode/starcoder) [May 2023]
- [MechGPT📑](https://arxiv.org/abs/2310.10445): Language Modeling Strategies for Mechanics and Materials [✨](https://github.com/lamm-mit/MeLM) [16 Oct 2023] 
- [MeshGPT](https://nihalsid.github.io/mesh-gpt/): Generating Triangle Meshes with Decoder-Only Transformers [27 Nov 2023]
- [OpenCoder✨](https://github.com/OpenCoder-llm/OpenCoder-llm): 1.5B and 8B base
…(truncated)