Generative AI Guide
A skill providing a comprehensive, curated guide to generative AI research and practice, covering large language models (LLMs), diffusion models, transformer architectures, prompt engineering, and evaluation methodologies. Based on the awesome-generative-ai-guide repository (25K stars), this skill equips researchers with structured knowledge of the rapidly evolving generative AI landscape.
Overview
Generative AI has become one of the most active areas of research across computer science, with implications spanning natural language processing, computer vision, audio synthesis, code generation, scientific discovery, and creative applications. The pace of development makes it challenging for researchers to maintain a current understanding of the field. This skill provides a structured map of the generative AI landscape, organized by topic and application area, with guidance on key papers, methods, and practical considerations.
Whether you are an AI researcher staying current with the field, a domain scientist exploring how generative AI can accelerate your work, or a student entering the field, this skill provides the orientation and resources needed to navigate the space effectively.
Large Language Models
Architecture Foundations
- Transformer architecture: self-attention mechanism, positional encoding, layer normalization
- Scaling laws: the relationship between model size, data, compute, and performance
- Training objectives: causal language modeling, masked language modeling, instruction tuning
- Context windows: evolution from 512 tokens to 100K+ tokens and associated techniques
- Mixture of Experts (MoE): sparse activation for efficient scaling
Key Model Families
- GPT series (OpenAI): decoder-only architecture, scaling-driven approach
- Claude series (Anthropic): emphasis on safety, instruction following, and long context
- Llama series (Meta): open-weight models enabling community research
- Gemini series (Google): multimodal from the ground up
- Open-source ecosystem: Mistral, Qwen, DeepSeek, and community fine-tunes
Training Pipeline
- Pre-training: large-scale unsupervised learning on web-scale text corpora
- Supervised fine-tuning (SFT): training on high-quality instruction-response pairs
- Reinforcement learning from human feedback (RLHF): aligning outputs with human preferences
- Direct preference optimization (DPO): simplified alignment without reward models
- Constitutional AI: self-improvement using principle-based critique
Inference Optimization
- Quantization: reducing model precision (FP16, INT8, INT4) for faster inference
- KV-cache optimization: efficient memory management for long sequences
- Speculative decoding: using small models to draft and large models to verify
- Batching strategies: continuous batching for throughput optimization
- Serving frameworks: vLLM, TGI, and other high-performance inference engines
Diffusion Models
Core Concepts
- Forward process: gradually adding noise to data until reaching pure noise
- Reverse process: learning to denoise step by step to generate new data
- Score matching: estimating the gradient of the data distribution
- Classifier-free guidance: controlling generation fidelity and diversity
- Latent diffusion: operating in compressed latent space for efficiency
Key Architectures
- DDPM (Denoising Diffusion Probabilistic Models): foundational formulation
- Stable Diffusion: latent space diffusion with text conditioning
- DALL-E series: text-to-image generation with CLIP-based conditioning
- Imagen: text-to-image with cascaded diffusion models
- Video diffusion models: extending to temporal generation
Applications in Research
- Molecular generation: designing new drug candidates and materials
- Protein structure prediction: generating plausible protein conformations
- Scientific data augmentation: creating synthetic training data
- Image restoration: denoising, super-resolution, inpainting for microscopy
- Simulation acceleration: approximating expensive physical simulations
Prompt Engineering
Fundamental Techniques
- Zero-shot prompting: direct instruction without examples
- Few-shot prompting: providing examples to establish the desired pattern
- Chain-of-thought (CoT): requesting step-by-step reasoning
- Self-consistency: sampling multiple reasoning chains and selecting the majority
- Tree of thought: exploring multiple reasoning branches systematically
Advanced Strategies
- ReAct (Reasoning + Acting): interleaving reasoning with tool use
- Retrieval-augmented generation (RAG): grounding responses in retrieved documents
- Program-aided language models: generating and executing code for precise computation
- Structured output: constraining generation to valid JSON, XML, or other formats
- Multi-agent prompting: orchestrating multiple LLM instances for complex tasks
Research-Specific Prompting
- Literature synthesis: prompting for balanced integration of multiple sources
- Hypothesis generation: structured prompts for creative scientific reasoning
- Code debugging: providing error context and asking for systematic diagnosis
- Data analysis: chaining prompts through exploratory analysis to interpretation
- Writing assistance: iterative refinement prompts that preserve the author's voice
Evaluation and Benchmarks
Language Model Evaluation
- Perplexity: intrinsic measure of model quality on held-out text
- MMLU: massive multi-task language understanding across 57 subjects
- HumanEval: code generation benchmark with function completion tasks
- MT-Bench: multi-turn conversation quality assessment
- Arena Elo: head-to-head comparison ratings from human preferences
Generation Quality Metrics
- FID (Frechet Inception Distance): image generation quality and diversity
- CLIP score: text-image alignment for conditional generation
- BLEU, ROUGE: text generation overlap metrics (limited but widely used)
- Human evaluation: gold standard requiring careful protocol design
- Calibration: measuring whether model confidence matches actual accuracy
Safety and Alignment Evaluation
- Red-teaming: adversarial testing for harmful outputs
- Bias benchmarks: measuring demographic and cultural biases
- Hallucination detection: identifying fabricated facts in generated text
- Instruction following: measuring compliance with complex multi-step instructions
- Robustness testing: evaluating consistency under paraphrased inputs
Integration with Research-Claw
This skill provides the Research-Claw agent with generative AI domain expertise:
- Help researchers understand and apply generative AI techniques to their domain
- Guide model selection based on task requirements and resource constraints
- Assist with prompt engineering for research-specific applications
- Connect with analysis skills for evaluating generative model outputs
- Support writing skills with knowledge of the latest developments for literature reviews
Best Practices
- Stay current by monitoring key conferences (NeurIPS, ICML, ICLR, ACL, CVPR) and arXiv
- Distinguish between benchmark performance and real-world applicability
- Consider computational costs and environmental impact when selecting models
- Evaluate models on your specific task rather than relying solely on leaderboard rankings
- Document prompt strategies and model versions for reproducibility
- Be aware of the limitations: hallucination, bias, and sensitivity to prompt phrasing
1---2name: generative-ai-guide3description: Curated guide to generative AI covering LLMs and diffusion models4---5
6# Generative AI Guide
7
8A skill providing a comprehensive, curated guide to generative AI research and practice, covering large language models (LLMs), diffusion models, transformer architectures, prompt engineering, and evaluation methodologies. Based on the awesome-generative-ai-guide repository (25K stars), this skill equips researchers with structured knowledge of the rapidly evolving generative AI landscape.
9
10## Overview
11
12Generative AI has become one of the most active areas of research across computer science, with implications spanning natural language processing, computer vision, audio synthesis, code generation, scientific discovery, and creative applications. The pace of development makes it challenging for researchers to maintain a current understanding of the field. This skill provides a structured map of the generative AI landscape, organized by topic and application area, with guidance on key papers, methods, and practical considerations.
13
14Whether you are an AI researcher staying current with the field, a domain scientist exploring how generative AI can accelerate your work, or a student entering the field, this skill provides the orientation and resources needed to navigate the space effectively.
15
16## Large Language Models
17
18**Architecture Foundations**
19- Transformer architecture: self-attention mechanism, positional encoding, layer normalization
20- Scaling laws: the relationship between model size, data, compute, and performance
21- Training objectives: causal language modeling, masked language modeling, instruction tuning
22- Context windows: evolution from 512 tokens to 100K+ tokens and associated techniques
23- Mixture of Experts (MoE): sparse activation for efficient scaling
24
25**Key Model Families**
26- GPT series (OpenAI): decoder-only architecture, scaling-driven approach
27- Claude series (Anthropic): emphasis on safety, instruction following, and long context
28- Llama series (Meta): open-weight models enabling community research
29- Gemini series (Google): multimodal from the ground up
30- Open-source ecosystem: Mistral, Qwen, DeepSeek, and community fine-tunes
31
32**Training Pipeline**
33- Pre-training: large-scale unsupervised learning on web-scale text corpora
34- Supervised fine-tuning (SFT): training on high-quality instruction-response pairs
35- Reinforcement learning from human feedback (RLHF): aligning outputs with human preferences
36- Direct preference optimization (DPO): simplified alignment without reward models
37- Constitutional AI: self-improvement using principle-based critique
38
39**Inference Optimization**
40- Quantization: reducing model precision (FP16, INT8, INT4) for faster inference
41- KV-cache optimization: efficient memory management for long sequences
42- Speculative decoding: using small models to draft and large models to verify
43- Batching strategies: continuous batching for throughput optimization
44- Serving frameworks: vLLM, TGI, and other high-performance inference engines
45
46## Diffusion Models
47
48**Core Concepts**
49- Forward process: gradually adding noise to data until reaching pure noise
50- Reverse process: learning to denoise step by step to generate new data
51- Score matching: estimating the gradient of the data distribution
52- Classifier-free guidance: controlling generation fidelity and diversity
53- Latent diffusion: operating in compressed latent space for efficiency
54
55**Key Architectures**
56- DDPM (Denoising Diffusion Probabilistic Models): foundational formulation
57- Stable Diffusion: latent space diffusion with text conditioning
58- DALL-E series: text-to-image generation with CLIP-based conditioning
59- Imagen: text-to-image with cascaded diffusion models
60- Video diffusion models: extending to temporal generation
61
62**Applications in Research**
63- Molecular generation: designing new drug candidates and materials
64- Protein structure prediction: generating plausible protein conformations
65- Scientific data augmentation: creating synthetic training data
66- Image restoration: denoising, super-resolution, inpainting for microscopy
67- Simulation acceleration: approximating expensive physical simulations
68
69## Prompt Engineering
70
71**Fundamental Techniques**
72- Zero-shot prompting: direct instruction without examples
73- Few-shot prompting: providing examples to establish the desired pattern
74- Chain-of-thought (CoT): requesting step-by-step reasoning
75- Self-consistency: sampling multiple reasoning chains and selecting the majority
76- Tree of thought: exploring multiple reasoning branches systematically
77
78**Advanced Strategies**
79- ReAct (Reasoning + Acting): interleaving reasoning with tool use
80- Retrieval-augmented generation (RAG): grounding responses in retrieved documents
81- Program-aided language models: generating and executing code for precise computation
82- Structured output: constraining generation to valid JSON, XML, or other formats
83- Multi-agent prompting: orchestrating multiple LLM instances for complex tasks
84
85**Research-Specific Prompting**
86- Literature synthesis: prompting for balanced integration of multiple sources
87- Hypothesis generation: structured prompts for creative scientific reasoning
88- Code debugging: providing error context and asking for systematic diagnosis
89- Data analysis: chaining prompts through exploratory analysis to interpretation
90- Writing assistance: iterative refinement prompts that preserve the author's voice
91
92## Evaluation and Benchmarks
93
94**Language Model Evaluation**
95- Perplexity: intrinsic measure of model quality on held-out text
96- MMLU: massive multi-task language understanding across 57 subjects
97- HumanEval: code generation benchmark with function completion tasks
98- MT-Bench: multi-turn conversation quality assessment
99- Arena Elo: head-to-head comparison ratings from human preferences
100
101**Generation Quality Metrics**
102- FID (Frechet Inception Distance): image generation quality and diversity
103- CLIP score: text-image alignment for conditional generation
104- BLEU, ROUGE: text generation overlap metrics (limited but widely used)
105- Human evaluation: gold standard requiring careful protocol design
106- Calibration: measuring whether model confidence matches actual accuracy
107
108**Safety and Alignment Evaluation**
109- Red-teaming: adversarial testing for harmful outputs
110- Bias benchmarks: measuring demographic and cultural biases
111- Hallucination detection: identifying fabricated facts in generated text
112- Instruction following: measuring compliance with complex multi-step instructions
113- Robustness testing: evaluating consistency under paraphrased inputs
114
115## Integration with Research-Claw
116
117This skill provides the Research-Claw agent with generative AI domain expertise:
118
119- Help researchers understand and apply generative AI techniques to their domain
120- Guide model selection based on task requirements and resource constraints
121- Assist with prompt engineering for research-specific applications
122- Connect with analysis skills for evaluating generative model outputs
123- Support writing skills with knowledge of the latest developments for literature reviews
124
125## Best Practices
126
127- Stay current by monitoring key conferences (NeurIPS, ICML, ICLR, ACL, CVPR) and arXiv
128- Distinguish between benchmark performance and real-world applicability
129- Consider computational costs and environmental impact when selecting models
130- Evaluate models on your specific task rather than relying solely on leaderboard rankings
131- Document prompt strategies and model versions for reproducibility
132- Be aware of the limitations: hallucination, bias, and sensitivity to prompt phrasing
gabrielmoreira