tomevault-io Research Methodology: Tips for Empirical Alignment Research
Based on Ethan Perez's post. Applies to highly experimental LLM alignment research (scalable oversight, adversarial robustness, chain-of-thought faithfulness, process-based oversight, model organisms of misalignment).
Core Principles
1. Hard Work Pays Off
- Run as many experiments as you can, tinker a lot, try lots of stuff
- Often many reasonable-sounding ideas need testing—sometimes the 5th or 20th thing is what works
- "Rich get richer" effect: the more you experiment, the better you get at picking the right experiments
- More things work when you try more stuff → easier to stay motivated
2. Rapid Iteration (Velocity)
- Reduce uncertainty at the fastest possible rate (Jacob Steinhardt)
- Get quick feedback and iterate on ideas rapidly
- With LLMs, you can reduce uncertainty really quickly—even with a single message to GPT-4/Claude
- Can gain 1+ OOMs more information per unit time by derisking ideas in the quickest way possible
3. Avoid the "Swamp"
- The swamp = getting stuck when you really want technique X to work but nothing succeeds
- Solution: high velocity—test as many ideas as possible per unit time until you escape
- For LLM research: you should rarely be stuck in a swamp
- If stuck → you've likely exhausted low-hanging fruit on that problem/approach
- There's other low-hanging fruit elsewhere—go pick that instead
4. Low-Hanging Fruit Abundance
- The field is moving so quickly that there's much more low-hanging fruit than almost any other field
- Each new model capability = time to be opportunistic, explore what's now possible
- High velocity is useful for both:
- Picking low-hanging fruit quickly
- Getting through swamps when you must solve a particular problem
Collaboration & Communication
Strong Collaborator Qualities
Ethan puts 70% weight on "getting ideas to work quickly" as criteria. Other qualities:
- Receptive to feedback
- Adds emotional energy rather than draining it
- Transparent/communicative about issues faced
- High-trust relationship where various topics can be discussed easily
- Notices and calls out room for improvement in collaboration
Communication Best Practices
- Overcommunicate: bring up issues during meetings or privately—nip problems in the bud
- Close mentorship is maybe the fastest path to become an expert in a domain
- Take agency: organize coworking, discussion groups, standups—whatever helps
- Get feedback from peers, iterate on project plans
Research Workflow
- Have a clear project plan with motivation and research goals
- List all experiments you can think of running
- Think about milestones and deliverables to stay accountable
- Know what tools are available—sharing tooling increases experimental velocity
Paper Writing
- See ethanperez.net/easy-paper-writing-tips/ for ML paper writing tips
- Write short paragraphs composed of short sentences
- Write comprehensive abstracts
- Seek feedback from a naive audience
- Distinguish confirmation from exploration in your writeup
Related Resources
- Tips and Code for Empirical Research Workflows
- Jacob Steinhardt's blog on reducing uncertainty
- Michael Bernstein's slides on velocity
Converted and distributed by TomeVault — claim your Tome and manage your conversions.