adu2021 Technique: Task-Guided LLM-Based Observation Filtering
Web agents navigate complex pages by parsing accessibility trees (AxTree)—DOM representations containing hundreds or thousands of elements. Traditional agents pass the entire tree as context, consuming excessive tokens and slowing inference. FocusAgent uses a small language model to selectively extract only task-relevant content lines from the accessibility tree at each step.
The key insight is that web navigation requires understanding page state and action consequences within a specific task context. A general embedding-based retrieval would miss crucial interactive relationships. By having a dedicated LLM analyze what matters for the current goal, the agent retains planning-relevant information while discarding layout noise, redundant navigation elements, and off-topic content.
Core Concept
FocusAgent employs a two-level filtering architecture. A base web agent generates the full accessibility tree observation. A smaller "retriever" LLM (GPT-4o-mini) examines this tree along with a task goal and current step context, then identifies which lines are relevant to future decision-making. The filtered observation becomes the input for the main agent policy.
The retriever uses a "soft" strategy: when uncertain, it errs toward inclusion. This preserves task-critical information at the cost of slightly higher token usage, avoiding the risk of filtering away essential context.
Architecture Overview
- Observation Layer: Generate full accessibility tree with numbered lines for each DOM element
- Retriever Module: Lightweight LLM that analyzes the tree and task context
- Filtering Decision: Line-level selection producing a compressed tree
- Policy Input: Main agent receives filtered observation as context
- Feedback Loop: Agent actions and results inform the next observation filtering step
Implementation Steps
Create a function to generate numbered accessibility tree lines. Each element receives a unique ID for reference.
def generate_numbered_axtree(page_html, include_attributes=True):
"""
Parse page HTML into numbered accessibility tree lines.
Args:
page_html: Raw HTML from the web page
include_attributes: Include element attributes in lines
Returns:
axtree_lines: List of (line_number, element_text, attributes) tuples
"""
from html.parser import HTMLParser
axtree_lines = []
line_num = 1
# Parse HTML and build tree representation
for element in parse_html_to_tree(page_html):
element_text = extract_element_text(element)
attributes = extract_attributes(element) if include_attributes else {}
# Format: "[1] Button: Click here" or "[1] Link: https://example.com - Example Page"
formatted_line = f"[{line_num}] {element_text}"
if attributes and 'href' in attributes:
formatted_line += f" - {attributes['href']}"
axtree_lines.append((line_num, formatted_line, attributes))
line_num += 1
return axtree_lines
Implement the retriever LLM call that identifies relevant lines.
def filter_axtree_with_retriever(axtree_lines, task_goal, current_step,
retriever_model="gpt-4o-mini"):
"""
Use a lightweight LLM to filter accessibility tree by task relevance.
Args:
axtree_lines: List of formatted accessibility tree lines
task_goal: The overall task objective (e.g., "Find and click on checkout button")
current_step: Current action context (e.g., what was just attempted)
retriever_model: LLM to use for filtering decision
Returns:
relevant_line_numbers: Set of line indices to retain
"""
axtree_text = "\n".join([line[1] for line in axtree_lines])
prompt = f"""Task Goal: {task_goal}
Current Step: {current_step}
Accessibility Tree:
{axtree_text}
Which lines are most relevant for the next decision? List line numbers [1], [2], etc.
Be inclusive when uncertain—better to keep slightly verbose context than miss crucial information.
Return only line numbers, one per line."""
response = call_llm(retriever_model, prompt)
# Parse response to extract line numbers
relevant_lines = extract_line_numbers_from_response(response)
return relevant_lines
Build the compressed observation by selecting only relevant lines.
def compress_observation(axtree_lines, relevant_line_numbers):
"""
Extract relevant lines and reconstruct accessibility tree.
Args:
axtree_lines: Full accessibility tree
relevant_line_numbers: Set of line indices to keep
Returns:
compressed_tree: Filtered accessibility tree as text
compression_ratio: Fraction of lines retained
"""
compressed_lines = [
line[1] for line in axtree_lines
if line[0] in relevant_line_numbers
]
compressed_tree = "\n".join(compressed_lines)
compression_ratio = len(compressed_lines) / len(axtree_lines)
return compressed_tree, compression_ratio
Integrate into the web agent's observation-action loop.
def web_agent_step_with_focus(agent_policy, page_html, task_goal,
action_history, retriever_model="gpt-4o-mini"):
"""
Single step of web agent with FocusAgent filtering.
Args:
agent_policy: Main agent LLM policy
page_html: Current page HTML
task_goal: Task objective
action_history: Previous actions taken
retriever_model: Retriever LLM for filtering
Returns:
action: Next action to execute (e.g., click element [42])
compressed_observation: The filtered context used for decision
"""
# Generate full accessibility tree
axtree_lines = generate_numbered_axtree(page_html)
# Get current step context from recent actions
current_step = action_history[-1] if action_history else "Initial page load"
# Filter using lightweight retriever
relevant_lines = filter_axtree_with_retriever(
axtree_lines, task_goal, current_step, retriever_model
)
# Compress observation
compressed_obs, ratio = compress_observation(axtree_lines, relevant_lines)
# Main agent policy uses compressed observation
prompt = f"""Goal: {task_goal}
Available elements:
{compressed_obs}
What is the next action?"""
action = agent_policy(prompt)
return action, compressed_obs
Practical Guidance
| Aspect | Recommendation | Notes |
|---|---|---|
| Retriever model | GPT-4o-mini or similar | Small models sufficient; balance cost vs. accuracy |
| Inclusion strategy | Soft/erring toward inclusion | Avoids filtering critical elements at cost of slightly higher tokens |
| Filter frequency | Every step | Recompute for each observation to capture changing relevance |
| Batch operations | Apply filter to multiple pages | Amortize retriever LLM cost across agents |
| Compression target | 50-80% reduction | Achievable on most pages; higher reduction increases risk |
| When to use | Complex multi-step web navigation | E-commerce, SaaS, information retrieval tasks |
| When NOT to use | Highly visual tasks or pages with minimal text | May filter out layout-critical information |
| Common pitfall | Over-filtering with aggressive thresholds | Safety margin is important; preserve questionable elements |
When to Use FocusAgent
- Web agents navigating e-commerce sites, SaaS dashboards, or complex document sites
- Scenarios where token budget is constrained (cost-sensitive inference)
- Tasks requiring multiple steps with evolving context
- Pages with hundreds+ DOM elements
When NOT to Use FocusAgent
- Single-page lookups where compression adds overhead
- Highly visual tasks where layout structure is essential
- Pages with sparse text and important implicit structure
- Real-time systems where latency of additional LLM call is prohibitive
Common Pitfalls
- Retriever timeout: Set timeouts to fall back to full tree if retriever is slow
- Loss of critical navigation elements: Headers/footers often filtered despite importance for multi-step tasks
- Lack of context carry-over: Each step's filtering ignores previous steps' progress; maintain memory
- Attribute loss: Filtering removes element IDs/classes needed for reliable clicking—preserve these
- Task goal drift: Retriever may misinterpret task scope; provide precise goal descriptions