Toolsafe Agent Safety

Overview

Implement a safety layer that monitors LLM agent tool invocations in real-time and prevents potentially harmful actions before execution. The system analyzes interaction history to identify risky requests and tool-usage patterns that could lead to data exfiltration, privilege escalation, or prompt injection attacks.

When to Use

• For agents that invoke external tools (file systems, APIs, databases)
• When operating in untrusted or adversarial environments
• For systems where tool misuse could cause financial, privacy, or security harm
• When agents interact with user-supplied prompts or content

When NOT to Use

• For sandboxed environments with no external tool access
• When tool invocation is fully controlled by trusted operators
• For read-only tool operations with no side effects
• In low-stakes applications where errors are harmless

Key Technical Components

TS-Guard: Multi-Task Risk Detection Model

Train a smaller guardrail model that analyzes agent interaction history to identify risky tool invocations.

# Multi-task risk detection
class TSGuard:
    def __init__(self, model_name="ts-guard-base"):
        self.model = load_model(model_name)  # Smaller, faster model

    def assess_tool_risk(self, interaction_history, tool_name, tool_args):
        """Assess risk of proposed tool invocation"""
        context = format_interaction_history(interaction_history)
        risk_assessment = self.model.predict({
            "context": context,
            "tool": tool_name,
            "args": tool_args
        })
        return {
            "harmfulness_score": risk_assessment["harm"],
            "attack_likelihood": risk_assessment["attack_likelihood"],
            "risk_category": risk_assessment["category"]
        }

Risk Category Classification

Identify specific types of risks to enable targeted mitigation.

# Risk categories
RISK_CATEGORIES = {
    "data_exfiltration": {
        "severity": "high",
        "indicators": ["write_to_external", "network_transmission", "log_access"]
    },
    "privilege_escalation": {
        "severity": "high",
        "indicators": ["permission_change", "sudo_execution", "admin_access"]
    },
    "prompt_injection": {
        "severity": "medium",
        "indicators": ["unescaped_input", "dynamic_query_construction"]
    },
    "resource_exhaustion": {
        "severity": "medium",
        "indicators": ["infinite_loop", "memory_allocation", "cpu_intensive"]
    }
}

TS-Flow: Integrated Reasoning with Guardrails

Incorporate guardrail feedback directly into agent behavior.

# Integrated safety reasoning
class TSFlow:
    def __init__(self, agent, guard):
        self.agent = agent
        self.guard = guard

    def step_with_safety(self, observation):
        """Execute agent step with safety monitoring"""
        # Get agent's proposed tool invocation
        action = self.agent.decide(observation)

        # Assess risk
        risk = self.guard.assess_tool_risk(
            self.agent.history,
            action["tool"],
            action["args"]
        )

        # If risky, modify action or reject
        if risk["harmfulness_score"] > THRESHOLD:
            action = self.mitigate_risk(action, risk)
            if action is None:
                return self.agent.handle_blocked_action(risk)

        return action

Interaction History Analysis

Maintain and analyze agent history to detect attack patterns.

# Pattern-based attack detection
def detect_attack_patterns(history):
    """Identify sequences indicating prompt injection or social engineering"""
    patterns = []

    # Check for escalating privileges
    if has_increasing_permissions(history):
        patterns.append("privilege_escalation_attempt")

    # Check for credential extraction attempts
    if contains_credential_requests(history):
        patterns.append("credential_harvesting")

    # Check for repetitive denied attempts
    if has_repeated_blocked_actions(history):
        patterns.append("persistent_attack_pattern")

    return patterns

Adaptive Risk Thresholding

Adjust risk tolerance based on context and task requirements.

# Context-aware thresholding
class AdaptiveThreshold:
    def get_threshold(self, context):
        """Determine risk threshold based on task and environment"""
        base_threshold = 0.5

        if context["environment"] == "untrusted":
            return base_threshold * 0.7  # More strict
        elif context["task_criticality"] == "high":
            return base_threshold * 0.8
        else:
            return base_threshold

Performance Characteristics

• Harmful tool invocation reduction: 65% average across ReAct-style agents
• Task success rate improvement: ~10% on prompt injection scenarios
• Runtime overhead: <50ms per tool invocation assessment
• Detection precision: 86.7%, Recall: 82.5%

Integration Pattern

1. Initialize TS-Guard model and load interaction history
2. Before each tool invocation, call assess_tool_risk()
3. If risk score exceeds threshold, apply mitigation
4. Log all assessments for audit trail
5. Update guardrail model periodically with new attack patterns

Mitigation Strategies

• Rejection: Block invocation entirely
• Modification: Rewrite unsafe arguments (e.g., sanitize file paths)
• Confirmation: Request human approval for medium-risk actions
• Sandboxing: Execute in restricted environment with reduced permissions

References

• LLM agents are vulnerable to prompt injection through tool invocations
• Step-level monitoring is more effective than outcome-only verification
• Real-time guardrails improve both safety and task success rates