tools-only Message Metadata Feature
Version Implemented: 0.229.001
Overview
The Message Metadata feature introduces comprehensive tracking and display of detailed information about each message in conversations, including timestamps, token usage, model information, processing time, and other relevant metadata that enhances transparency and debugging capabilities.
Purpose
Message Metadata enables users and administrators to:
- Track token usage and costs for each message exchange
- Monitor processing times and performance metrics
- Identify which AI models were used for specific responses
- Debug conversation issues with detailed technical information
- Analyze conversation patterns and efficiency
- Provide transparency in AI interactions
Technical Specifications
Architecture Overview
- Real-time Collection: Metadata captured during message processing
- Structured Storage: Metadata stored alongside message content in Cosmos DB
- UI Integration: Expandable metadata display in chat interface
- Performance Tracking: Processing time and token usage monitoring
Database Schema
{
"message_id": "msg-uuid-12345",
"conversation_id": "conv-uuid-67890",
"user_id": "user-uuid-abcde",
"content": "What is machine learning?",
"role": "user",
"timestamp": "2025-09-12T10:30:00Z",
"metadata": {
"processing_time_ms": 1250,
"token_usage": {
"prompt_tokens": 45,
"completion_tokens": 187,
"total_tokens": 232
},
"model_info": {
"deployment_name": "gpt-4o",
"model_version": "2024-11-30",
"temperature": 0.7,
"max_tokens": 4000
},
"search_metadata": {
"documents_searched": 12,
"relevant_chunks": 3,
"search_time_ms": 340,
"search_query": "machine learning definition"
},
"safety_check": {
"content_safety_enabled": true,
"safety_check_time_ms": 156,
"safety_result": "safe"
},
"response_metadata": {
"finish_reason": "stop",
"created_at": "2025-09-12T10:30:01.250Z",
"response_id": "resp-uuid-fghij"
},
"user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)...",
"ip_address": "192.168.1.100", // Hashed for privacy
"session_id": "session-uuid-klmno"
}
}
Metadata Collection System
class MessageMetadataCollector {
constructor() {
this.startTime = null;
this.endTime = null;
this.metadata = {};
}
startCollection(messageData) {
this.startTime = performance.now();
this.metadata = {
message_id: messageData.message_id,
user_id: messageData.user_id,
conversation_id: messageData.conversation_id,
timestamp: new Date().toISOString(),
user_agent: navigator.userAgent,
session_id: this.getSessionId(),
client_metadata: {
viewport_size: {
width: window.innerWidth,
height: window.innerHeight
},
browser_info: this.getBrowserInfo(),
timezone: Intl.DateTimeFormat().resolvedOptions().timeZone
}
};
}
addTokenUsage(tokenData) {
this.metadata.token_usage = {
prompt_tokens: tokenData.prompt_tokens,
completion_tokens: tokenData.completion_tokens,
total_tokens: tokenData.total_tokens,
estimated_cost: this.calculateCost(tokenData)
};
}
addModelInfo(modelData) {
this.metadata.model_info = {
deployment_name: modelData.deployment,
model_version: modelData.version,
temperature: modelData.temperature,
max_tokens: modelData.max_tokens,
provider: modelData.provider || 'azure_openai'
};
}
addSearchMetadata(searchData) {
this.metadata.search_metadata = {
documents_searched: searchData.total_documents,
relevant_chunks: searchData.returned_chunks,
search_time_ms: searchData.processing_time,
search_query: searchData.query,
search_filters: searchData.filters
};
}
finishCollection(responseData) {
this.endTime = performance.now();
this.metadata.processing_time_ms = Math.round(this.endTime - this.startTime);
this.metadata.response_metadata = {
finish_reason: responseData.finish_reason,
created_at: responseData.created_at,
response_id: responseData.id
};
return this.metadata;
}
calculateCost(tokenData) {
// Token cost calculation based on model pricing
const costPerThousand = {
'gpt-4o': { input: 0.005, output: 0.015 },
'gpt-35-turbo': { input: 0.0015, output: 0.002 }
};
const modelCost = costPerThousand[this.metadata.model_info?.deployment_name] ||
costPerThousand['gpt-4o'];
const inputCost = (tokenData.prompt_tokens / 1000) * modelCost.input;
const outputCost = (tokenData.completion_tokens / 1000) * modelCost.output;
return {
input_cost: inputCost,
output_cost: outputCost,
total_cost: inputCost + outputCost,
currency: 'USD'
};
}
}
Configuration Options
Admin Settings
<div class="message-metadata-settings">
<h5>Message Metadata Configuration</h5>
<div class="form-group">
<div class="form-check">
<input type="checkbox" id="enable-message-metadata" class="form-check-input" checked>
<label class="form-check-label">Enable Message Metadata Collection</label>
<small class="form-text text-muted">Collect detailed information about each message</small>
</div>
</div>
<div class="form-group">
<div class="form-check">
<input type="checkbox" id="show-metadata-to-users" class="form-check-input" checked>
<label class="form-check-label">Show Metadata to Users</label>
<small class="form-text text-muted">Allow users to view message metadata in chat interface</small>
</div>
</div>
<div class="form-group">
<div class="form-check">
<input type="checkbox" id="collect-token-usage" class="form-check-input" checked>
<label class="form-check-label">Collect Token Usage Data</label>
</div>
</div>
<div class="form-group">
<div class="form-check">
<input type="checkbox" id="collect-timing-data" class="form-check-input" checked>
<label class="form-check-label">Collect Processing Time Data</label>
</div>
</div>
<div class="form-group">
<div class="form-check">
<input type="checkbox" id="collect-search-metadata" class="form-check-input" checked>
<label class="form-check-label">Collect Document Search Metadata</label>
</div>
</div>
<div class="form-group">
<label>Privacy Settings</label>
<div class="form-check">
<input type="checkbox" id="hash-ip-addresses" class="form-check-input" checked>
<label class="form-check-label">Hash IP Addresses for Privacy</label>
</div>
<div class="form-check">
<input type="checkbox" id="anonymize-user-agents" class="form-check-input">
<label class="form-check-label">Anonymize User Agent Strings</label>
</div>
</div>
<div class="form-group">
<label>Data Retention</label>
<select class="form-control" id="metadata-retention-days">
<option value="30">30 Days</option>
<option value="90" selected>90 Days</option>
<option value="180">180 Days</option>
<option value="365">1 Year</option>
<option value="-1">Indefinite</option>
</select>
<small class="form-text text-muted">How long to retain detailed metadata</small>
</div>
</div>
Usage Instructions
For End Users
Viewing Message Metadata
- Access Metadata: Click the info icon (ⓘ) next to any message
- Expandable Display: Click "Show Details" to expand full metadata
- Copy Information: Use copy buttons to copy specific metadata values
- Performance Insights: View response times and token usage
Metadata Display Interface
<div class="message-metadata-display">
<div class="metadata-summary">
<span class="metadata-item">
<i class="fas fa-clock"></i>
<span class="metadata-value">1.25s</span>
</span>
<span class="metadata-item">
<i class="fas fa-coins"></i>
<span class="metadata-value">232 tokens</span>
</span>
<span class="metadata-item">
<i class="fas fa-brain"></i>
<span class="metadata-value">GPT-4o</span>
</span>
</div>
<div class="metadata-details" style="display: none;">
<div class="metadata-section">
<h6>Token Usage</h6>
<table class="table table-sm">
<tr>
<td>Prompt Tokens:</td>
<td>45</td>
</tr>
<tr>
<td>Completion Tokens:</td>
<td>187</td>
</tr>
<tr>
<td>Total Tokens:</td>
<td>232</td>
</tr>
<tr>
<td>Estimated Cost:</td>
<td>$0.003</td>
</tr>
</table>
</div>
<div class="metadata-section">
<h6>Model Information</h6>
<table class="table table-sm">
<tr>
<td>Deployment:</td>
<td>gpt-4o</td>
</tr>
<tr>
<td>Version:</td>
<td>2024-11-30</td>
</tr>
<tr>
<td>Temperature:</td>
<td>0.7</td>
</tr>
</table>
</div>
<div class="metadata-section">
<h6>Performance</h6>
<table class="table table-sm">
<tr>
<td>Processing Time:</td>
<td>1,250ms</td>
</tr>
<tr>
<td>Search Time:</td>
<td>340ms</td>
</tr>
<tr>
<td>Safety Check:</td>
<td>156ms</td>
</tr>
</table>
</div>
</div>
<button class="btn btn-sm btn-outline-secondary toggle-details">
Show Details
</button>
</div>
For Administrators
Metadata Analytics Dashboard
class MetadataAnalytics {
async generateUsageReport(dateRange) {
const report = {
totalMessages: 0,
totalTokens: 0,
totalCost: 0,
averageResponseTime: 0,
modelUsage: {},
performanceMetrics: {
fastest_response: null,
slowest_response: null,
average_search_time: 0
}
};
const messages = await this.fetchMessagesInRange(dateRange);
messages.forEach(msg => {
if (msg.metadata) {
report.totalMessages++;
report.totalTokens += msg.metadata.token_usage?.total_tokens || 0;
report.totalCost += msg.metadata.token_usage?.estimated_cost?.total_cost || 0;
// Model usage tracking
const model = msg.metadata.model_info?.deployment_name;
if (model) {
report.modelUsage[model] = (report.modelUsage[model] || 0) + 1;
}
// Performance tracking
const responseTime = msg.metadata.processing_time_ms;
if (responseTime) {
if (!report.performanceMetrics.fastest_response ||
responseTime < report.performanceMetrics.fastest_response) {
report.performanceMetrics.fastest_response = responseTime;
}
if (!report.performanceMetrics.slowest_response ||
responseTime > report.performanceMetrics.slowest_response) {
report.performanceMetrics.slowest_response = responseTime;
}
}
}
});
report.averageResponseTime = this.calculateAverageResponseTime(messages);
return report;
}
}
Metadata Export Functionality
import csv
import json
from datetime import datetime, timedelta
class MetadataExporter:
def export_metadata_csv(self, date_range, include_sensitive=False):
"""Export message metadata to CSV format"""
messages = self.get_messages_with_metadata(date_range)
csv_data = []
for message in messages:
metadata = message.get('metadata', {})
row = {
'message_id': message['message_id'],
'timestamp': message['timestamp'],
'processing_time_ms': metadata.get('processing_time_ms'),
'total_tokens': metadata.get('token_usage', {}).get('total_tokens'),
'estimated_cost': metadata.get('token_usage', {}).get('estimated_cost', {}).get('total_cost'),
'model_deployment': metadata.get('model_info', {}).get('deployment_name'),
'model_version': metadata.get('model_info', {}).get('model_version'),
'documents_searched': metadata.get('search_metadata', {}).get('documents_searched'),
'search_time_ms': metadata.get('search_metadata', {}).get('search_time_ms')
}
# Include sensitive data only if requested and authorized
if include_sensitive:
row['user_id'] = message['user_id']
row['conversation_id'] = message['conversation_id']
row['ip_address'] = metadata.get('ip_address')
csv_data.append(row)
return csv_data
Integration Points
Chat Interface Integration
// Add metadata display to each message
function renderMessageWithMetadata(message) {
const messageElement = document.createElement('div');
messageElement.className = 'chat-message';
// Message content
const contentElement = document.createElement('div');
contentElement.className = 'message-content';
contentElement.innerHTML = message.content;
// Metadata toggle
const metadataToggle = document.createElement('button');
metadataToggle.className = 'btn btn-sm btn-outline-secondary metadata-toggle';
metadataToggle.innerHTML = '<i class="fas fa-info-circle"></i>';
metadataToggle.onclick = () => toggleMetadata(message.message_id);
// Metadata display
const metadataDisplay = document.createElement('div');
metadataDisplay.className = 'message-metadata';
metadataDisplay.id = `metadata-${message.message_id}`;
metadataDisplay.style.display = 'none';
metadataDisplay.innerHTML = renderMetadataDetails(message.metadata);
messageElement.appendChild(contentElement);
messageElement.appendChild(metadataToggle);
messageElement.appendChild(metadataDisplay);
return messageElement;
}
Performance Monitoring Integration
class PerformanceMonitor {
constructor() {
this.thresholds = {
response_time_warning: 5000, // 5 seconds
response_time_critical: 10000, // 10 seconds
token_usage_warning: 4000,
token_usage_critical: 8000
};
}
analyzeMessagePerformance(metadata) {
const alerts = [];
// Check response time
if (metadata.processing_time_ms > this.thresholds.response_time_critical) {
alerts.push({
type: 'critical',
message: `Response time (${metadata.processing_time_ms}ms) exceeded critical threshold`
});
} else if (metadata.processing_time_ms > this.thresholds.response_time_warning) {
alerts.push({
type: 'warning',
message: `Response time (${metadata.processing_time_ms}ms) exceeded warning threshold`
});
}
// Check token usage
const totalTokens = metadata.token_usage?.total_tokens;
if (totalTokens > this.thresholds.token_usage_critical) {
alerts.push({
type: 'critical',
message: `Token usage (${totalTokens}) exceeded critical threshold`
});
} else if (totalTokens > this.thresholds.token_usage_warning) {
alerts.push({
type: 'warning',
message: `Token usage (${totalTokens}) exceeded warning threshold`
});
}
return alerts;
}
}
Privacy and Security Considerations
Data Privacy
import hashlib
import hmac
class MetadataPrivacy:
def __init__(self, secret_key):
self.secret_key = secret_key
def hash_ip_address(self, ip_address):
"""Hash IP addresses for privacy while maintaining uniqueness"""
return hmac.new(
self.secret_key.encode(),
ip_address.encode(),
hashlib.sha256
).hexdigest()[:16]
def anonymize_user_agent(self, user_agent):
"""Remove potentially identifying information from user agent"""
# Remove version numbers and specific browser versions
import re
anonymized = re.sub(r'\d+\.\d+\.\d+\.\d+', 'x.x.x.x', user_agent)
anonymized = re.sub(r'Chrome/\d+\.\d+\.\d+\.\d+', 'Chrome/xxx', anonymized)
return anonymized
def sanitize_metadata_for_export(self, metadata, user_role):
"""Remove sensitive information based on user role"""
sanitized = metadata.copy()
if user_role != 'Admin':
# Remove sensitive fields for non-admin users
sanitized.pop('ip_address', None)
sanitized.pop('session_id', None)
sanitized.pop('user_agent', None)
return sanitized
Data Retention
from datetime import datetime, timedelta
class MetadataRetention:
def cleanup_old_metadata(self, retention_days):
"""Remove metadata older than specified retention period"""
cutoff_date = datetime.utcnow() - timedelta(days=retention_days)
# Query messages older than cutoff date
old_messages = self.cosmos_client.query_items(
container=self.conversations_container,
query="SELECT * FROM c WHERE c.timestamp < @cutoff_date",
parameters=[{"name": "@cutoff_date", "value": cutoff_date.isoformat()}]
)
for message in old_messages:
# Remove detailed metadata but keep basic info
if 'metadata' in message:
# Keep only essential metadata
essential_metadata = {
'processing_time_ms': message['metadata'].get('processing_time_ms'),
'total_tokens': message['metadata'].get('token_usage', {}).get('total_tokens'),
'model_deployment': message['metadata'].get('model_info', {}).get('deployment_name')
}
message['metadata'] = essential_metadata
# Update the document
self.cosmos_client.upsert_item(
container=self.conversations_container,
body=message
)
Testing and Validation
Functional Testing
- Verify metadata collection for all message types (user, assistant, system)
- Test metadata display toggle functionality in chat interface
- Validate token usage calculations for different models
- Confirm processing time measurements are accurate
Performance Testing
- Measure impact of metadata collection on message processing time
- Test metadata storage and retrieval performance with large datasets
- Validate memory usage during extended conversations with metadata
Privacy Testing
- Verify IP address hashing works correctly
- Test data retention cleanup functionality
- Confirm sensitive data removal for non-admin users
- Validate export functionality respects privacy settings
Known Limitations
- Metadata collection adds slight overhead to message processing
- Some metadata fields may not be available for all message types
- Token usage calculations are estimates based on known model pricing
- Processing time includes network latency and may vary significantly
Future Enhancements
- Real-time performance alerting for slow responses
- Advanced analytics dashboard with trend analysis
- Integration with external monitoring systems
- Custom metadata fields for organization-specific tracking
- Automated performance optimization recommendations based on metadata analysis