Unicode Table Rendering Fix

Unicode Table Rendering Fix

Fixed in version: 0.229.014

Overview

Fixed the issue where AI-generated tables (particularly from the ESAM Agent) were not rendering as proper HTML tables in the chat interface. The problem was that AI agents were generating Unicode box-drawing tables instead of markdown table format.

Root Cause Analysis

The ESAM Agent was generating table data using Unicode box-drawing characters:

┌───────────┬─────────────┬───────────────┬───────┬─────────────────┐
│ LicenseID │ ProductName │ TotalQuantity │ InUse │ AvailableQuantity │
├───────────┼─────────────┼───────────────┼───────┼─────────────────┤
│ 1         │ Office 365  │ 229           │ 5     │ 224             │

This format, while visually appealing in plain text, cannot be parsed by markdown processors as table syntax, resulting in the content being displayed as plain text rather than rendered HTML tables.

Solution Implemented

Fixed in version: 0.229.004

Technical Changes

1. Enhanced Message Processing Pipeline

Modified chat-messages.js to include table preprocessing before markdown parsing:

// Parse content
let cleaned = messageContent.trim().replace(/\n{3,}/g, "\n\n");
cleaned = cleaned.replace(/(\bhttps?:\/\/\S+)(%5D|\])+/gi, (_, url) => url);
const withInlineCitations = parseCitations(cleaned);
const withUnwrappedTables = unwrapTablesFromCodeBlocks(withInlineCitations);
const withMarkdownTables = convertUnicodeTableToMarkdown(withUnwrappedTables);
const htmlContent = DOMPurify.sanitize(marked.parse(withMarkdownTables));

2. Unicode Table Conversion Function

Added convertUnicodeTableToMarkdown() function that:

• Detects Unicode box-drawing table patterns
• Extracts header and data rows from Unicode table structure
• Converts to proper markdown table format
• Preserves original text content outside the table

3. Enhanced Code Block Processing

Improved unwrapTablesFromCodeBlocks() function that:

• Detects markdown tables mistakenly wrapped in code blocks
• Unwraps them to allow proper table rendering
• Preserves legitimate code blocks

Files Modified

• application/single_app/static/js/chat/chat-messages.js
• application/single_app/config.py (version update)

Testing Coverage

Created comprehensive test files:

• functional_tests/test_table_markdown_analysis.py
• functional_tests/test_unicode_table_conversion.py
• functional_tests/unicode_conversion_direct_test.html

Impact and Benefits

Before Fix

• Unicode tables displayed as plain text
• No visual structure or formatting
• Poor readability for tabular data
• Inconsistent user experience

After Fix

• Unicode tables automatically converted to HTML tables
• Proper styling with Bootstrap CSS
• Responsive design with hover effects
• Consistent table formatting across all AI responses

Usage Examples

Input (Unicode Table)

┌─────────┬───────────┐
│ Product │ Licenses  │
├─────────┼───────────┤
│ Office  │ 229       │
│ Adobe   │ 187       │
└─────────┴───────────┘

Output (HTML Table)

Renders as a properly formatted HTML table with:

• Header styling (blue gradient background)
• Alternating row colors
• Hover effects
• Responsive design
• Professional appearance

Configuration

No configuration changes required. The fix is automatically applied to all AI message processing.

Compatibility

• Backward Compatible: Existing markdown tables continue to work
• Forward Compatible: Supports both Unicode and markdown table formats
• Agent Agnostic: Works with any AI agent generating Unicode tables
• Performance: Minimal processing overhead with efficient regex patterns

Known Limitations

1. Large Tables: Automatically limits display to first 10 rows for performance
2. Complex Tables: Basic table structures only (no nested tables or complex formatting)
3. Unicode Variants: Supports standard box-drawing characters only

Maintenance Notes

The table conversion logic is contained in the convertUnicodeTableToMarkdown() function. Future enhancements can be made by:

• Adding support for additional Unicode table formats
• Implementing table pagination for large datasets
• Adding configuration options for table display limits

Testing and Validation

All functionality has been validated through:

• Direct conversion testing with ESAM Agent output
• Regression testing with existing markdown tables
• Cross-browser compatibility testing
• Performance impact assessment

The fix successfully addresses the original issue while maintaining compatibility with existing table functionality.