Content here
", "content_type": "text/html", "base_url": "https://example.com", "markdown_engine": "html2text" } ``` ### Convert Local File ```json { "file_path": "./document.pdf", "include_images": true, "clean_content": true } ``` ## Integration ### With ContextForge ```bash # Start the URL-to-markdown server via HTTP make serve-http-fastmcp # Register with ContextForge curl -X POST http://localhost:8000/gateways \ -H "Content-Type: application/json" \ -d '{ "name": "url-to-markdown", "url": "http://localhost:9000", "description": "Universal content to markdown conversion server" }' ``` ### Programmatic Usage ```python import asyncio from mcp import ClientSession, StdioServerParameters from mcp.client.stdio import stdio_client async def convert_content(): server_params = StdioServerParameters( command="python", args=["-m", "url_to_markdown_server.server_fastmcp"] ) async with stdio_client(server_params) as (read, write): async with ClientSession(read, write) as session: await session.initialize() # Convert a web page result = await session.call_tool("convert_url", { "url": "https://example.com/article", "markdown_engine": "readability", "clean_content": True }) print(result.content[0].text) asyncio.run(convert_content()) ``` ## Supported Formats ### Web Content - **HTML/XHTML**: Full HTML parsing and conversion - **XML**: Basic XML to markdown conversion - **JSON**: Structured JSON to markdown ### Document Formats - **PDF**: Text extraction with PyMuPDF - **DOCX**: Microsoft Word documents - **PPTX**: PowerPoint presentations - **XLSX**: Excel spreadsheets - **TXT**: Plain text files ## Conversion Engines ### HTML-to-Markdown Engines #### html2text (Recommended) - Most accurate HTML parsing - Excellent link and image handling - Configurable output options - Best for general web content #### markdownify - Clean, minimal output - Good for simple HTML - Flexible configuration options - Fast processing #### beautifulsoup (Custom) - Intelligent content extraction - Removes navigation and sidebar elements - Good for complex websites - Custom markdown generation #### readability - Extracts main article content - Removes ads and navigation - Best for news articles and blog posts - Clean, focused output #### basic (Fallback) - No external dependencies - Basic regex-based conversion - Always available - Limited functionality ## Response Formats ### Successful Conversion ```json { "success": true, "conversion_id": "uuid-here", "url": "https://example.com/article", "content_type": "text/html", "markdown": "# Article Title\n\nThis is the converted content...", "length": 1542, "engine": "readability", "metadata": { "original_size": 45123, "compression_ratio": 0.034, "processing_time": 1234567890 } } ``` ### Batch Conversion Response ```json { "success": true, "batch_id": "uuid-here", "total_urls": 3, "successful": 2, "failed": 1, "results": [ { "success": true, "url": "https://example.com/page1", "markdown": "# Page 1\n\nContent...", "engine": "html2text" }, { "success": false, "url": "https://example.com/page2", "error": "HTTP 404: Not Found" } ] } ``` ### Error Response ```json { "success": false, "error": "Request timeout after 30 seconds", "conversion_id": "uuid-here" } ``` ## Engine Comparison | Engine | Quality | Speed | Dependencies | Best For | |--------|---------|-------|--------------|----------| | html2text | ⭐⭐⭐⭐⭐ | ⭐⭐⭐⭐ | html2text | General web content | | readability | ⭐⭐⭐⭐⭐ | ⭐⭐⭐ | readability-lxml | News articles, blogs | | markdownify | ⭐⭐⭐⭐ | ⭐⭐⭐⭐⭐ | markdownify | Simple HTML | | beautifulsoup | ⭐⭐⭐ | ⭐⭐⭐ | beautifulsoup4 | Complex sites | | basic | ⭐⭐ | ⭐⭐⭐⭐⭐ | None | Fallback option | ## Advanced Features ### Content Cleaning - Removes excessive whitespace - Fixes heading spacing - Optimizes list formatting - Removes empty links - Standardizes formatting ### Image Processing - Extracts image URLs - Resolves relative image paths - Handles different image formats - Optional image size filtering ### Link Handling - Preserves all link types - Resolves relative URLs - Maintains link text and structure - Optional link filtering ### Error Recovery - Automatic fallback to alternative engines - Graceful handling of network issues - Comprehensive error reporting - Retry mechanisms for transient failures ## Use Cases ### Documentation Conversion ```python # Convert API documentation { "url": "https://docs.example.com/api/reference", "markdown_engine": "html2text", "include_links": True, "clean_content": True } ``` ### Research Paper Processing ```python # Convert academic papers { "url": "https://arxiv.org/pdf/2301.12345.pdf", "clean_content": True } ``` ### News Article Extraction ```python # Extract clean article content { "url": "https://news.example.com/article/123", "extraction_method": "readability", "markdown_engine": "readability", "include_images": False } ``` ### Bulk Content Migration ```python # Convert multiple pages for migration { "urls": [ "https://old-site.com/page1", "https://old-site.com/page2", "https://old-site.com/page3" ], "max_concurrent": 5, "clean_content": True, "timeout": 45 } ``` ## Security Features - **Input Validation**: URL and content validation - **Size Limits**: Configurable content size limits - **Timeout Protection**: Prevents hanging requests - **User Agent Control**: Configurable user agent strings - **Redirect Limits**: Prevents redirect loops - **Content Type Validation**: Verifies expected content types ## Performance Optimizations - **Concurrent Processing**: Async HTTP with connection pooling - **Streaming Downloads**: Memory-efficient content retrieval - **Lazy Loading**: Load engines only when needed - **Caching**: HTTP response caching where appropriate - **Batch Processing**: Efficient multi-URL processing ## Engine Selection Guide - **News/Blog Articles**: Use `readability` engine - **Technical Documentation**: Use `html2text` engine - **Simple Web Pages**: Use `markdownify` engine - **Complex Layouts**: Use `beautifulsoup` engine - **No Dependencies**: Use `basic` engine ## Limitations - **JavaScript Content**: Does not execute JavaScript (static content only) - **Authentication**: No built-in authentication support - **Rate Limiting**: Implements basic rate limiting only - **Image Processing**: Images are referenced, not embedded - **Large Files**: Size limits prevent processing very large documents