Latcoder Layout Aware Code Generation

LaTCoder: Layout-as-Thought for Design-to-Code

LaTCoder addresses layout preservation in webpage design-to-code generation through Layout-as-Thought (LaT): decompose visual designs into geometric blocks, reason about layout first, then generate code for each block. This divide-and-conquer approach avoids MLLM weaknesses in spatial reasoning and numerical understanding, achieving >60% preference over baselines.

Core Concept

Converting webpage designs to code requires understanding spatial layout—where elements are positioned and sized. MLLMs struggle with: (1) "factual interpretation" of visual coordinates, and (2) "numerical reasoning" for dimensions. LaTCoder sidesteps this by anchoring code generation to explicit spatial coordinates: detect layout blocks in the design, treat each block as an independent reasoning step, and assemble via layout-aware assembly strategies.

Architecture Overview

• Layout-Aware Division: Detect horizontal/vertical dividing lines, extract distinct layout blocks
• Block-Wise Code Synthesis: Generate HTML/CSS per block via Chain-of-Thought
• Layout-Preserved Assembly: Two strategies (absolute positioning or MLLM-based assembly)
• Verifier: MAE + CLIP similarity for selecting best assembly strategy

Implementation Steps

Step 1: Detect Layout Structure

import cv2
import numpy as np
from PIL import Image
from typing import List, Tuple, Dict

class LayoutDetector:
    """Detect rectangular layout blocks in webpage designs."""

    def __init__(self):
        self.dividing_line_threshold = 200  # Pixel threshold for solid lines

    def detect_dividing_lines(self, image: Image.Image) -> Tuple[List[int], List[int]]:
        """
        Detect horizontal and vertical dividing lines.
        Lines are solid-colored regions where layout blocks meet.
        """
        img_array = np.array(image)
        gray = cv2.cvtColor(img_array, cv2.COLOR_RGB2GRAY)

        # Detect horizontal lines (constant rows with uniform color)
        horizontal_lines = []
        for row in range(1, gray.shape[0] - 1):
            row_color = gray[row, :]
            # Check if row is mostly one color (solid line)
            if np.std(row_color) < 30:  # Low variance = solid
                horizontal_lines.append(row)

        # Detect vertical lines (constant columns with uniform color)
        vertical_lines = []
        for col in range(1, gray.shape[1] - 1):
            col_color = gray[:, col]
            if np.std(col_color) < 30:
                vertical_lines.append(col)

        # Consolidate adjacent lines into single dividers
        horizontal = self._consolidate_lines(horizontal_lines, threshold=10)
        vertical = self._consolidate_lines(vertical_lines, threshold=10)

        return horizontal, vertical

    def _consolidate_lines(self, lines: List[int], threshold: int = 10) -> List[int]:
        """Merge nearby lines into single dividers."""
        if not lines:
            return []

        consolidated = [lines[0]]
        for line in lines[1:]:
            if line - consolidated[-1] > threshold:
                consolidated.append(line)

        return consolidated

    def extract_layout_blocks(self, image: Image.Image, h_dividers: List[int],
                            v_dividers: List[int]) -> List[Dict]:
        """
        Extract rectangular blocks using dividing lines.
        """
        # Add image boundaries
        h_dividers = [0] + sorted(h_dividers) + [image.height]
        v_dividers = [0] + sorted(v_dividers) + [image.width]

        blocks = []

        for i in range(len(h_dividers) - 1):
            for j in range(len(v_dividers) - 1):
                top = h_dividers[i]
                bottom = h_dividers[i + 1]
                left = v_dividers[j]
                right = v_dividers[j + 1]

                # Skip very small blocks (likely artifacts)
                if (bottom - top) < 20 or (right - left) < 20:
                    continue

                # Crop block image
                block_image = image.crop((left, top, right, bottom))

                block = {
                    'index': len(blocks),
                    'bbox': (left, top, right, bottom),
                    'width': right - left,
                    'height': bottom - top,
                    'image': block_image,
                }

                blocks.append(block)

        return blocks

Step 2: Implement Chain-of-Thought Code Generation per Block

class BlockCodeGenerator:
    """Generate HTML/CSS for individual layout blocks."""

    def __init__(self, mlm_model):
        self.mlm = mlm_model  # Multimodal LLM

    def generate_block_code(self, block: Dict, block_position: int,
                          previous_blocks_context: str = "") -> str:
        """
        Generate code for single block with Layout-as-Thought.
        CoT prompts emphasize layout fidelity before content.
        """
        block_image = block['image']
        width = block['width']
        height = block['height']
        bbox = block['bbox']

        # Layout-as-Thought prompt: emphasize structure first
        cot_prompt = f"""Webpage Design Block #{block_position}:
Layout dimensions: {width}px × {height}px
Position in page: top-left ({bbox[0]}, {bbox[1]})

Let's think step by step about the layout:
1. What is the spatial structure of this block? (grid, flex, absolute positioning)
2. What are the key layout dimensions? (widths, heights, gaps)
3. What visual content should be in this block? (text, images, buttons)
4. How should padding/margins be set for the layout?

Generate HTML and CSS for this block that preserves the visual layout.
Include all necessary styling for positioning and spacing.

Code:
```html
[HTML/CSS code for this block]
```"""

        # Use vision-language model with CoT
        code = self.mlm.generate(
            prompt=cot_prompt,
            image=block_image,
            max_tokens=500
        )

        return code.strip()

    def generate_blocks_with_context(self, blocks: List[Dict]) -> Dict[int, str]:
        """Generate code for all blocks with inter-block context."""
        block_codes = {}

        for i, block in enumerate(blocks):
            # Build context from previous blocks
            previous_context = "\n".join(
                f"Block {j}: {block_codes[j][:100]}..."
                for j in range(max(0, i - 2), i)
            )

            code = self.generate_block_code(block, i, previous_context)
            block_codes[i] = code

        return block_codes

Step 3: Implement Layout-Preserved Assembly

class AssemblyStrategy:
    """Combine blocks into full webpage while preserving layout."""

    def absolute_positioning_assembly(self, blocks: List[Dict],
                                     block_codes: Dict[int, str]) -> str:
        """
        Assembly via absolute positioning: use bounding boxes for placement.
        Each block positioned using its extracted coordinates.
        """
        html_parts = ['<!DOCTYPE html>\n<html>\n<head>\n<style>\n']

        # CSS: position all blocks absolutely
        html_parts.append('body { position: relative; }\n')

        for block in blocks:
            left, top, right, bottom = block['bbox']
            width = right - left
            height = bottom - top

            block_css = f"""
.block_{block['index']} {{
    position: absolute;
    left: {left}px;
    top: {top}px;
    width: {width}px;
    height: {height}px;
}}
"""
            html_parts.append(block_css)

        html_parts.append('</style>\n</head>\n<body>\n')

        # HTML: insert blocks with positioning class
        for block in blocks:
            block_code = block_codes[block['index']]

            # Extract inner HTML from block code
            inner_html = self._extract_inner_html(block_code)

            html_parts.append(f'<div class="block_{block["index"]}">\n')
            html_parts.append(inner_html)
            html_parts.append('</div>\n')

        html_parts.append('</body>\n</html>')

        return ''.join(html_parts)

    def mllm_assembly(self, blocks: List[Dict], block_codes: Dict[int, str],
                     mlm_model) -> str:
        """
        Assembly via MLLM reasoning: let LLM compose blocks coherently.
        Can handle relative positioning, flex layouts, etc.
        """
        assembly_prompt = f"""You have generated code for {len(blocks)} layout blocks:

"""
        for i, block in enumerate(blocks):
            assembly_prompt += f"Block {i} ({block['width']}×{block['height']}):\n"
            assembly_prompt += f"{block_codes[i]}\n\n"

        assembly_prompt += """Now assemble these blocks into a complete HTML page.
Preserve the spatial layout from the original design.
Use appropriate CSS (flexbox, grid, or positioning) to arrange blocks.
Combine related styles and eliminate duplication.

Complete HTML:"""

        html = mlm_model.generate(assembly_prompt, max_tokens=2000)
        return html

    def _extract_inner_html(self, block_code: str) -> str:
        """Extract inner HTML from generated block code."""
        # Simple: remove outer HTML tags
        lines = block_code.split('\n')
        inner_lines = [l for l in lines if not l.strip().startswith('<html')
                       and not l.strip().startswith('<body')
                       and not l.strip().startswith('<head')]
        return '\n'.join(inner_lines)

Step 4: Implement Verifier

class AssemblyVerifier:
    """Select best assembly strategy using MAE and CLIP similarity."""

    def __init__(self):
        self.clip_model = load_clip_model()
        self.mae_model = load_mae_model()  # Masked Autoencoder for reconstruction

    def compute_mae_loss(self, original_design: Image.Image,
                       generated_html: str) -> float:
        """
        Render generated HTML, compare pixel-level with original.
        MAE = Mean Absolute Error between designs.
        """
        rendered_image = render_html_to_image(generated_html)

        # Resize to same dimensions
        original_array = np.array(original_design)
        rendered_array = np.array(rendered_image.resize(original_design.size))

        # Compute MAE
        mae = np.mean(np.abs(original_array.astype(float) - rendered_array.astype(float)))

        return mae

    def compute_clip_similarity(self, original_design: Image.Image,
                              generated_html: str) -> float:
        """
        Semantic similarity via CLIP: encode both visually.
        High similarity = good content preservation.
        """
        rendered_image = render_html_to_image(generated_html)

        # Encode visually
        original_features = self.clip_model.encode_image(original_design)
        rendered_features = self.clip_model.encode_image(rendered_image)

        # Cosine similarity
        similarity = torch.nn.functional.cosine_similarity(
            original_features.unsqueeze(0),
            rendered_features.unsqueeze(0)
        ).item()

        return (similarity + 1) / 2  # Normalize to [0, 1]

    def select_best_assembly(self, original_design: Image.Image,
                           assemblies: Dict[str, str]) -> Tuple[str, float]:
        """
        Evaluate both assembly strategies, return best.
        Score = 0.6 * mae_quality + 0.4 * clip_similarity
        """
        scores = {}

        for strategy_name, html in assemblies.items():
            mae = self.compute_mae_loss(original_design, html)
            mae_quality = 1.0 / (1.0 + mae)  # Convert to quality metric

            clip_sim = self.compute_clip_similarity(original_design, html)

            # Combined score
            score = 0.6 * mae_quality + 0.4 * clip_sim
            scores[strategy_name] = score

        best_strategy = max(scores, key=scores.get)
        return assemblies[best_strategy], scores[best_strategy]

Step 5: End-to-End Pipeline

def latcoder_design_to_code(design_image: Image.Image,
                           mlm_model) -> str:
    """
    Complete pipeline: detect layout → generate blocks → assemble.
    """
    # Step 1: Detect layout structure
    detector = LayoutDetector()
    h_dividers, v_dividers = detector.detect_dividing_lines(design_image)
    blocks = detector.extract_layout_blocks(design_image, h_dividers, v_dividers)

    print(f"Detected {len(blocks)} layout blocks")

    # Step 2: Generate code per block
    code_gen = BlockCodeGenerator(mlm_model)
    block_codes = code_gen.generate_blocks_with_context(blocks)

    # Step 3: Try both assembly strategies
    assembler = AssemblyStrategy()

    absolute_html = assembler.absolute_positioning_assembly(blocks, block_codes)
    mllm_html = assembler.mllm_assembly(blocks, block_codes, mlm_model)

    # Step 4: Verify and select
    verifier = AssemblyVerifier()
    final_html, quality_score = verifier.select_best_assembly(
        design_image,
        {'absolute_positioning': absolute_html, 'mllm': mllm_html}
    )

    print(f"Selected assembly strategy with quality: {quality_score:.3f}")

    return final_html

Practical Guidance

When to Use:

• Webpage design to code conversion
• Scenarios where spatial layout is critical
• Visual design systems with consistent structure
• Conversion of mockups/Figma designs to HTML/CSS

When NOT to Use:

• Complex interactive designs (animations, transitions)
• Designs with overlapping elements
• Real-time rendering requirements
• Scenarios where exact pixel-perfect layout is unnecessary

Hyperparameters:

Reference

Paper: LaTCoder: Converting Webpage Design to Code with Layout-as-Thought (2508.03560)

• 60% human preference over baselines

• Layout-as-Thought anchors reasoning to spatial coordinates
• Two assembly strategies: absolute positioning vs. MLLM reasoning
• Verifier uses MAE and CLIP for quality assessment