Sketch Rnn Eval

sketch-rnn-eval

A Neural Representation of Sketch Drawings — Ha et al. (2017) (arXiv:1704.03477, 2017)

What this evaluates

Evaluates a generative model's ability to produce coherent stroke-based vector sketches through reconstruction, latent space interpolation, and completion of incomplete drawings. It probes how well the model captures conceptual features and organizes them in a continuous latent manifold.

Datasets

• QuickDraw — total ?; splits: test (-1)

Metrics

• LR (primary) — range: other
  • Reconstruction loss, computed as the negative log-likelihood of the input stroke sequence given the latent code and previous strokes. Lower values indicate better reconstruction fidelity.
• LK — range: other
  • KL divergence between the approximate posterior distribution and the Gaussian prior over the latent space. Lower values indicate stronger regularization toward the prior.

Input / output format

Input: A sequence of stroke coordinates/commands representing a human-drawn sketch.

Output: A sequence of stroke coordinates/commands representing a generated, reconstructed, or completed sketch.

Scoring recipe

def compute_metrics(predictions, gold, w_KL=1.0):
    # LR: Reconstruction loss (negative log-likelihood)
    LR = -sum(log_prob(gold[t] | gold[:t], latent) for t in range(len(gold)))
    # LK: KL divergence between encoder posterior and Gaussian prior
    LK = kl_divergence(encoder_posterior, prior)
    return {"LR": LR, "LK": LK}

Common pitfalls

• Qualitative assessments (interpolation, analogies, completion) are subjective and lack quantitative metrics.
• Loss values are highly class-dependent due to varying sketch complexity, making cross-class comparisons misleading.
• Results are reported for multiple w_KL hyperparameters; comparing across settings without specifying w_KL is invalid.

Evidence (verbatim from paper)

The results for test set evaluation on various datasets are displayed in Table 1. We see that the reconstruction loss term $L_{R}$ decreases as we relax the $w_{KL}$ parameter controlling the weight for the KL loss term, and meanwhile the KL loss term $L_{R}$ increases as a result.

Citation

@misc{ha2017sketchrnn,
  title={A Neural Representation of Sketch Drawings},
  author={Ha et al. (2017)},
  year={2017},
  note={arXiv:1704.03477}
}

• arXiv: 1704.03477