Sound Of Water Eval

sound-of-water-eval

The Sound of Water: Inferring Physical Properties from Pouring Liquids — Bagad et al. (2024) (arXiv:2411.11222, 2024)

What this evaluates

Evaluates a model's ability to infer physical properties (air column length, container dimensions, flow rate, fill time, liquid weight) and classify container shapes solely from the acoustic characteristics of pouring liquids, without visual or tactile input.

Datasets

• Sound of Water 50 — total ?; splits: Test set I (-1), Test set II (-1), Test set III (434)
• Wilson et al. [96] dataset — total 136; splits: per-container splits (-1)

Metrics

• Mean Absolute Error (MAE) (primary) — range: other
  • Average of the absolute differences between predicted and ground-truth values: (1/N) Σ|ŷ_i - y_i|. Reported in cm for length/dimensions, ml/s for flow rate, seconds for fill time, and ounces for weight.
• Accuracy — range: [0, 1]
  • Proportion of correctly classified container shapes (cylindrical, semiconical, bottleneck). Reported as per-sample accuracy and mean class accuracy.

Input / output format

Input: Raw audio recordings of liquids being poured into containers.

Output: Continuous scalar predictions for physical properties (e.g., air column length in cm, height in cm, radius in cm, flow rate in ml/s, fill time in s, weight in oz) or discrete class labels for container shape (cylindrical, semiconical, bottleneck).

Scoring recipe

def compute_mae(predictions, targets):
    return np.mean(np.abs(predictions - targets))

def compute_accuracy(predictions, targets):
    return np.mean(predictions == targets)

Common pitfalls

• The pitch detection model is trained exclusively on cylindrical containers, so evaluation on semiconical and bottleneck shapes tests zero-shot generalization rather than in-distribution performance.
• Weight estimation uses linear probing on 0.4s audio snippets without fine-tuning the backbone, which may underperform compared to supervised baselines that fine-tune directly on the target dataset.
• Baseline pitch detectors are compared after the proposed method applies RANSAC line-fitting to wavelength estimates, introducing a post-processing step that is not applied to the baselines.

Evidence (verbatim from paper)

We compare our models with the baselines in estimating l(t) and report the mean absolute error averaged over all time points. ... To report performance, we consider the Test Set III set of our dataset consisting of 434 videos (227 semiconical, 107 bottleneck and 100 cylindrical). These are unseen samples not part of the training and evaluation set used previously to estimate physical properties. We split these randomly in an 80-20 split. On this split, we achieve a per sample accuracy of 90.91% and a mean class accuracy of 92.47%.

Citation

@misc{bagad2024soundofwater,
  title={The Sound of Water: Inferring Physical Properties from Pouring Liquids},
  author={Bagad et al. (2024)},
  year={2024},
  note={arXiv:2411.11222}
}

• arXiv: 2411.11222