mflux testing
This repo uses pytest with image-producing tests. Always preserve outputs for inspection and never update reference images unless explicitly asked.
Overview
The mflux testing framework provides comprehensive test coverage for model validation workflows. By leveraging pytest with image-producing capabilities, teams can maintain high-quality standards across development cycles. The system emphasizes preservation of test outputs to enable thorough inspection and comparison of results over time.
Usage
Test Execution Commands
/test-fast(fast tests, no image generation)/test-slow(slow tests, image generation)/test(full suite)
Configuration Guidelines
Always maintain MFLUX_PRESERVE_TEST_OUTPUT=1 during test runs. This environment variable is already integrated into the available commands and ensures consistent output handling throughout the testing lifecycle.
Guidelines
Operational Mode
- Prefer the existing Cursor commands for streamlined test execution.
- Always keep
MFLUX_PRESERVE_TEST_OUTPUT=1on test runs (already built into the commands). - If tests fail:
- Summarize the failing test names and the key assertion output.
- Point to any generated images/artifacts on disk for manual review.
- Do not regenerate/replace reference ("golden") images unless the user explicitly requests it.
Manual validation (config resolution + local model paths)
Use when a change touches model config resolution, mflux-save, or the model's generate CLI, or when a PR fixes local model-path handling for the model under investigation. Refer to the mflux-cli skill to find the correct generate command for the model you are testing.
- Run a local-path quantize/save:
- Use the
mflux-cliskill to look up the correct command and flags. - Verify CLI usage with the command's
--helpbefore running it. - Save to a known location (e.g., Desktop) to make follow-up steps explicit.
- Use the
- Run generation from the saved model using the correct model-specific generate CLI:
- Use the
mflux-cliskill to find the generate command and required flags. - Verify CLI usage with the command's
--helpbefore running it.
- Use the
- If the model has multiple size variants, repeat the above for each variant to confirm the correct overrides are applied.
- Do not commit output artifacts; delete or leave them untracked.
Examples
Test Output Preservation
When running the full test suite, the system generates comprehensive output artifacts that should be preserved for future reference. These outputs include test execution logs, image comparisons, and configuration snapshots that support ongoing quality assurance efforts.
Golden Image Management
Reference images serve as the foundation for visual validation. The system maintains these golden images without automatic updates, ensuring consistency across test cycles. Manual intervention is required when modifications to the testing framework necessitate image regeneration.
Model Configuration Workflow
The testing workflow integrates model configuration resolution with local path management. This approach ensures that model-specific parameters are properly aligned with the testing infrastructure, enabling accurate validation of model performance across different deployment scenarios.