qhjqhj00 dr-aid-eval
Dr.Aid: Supporting Data-governance Rule Compliance for Decentralized Collaboration in an Automated Way — Zhao et al. (2021) (arXiv:2110.01056, 2021)
What this evaluates
Evaluates an automated framework's ability to encode natural-language data-governance policies into a formal model, extract data-flow graphs from provenance traces, and correctly trigger compliance obligations across decentralized scientific workflows.
Datasets
- Cyclone tracking workflow — total ?; splits: case_study (-1)
- MT3D (Moment Tensor in 3D) workflow — total ?; splits: case_study (-1)
- Real-world data-governance policies — total ?; splits: case_study (-1)
Metrics
actioning rules(primary) — range: count- Count of rules explicitly describing required actions, behaviors, or obligations that must be encoded and tracked during workflow execution.
encoded rules— range: count- Count of actioning rules successfully translated into the formal model's user notation, measured against the original policy to assess encoding coverage.
obligation activation correctness— range: boolean/qualitative- Qualitative and quantitative validation that compliance obligations are triggered at the correct workflow stages (e.g., import, publish) based on provenance traces and expert review.
Input / output format
Input: Natural-language data-governance policies and provenance traces (CWLProv or S-Prov schemas) extracted from scientific workflows via SPARQL endpoints.
Output: Identified activated obligations (represented as Python dictionaries), derived data rules, and compliance reasoning results mapped to the intermediate data-flow graph representation.
Scoring recipe
def evaluate_dr_aid(policy_text, provenance_traces):
encoded_rules = encode_natural_language_policy(policy_text)
provenance_graph = load_provenance(provenance_traces)
activated_obligations = reason_over_graph(provenance_graph, encoded_rules)
# Validation against ground truth and expert opinion
expert_validation = consult_experts(encoded_rules, activated_obligations)
encoding_accuracy = count_correctly_encoded(encoded_rules) / count_original_actioning_rules
trigger_correctness = check_obligation_triggers(activated_obligations, expected_stages)
return {
'actioning_rules_count': count_actioning_rules(policy_text),
'encoded_rules_count': len(encoded_rules),
'expert_validation': expert_validation,
'trigger_correctness': trigger_correctness
}
Common pitfalls
- Not every sentence in natural-language policies can be modeled; contextual or implicit rules are intentionally excluded from encoding.
- Parallel workflow traces may produce seemingly duplicated obligations, which is semantically expected but can be mistaken for system errors.
- Expert validation is required to confirm completeness and meaning, as automated metrics alone cannot capture policy nuance or workflow context.
Evidence (verbatim from paper)
The most important one is the procedure to convert from natural-language policies to the formal representation. We have standardised the procedure for this: (1) Identify and obtain nested rules if any; (2) Remove unnecessary information from the rules; (3) Identify actioning rules, in particular obligations; (4) Find the terms in the rules that identify the data or critical properties of data that need to be carried with data, as attributes; (5) Identify implied rules; (6) Write in the user notation where possible... Therefore, following this standardized encoding procedure, we measure its effect using the following information: (1) The total number of sentences in the original natural-language (English) policy; (2) The total number of rules in the original policy; (3) The total number of actioning rules; (4) The total number of implicit rules; (5) The total number of encoded rules.
Citation
@misc{zhao2021draid,
title={Dr.Aid: Supporting Data-governance Rule Compliance for Decentralized Collaboration in an Automated Way},
author={Zhao et al. (2021)},
year={2021},
note={arXiv:2110.01056}
}
- arXiv: 2110.01056