qhjqhj00 surgery-mapping-eval
Neural Network Surgery with Sets — Raiman et al. (2019) (arXiv:1912.06719, 2019)
What this evaluates
Evaluates the correctness and efficiency of gradient-based versus boolean logic-based methods for identifying feature-parameter interactions and transferring trained weights when new features are added to a reinforcement learning model. It measures how well each mapping technique preserves model performance and computational speed during architectural surgery.
Datasets
- OpenAI Five (OG model) — total ?; splits: test (-1)
Metrics
Interactions Found(primary) — range: percent- Percentage of total feature-parameter interactions detected between the original and modified model versions.
Params Transferred— range: percent- Percentage of the original model’s parameters successfully transferred to the new model architecture under specific initialization and zero-gradient handling.
Time (secs)— range: seconds- Wall-clock time required to compute the feature-parameter interaction map using the specified mapping technique.
Input / output format
Input: Original model parameters and architecture, modified model architecture with added features (12 per-hero features), and forward pass data to compute gradients or boolean logic traces.
Output: A computed feature-parameter interaction map, a set of transferred parameters, and the computation time in seconds.
Scoring recipe
def evaluate_surgery(old_model, new_model, features, init_method, zero_grad_handling):
interactions_detected = count_interactions(old_model, new_model, features, method)
interactions_total = total_possible_interactions
interactions_pct = (interactions_detected / interactions_total) * 100
params_transferred = count_transferred_weights(old_model.params, new_model.params, init_method, zero_grad_handling)
params_total = len(old_model.params)
params_pct = (params_transferred / params_total) * 100
time_taken = measure_wall_clock_time(compute_mapping)
return interactions_pct, params_pct, time_taken
Common pitfalls
- Random initialization alone masks interactions (49.42%) and prevents optimal transfer; proper initialization (random positive) and zero-gradient function replacement are required for 100% detection.
- Boolean logic mapping is agnostic to initialization and zero-gradient functions, whereas gradient mapping requires specific handling to avoid masking interactions.
- Time measurements are hardware-dependent (reported on a 2.9 GHz Intel Core i7) and should not be directly compared across different machines without normalization.
Evidence (verbatim from paper)
In Table 3 we report the percentage of total feature-parameter interactions detected, as well as the percentage of the original model’s parameters that can be transferred to the new model under different initializations and zero-gradient function replacements. We find that random positive initialization and zero-gradient function replacement achieves the highest number of transferred parameters, and detects all feature-parameter interactions.
Citation
@misc{raiman2019neural,
title={Neural Network Surgery with Sets},
author={Raiman et al. (2019)},
year={2019},
note={arXiv:1912.06719}
}
- arXiv: 1912.06719