Falcon Eval

falcon-eval

Falcon: Accelerating Homomorphically Encrypted Convolutions for Efficient Private Mobile Network Inference — Tianshi Xu et al. (2023) (arXiv:2308.13189, 2023)

What this evaluates

Evaluates the efficiency and accuracy of homomorphically encrypted convolution operations and end-to-end private inference networks. It measures communication overhead, inference latency, and classification accuracy under simulated WAN/LAN bandwidths and varying polynomial degrees.

Datasets

• CIFAR-10 — total ?; splits: test (-1)
• CIFAR-100 — total ?; splits: test (-1)
• Tiny ImageNet — total ?; splits: test (-1)

Metrics

• latency (primary) — range: other
  • End-to-end inference time measured in seconds from input encryption to output decryption and classification.
• communication — range: other
  • Total network transfer volume between the two parties during homomorphic computation, measured in MB or GB.
• top-1 accuracy — range: percent
  • Percentage of correctly classified images out of the total test set after decryption.

Input / output format

Input: Encrypted input feature maps and homomorphically encrypted model weights for depthwise/group convolutions or full networks (MobileNetV2, EfficientNet-lite).

Output: Execution latency in seconds, communication volume in MB/GB, and top-1 classification accuracy percentage.

Scoring recipe

def evaluate(predictions, gold, config):
    latency = time.time() - start_time
    communication = sum(len(msg) for msg in network_transfers)
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    accuracy = (correct / len(gold)) * 100
    return {'latency_s': latency, 'communication_mb': communication / 1e6, 'accuracy_pct': accuracy}

Common pitfalls

• Latency and communication metrics are highly dependent on the simulated bandwidth (9 MBps vs 384 MBps) and the polynomial degree N, which must be reported alongside results.
• End-to-end comparisons with the baseline 'Iron' are unavailable due to missing open-source code, so only microbenchmark results can be compared for that method.
• Accuracy improvements are evaluated under iso-communication constraints, meaning models are compared at similar communication costs rather than fixed network settings.

Evidence (verbatim from paper)

As shown in Table IV, compared to Cheetah, Falcon reduces the communication by 1.31× and the latency by more than 1.2× on the Cifar10 and Cifar100 datasets. On the Tiny Imagenet dataset, the advantage of Falcon is greater, with a communication reduction of 1.44∼1.48× and a latency reduction of 1.35∼1.36×.

Citation

@misc{xu2023falcon,
  title={Falcon: Accelerating Homomorphically Encrypted Convolutions for Efficient Private Mobile Network Inference},
  author={Tianshi Xu et al. (2023)},
  year={2023},
  note={arXiv:2308.13189}
}

• arXiv: 2308.13189