## Article
Object removal has become a staple of modern generative editing, but evaluating *how well* an object is removed has proven surprisingly difficult. Traditional image-quality metrics such as PSNR, SSIM, and LPIPS were originally designed for generic image restoration and often misrank edited outputs—sometimes rewarding blatant copy-paste or even rewarding degraded images that happen to be closer pixel-wise to the original. More recent benchmarks like ROSE, RORD, and DAVIS exposed weaknesses in existing *no-reference* and *full-reference* methods, where metrics either fail to degrade when blur is increased or falsely penalize legitimate hallucinations.
Against this backdrop, the work “PROVE: Perceptual RemOVal cohErence” (accepted at ACM MM 2026), originating from MiLM Plus at Xiaomi Inc., introduces a principled evaluation framework designed specifically for object removal. Rather than relying on a single reference image, PROVE evaluates *perceptual coherence* of the edited region by comparing its local feature distribution to nearby background features. The approach is built on two complementary metrics—RC-S for spatial consistency and RC-T for temporal consistency—alongside a large-scale real-world benchmark called PROVE-Bench.
—
## How PROVE Works
At the core of PROVE is the insight that object removal is an **ill-posed, one-to-many problem**: many different restorations can be perceptually plausible for the same hole. Because there is no unique ground truth, any metric relying on direct pixel-wise comparison is fundamentally misaligned. PROVE sidesteps this issue by evaluating *local distribution matching* in a pre-trained vision model, without requiring a reference video.
### RC-S: Spatial Consistency
RC-S focuses on the edited region within a single frame. Each object mask is post-processed by:
– Connected-component analysis to handle multiple disjoint objects.
– Expanding each mask by one third of its side length to capture surrounding context.
– Extracting features with DINOv2.
– Applying a sliding window over the masked area and computing squared Maximum Mean Discrepancy (MMD) between masked features and local background features using a Gaussian RBF kernel.
– Averaging scores across components and across windows to produce a single spatial coherence score.
Because the metric operates locally, it is much less sensitive to copy-paste artifacts and better aligned with perceptual quality.
### RC-T: Temporal Consistency
For videos, temporal flicker and inconsistency are critical failure modes. RC-T addresses this by:
– Cropping two consecutive frames using the *union* of their masks to preserve alignment.
– Focusing evaluation only on the *intersection* of the masks—i.e., the region that is restored in both frames.
– Again using MMD in DINOv2 feature space to measure whether the restored region remains stable across time.
This design ensures that RC-T responds to temporal incoherence while remaining robust to unrelated background motion.
—
## Failure Modes of Prior Metrics
The paper documents three major failure modes of existing metrics:
1. **Full-reference bias**
PSNR, SSIM, and LPIPS assume one correct reconstruction per mask. As a result, they often reward simple copy-paste or slight blur, while penalizing valid but variant restorations.
2. **No-reference blind spots**
Methods like CFD use SAM-based hallucination detection and can incorrectly flag legitimate completions as hallucinations, while failing to degrade when blur is increased.
3. **Temporal insensitivity**
Existing temporal metrics average over entire frames, diluting flicker signals that affect only a small edited region.
These issues collectively motivate the need for a removal-specific evaluation framework.
—
## PROVE-Bench and Results
To support its metrics, the authors introduce PROVE-Bench, consisting of:
– **PROVE-M**: 80 paired videos, with clean or lightly blurred inputs and target-free footage captured within two minutes. Masks are refined frame-by-frame using SAM3, and motion is augmented with Ken Burns-style camera effects.
– **PROVE-H**: 100 harder videos featuring crowds, water, fire, textured terrain, multi-puddle reflections, and fast motion, using raw SAM3 masks without refinement.
Against human rankings (Borda-count aggregation over 20 participants):
– RC-S achieves **0.59 average Kendall’s τ** and **0.66 Spearman’s ρ**.
– ReMOVE scores **0.26 / 0.29**, and CFD scores **0.16 / 0.18**.
– RC-S ranks first on five of six benchmarks and is also **13.7× faster than CFD**.
Ablation studies confirm the contributions of DINOv2, sliding-window MMD, and spatial averaging, showing consistent gains from each design choice.
On the public leaderboard, the best systems currently are:
– **SVOR (1.3B)** for RC-S with a score of **0.5197**.
– **EffectErase** for RC-T with a score of **0.2525**.
—
## Summary of Key Insights
– **Metric design must match the task**: Pixelwise fidelity metrics are poorly aligned with perceptual quality in object removal.
– **Local distribution matching works**: MMD in DINOv2 features, applied within the mask and its surroundings, correlates strongly with human judgment.
– **Temporal evaluation is crucial**: Ignoring temporal consistency leads to missing flicker and instability, especially in dynamic scenes.
– **No ground truth is not an obstacle**: By focusing on *relative* consistency rather than absolute reconstruction, PROVE enables meaningful comparisons even in target-free settings.
—
## FAQ
**What problem does PROVE solve?**
PROVE addresses the mismatch between object removal editing and existing evaluation metrics, which often misrank outputs because there is no single ground truth. It provides perception-aligned metrics that evaluate how coherent the edited region is, both spatially and temporally, without requiring a reference image or video.
**What are RC-S and RC-T?**
RC-S (Removal Consistency — Spatial) measures how coherent the restored region within a single frame is compared to local background features using MMD in DINOv2 feature space. RC-T (Removal Consistency — Temporal) extends this to videos by evaluating stability of the restored region across adjacent frames using a union–intersection crop strategy.
**Does PROVE require a ground truth video?**
No. PROVE is a *no-reference* evaluation framework. It only requires the edited video, the object mask, and a pre-trained DINOv2 model.
**What is PROVE-Bench?**
PROVE-Bench is a real-world video benchmark with two tiers:
– PROVE-M: Easier, tripod-shot videos with SAM3-derived masks and motion augmentation.
– PROVE-H: Harder, real-world videos with complex backgrounds, motion, and unrefined masks.
**How does PROVE compare to PSNR, SSIM, LPIPS, ReMOVE, and CFD?**
Across human evaluations, RC-S significantly outperforms traditional and recent no-reference metrics, achieving much higher correlation with human rankings while being computationally cheaper than methods like CFD.
**Can RC-S be used for training or just evaluation?**
PROVE is designed primarily as an evaluation harness. It can be used in model comparisons, CI gates, reward model shaping, and data filtering, but it is not itself an editing model.
**What are the main limitations?**
– Not intended for real-time on-device use.
– Can miss large shadows or reflections that extend beyond the evaluation region.
– Requires accurate object masks; performance degrades with poor mask quality.
—
## Conclusion
The PROVE framework represents a significant step forward in evaluating object removal by aligning metric design with the intrinsic challenges of the task. By replacing global pixelwise comparisons with local, distribution-based consistency checks in a strong visual feature space, it overcomes many of the biases that plague prior metrics. Supported by a large real-world benchmark and a thorough error analysis, PROVE offers researchers and practitioners a more reliable, human-aligned way to assess and compare object removal systems. As object removal pipelines continue to evolve, frameworks like PROVE will be essential for ensuring that improvements in visual quality are properly measured and not lost in misleading numeric scores.
