Depth-to-RGB: Repurposing a Frozen Depth Estimator for Geometry-Guided Compositing

Key points

• Reference-based object compositing inserts or replaces an object using a background image, a reference image, and a 2D compositing mask.1
• It learns reference-conditioned corrections to a frozen depth estimator using encoder features of paired completed scenes as targets.1
• The unchanged decoder maps the corrected representation to the intended scene's depth, which a separately trained renderer holds fixed during RGB synthesis.1
• We also introduce AnyInsertion++ with paired in-distribution and category-disjoint splits to evaluate generalization beyond compositing training categories.1

Sources (1)

• [1]Depth-to-RGB: Repurposing a Frozen Depth Estimator for Geometry-Guided Compositing
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 6, 09:19 PM

  Our Depth-to-RGB (D2R) framework predicts composite depth for a scene not yet observed in the RGB inputs.

  Reference-based object compositing inserts or replaces an object using a background image, a reference image, and a 2D compositing mask.

Extractive summary: sentences quoted from the sources.