Video repairing

Jiaya Jia

Tai-Pang Wu

Yu-Wing Tai

Chi-Keung Tang

Abstract — This paper presents a complete system capable of synthesizing a large number of pixels that are missing due to occlusion or damage in an uncalibrated input video. These missing pixels may correspond to the static background or cyclicmotions of the captured scene. Our system employs user-assisted video layer segmentation, while themain processing in video repair is fully automatic. The input video is first decomposed into the color and illumination videos. The necessary temporal consistency is maintained by tensor voting in the spatio-temporal domain. Missing colors and illumination of the background are synthesized by applying image repairing. Finally, the occluded motions are inferred by spatio-temporal alignment of collected samples at multiple scales. We experimented our system with some difficult examples with variable illumination, where the capturing camera can be stationary or in motion.

Publications:

Video Repairing: Inference of Foreground and Background Under Severe Occlusion.
Jiaya Jia, Tai-Pang Wu, Yu-Wing Tai, and Chi-Keung Tang, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June-July 2004, Pages I: 364-371.
Video Repairing Under Variable Illumination Using Cyclic Motions.
Jiaya Jia, Yu-Wing Tai, Tai-Pang Wu and Chi-Keung Tang, IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), , vol. 28, no. 5, 2006.

BibTex:

@inproceedings{Jia2004vr,
    author = {Jiaya Jia and Tai-Pang Wu and Yu-Wing Tai and Chi-Keung Tang},
    title = {Video Repairing: Inference of Foreground and Background Under Severe Occlusion},
    booktitle = {CVPR},
    year = {2004},
    pages = {I: 364-371}
}

@article{jia_pami_vr,
    author = {Jiaya Jia and Yu-Wing Tai and Tai-Pang Wu and Chi-Keung Tang},
    title = {Video Repairing Under Variable Illumination Using Cyclic Motions},
    journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
    volume = {28},
    year = {2006},
    number = {5},
    pages = {832-839}
}

In this project, we analyze the cyclic motions in videos and propose the approach to repair either occluded static background or motions.

Repairing static background with camera motions:

　

Input video Result video with the bridge removed and a keyed-in car

Repairing cyclic motions:

Input video	Occluder removed	Repaired video

Complex scene containing occluded multiple-layer cyclic motions and background:

It is noted that the shadow is removed in the background completion in this example.

Input video	Repaired video with completed motions and background

Repairing video with varied luminance:

Input video	Repaired video with varied luminance

Repairing video with varied luminance and moving camera:

Input video	Repaired video with varied luminance and moving camera

Input video	Result video with the bridge removed and a keyed-in car