Ziyang Ma1 Renjie Liao2 Xin Tao2 Li Xu2 Jiaya Jia2 Enhua Wu1,3
1University of Chinese Academy of Sciences & State Key Lab. of Computer Science, Inst. of Software, CAS
2The Chinese Univeristy of Hong Kong 3FST, University of Macau
 |
Fig. 1 Multi-frame super-resolution (SR) results on a real video sequence. Green box: Input frames (150 × 120) directly cropped from a video captured by an iPhone. Three clearest frames are shown. Motion blur and compression artifacts are present. (a) Result of single image deblurring [1]. (b) Result of video deblurring [2]. (c) Result of video upsampling [3]. (d) MFSR result [4]. (e) Our result (×3). |
Abstract
Ubiquitous motion blur easily fails multi-frame superresolution (MFSR). Our method proposed in this paper tackles this issue by optimally searching least blurred pixels in MFSR. An EM framework is proposed to guide residual blur estimation and high-resolution image reconstruction. To suppress noise, we employ a family of sparse penalties as natural image priors, along with an effective solver. Theoretical analysis is performed on how and when our method works. The relationship between estimation errors of motion blur and the quality of input images is discussed. Our method produces sharp and higher-resolution results given input of challenging low-resolution noisy and blurred sequences.
Downloads
 |
"Handling Motion Blur in Multi-Frame Super-Resolution" Ziyang Ma, Renjie Liao, Xin Tao, Li Xu, Jiaya Jia, Enhua Wu IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015 
[Paper]
[Supplementary Material]  [Code v0.2 (MATLAB p-code, 2016-01-01)]
 [Data and Results (2015-07-29)] |
Experiments
 |
 |
 |
 |
 |
 |
| (a) Bicubic ×4 | (b) Results with high-res flow | (c) Results with interpolated low-res flow |
Fig. 2 Using TV-L1 based optical flow on the low-res grid yields reasonable results. (a) One input frame with bicubic ×4. (b) Results of using high-res flow [4]. (c) Results of using the interpolated low-res TV-L1 flow [5].
 |
 |
 |
 |
 |
 |
 |
 |
| (a) Selected input frames | (b) Deblur + SR | (c) Results w/o sharpness mask | (d) Our final results |
Fig. 3 More results and comparison (×3). (a) Four input frames from each sequence. (b) Results of multi-image deblurring [6] followed by super-resolution [4]. (c) Results without sharpness mask (β = ∞). (d) Our results with the sharpness masks (β = 1.4).
 |
 |
| (a) Selected input frames | (b) Our results |
Fig. 4 Our method is robust to noise. (a) A few relative sharp input frames. (b) Our results (×3).
Results
All results of our method are available here (coming soon).
 |
|
| (a) Selected input frames | |
 |
 |
| (b) Result of [7] | (c) Our result |
Fig. 5 More results and comparison (×3). (a) Four input frames from a sequence. (b) Multi-shot image result [7]. (c) Our result.
 |
 |
 |
 |
| (a) Selected input frames (with zoom-in) | (b) Our results |
Fig. 6 More natural video results. (a) Sample input frames. (b) Our results estimated using 31 low-res frames each.
Reference
[1] L. Xu, S. Zheng, and J. Jia. Unnatural l0 sparse representation for natural image deblurring. In CVPR, pages 1107–1114, 2013.
[2] S. Cho, J. Wang, and S. Lee. Video deblurring for hand-held cameras using patch-based synthesis. ACM Transactions on Graphics (TOG), 31(4):64, 2012.
[3] Q. Shan, Z. Li, J. Jia, and C.-K. Tang. Fast image/video upsampling. ACM Transactions on Graphics (TOG), 27(5):153, 2008.
[4] C. Liu and D. Sun. A bayesian approach to adaptive video super resolution. In CVPR, pages 209–216, 2011.
[5] T. Brox, A. Bruhn, N. Papenberg, and J. Weickert. High accuracy optical flow estimation based on a theory for warping. In ECCV, pages 25–36, 2004.
[6] X. Zhu, F. ˇSroubek, and P. Milanfar. Deconvolving psfs for a better motion deblurring using multiple images. In ECCV, pages 636–647, 2012.
[7] H. Zhang and L. Carin. Multi-shot imaging: Joint alignment, deblurring, and resolution-enhancement. In CVPR, pages 2925–2932, 2014.