Research Interest

Computer vision, graphics, and computational photography

  • Image deblurring
  • Alpha matting
  • Optical flow estimation and stereo matching
  • Image/video manipulation and enhancement
  • Camera tracking and video depth/layer estimation

 

Codes, Executables, and Software

 

Research Highlight

Abstract: A new image editing method, particularly effective for sharpening major edges by increasing the steepness of transitions while eliminating a manageable degree of low-amplitude structures, is proposed. The seemingly contradictive effect is achieved in an unconventional optimization framework making use of L0 gradient minimization, which can globally control how many non-zero gradients are resulted to approximate prominent structures in a structure-sparsity-management manner. It finds many applications and is particularly beneficial to edge extraction, clip-art JPEG artifact removal, and non-photorealistic image generation.
Motion Blur Removal from A Single Image
Transparency-Based Method (2007)
Abstract: One of the key problems of restoring a degraded image from motion blur is the estimation of the unknown shift-invariant linear blur kernel. We separate the image deblurring into kernel estimation and image deconvolution processes, and propose a novel algorithm to estimate the motion blur kernel from a perspective of alpha values. The relationship between the object boundary transparency and the image motion blur is investigated.
Rotational Motion Deblurring (2007)
Abstract: We model the physical properties of a 2-D rigid body movement and propose a practical framework to deblur rotational motion from a single image. Our main observation is that the transparency cue of a blurred object, which represents the motion blur formation from an imaging perspective, provides sufficient information in determining the object movement. Comparatively, single image motion deblurring using pixel color/gradient information has large uncertainties in motion representation and computation.
Iterative Optimization (2008)
Abstract: This method computes a deblurred image using a unified probabilistic model of both blur kernel estimation and unblurred image restoration. We present an analysis of the causes of common artifacts found in current deblurring methods, and then introduce several novel terms within this probabilistic model that are inspired by our analysis. These terms include a model of the spatial randomness of noise in the blurred image, as well a new local smoothness prior that reduces ringing artifacts by constraining contrast in the unblurred image wherever the blurred image exhibits low contrast. Finally, we describe an efficient optimization scheme that alternates between blur kernel estimation and unblurred image restoration until convergence.

Large Blur Removal (2010)
Abstract: We discuss a few new motion deblurring problems that are significant to kernel estimation and non-blind deconvolution. We found that strong edges do not always profit kernel estimation, but instead under certain circumstance degrade it. This finding leads to a new metric to measure the usefulness of image edges in motion deblurring and a gradient selection process to mitigate their possible adverse effect. We also propose an efficient and high-quality kernel estimation method based on using the spatial prior and the iterative support detection (ISD) kernel refinement, which avoids hard threshold of the kernel elements to enforce sparsity.

Abstract: We propose a simple upsampling method for automatically enhancing the image/video resolution, while preserving the essential structural information. The main advantage of our method lies in a feedback-control framework which faithfully recovers the high-resolution image information from the input data, without imposing additional local structure constraints learned from other examples. This makes our method independent of the quality and number of the selected examples, which are issues typical of learning-based algorithms, while producing results without observable unsightly artifacts. Our method naturally extends to video upsampling.

Abstract: We present a novel method for recovering consistent depth maps from a video sequence. We propose a bundle optimization framework to address the major difficulties in stereo reconstruction, such as dealing with image noise, occlusions, and outliers. Different from the typical multi-view stereo methods, our approach not only imposes the photo-consistency constraint, but also explicitly associates the geometric coherence with multiple frames in a statistical way. It thus can naturally maintain the temporal coherence of the recovered dense depth maps without over-smoothing.
Abstract: We present a user-friendly system for seamless image composition, which we call drag-and-drop pasting. We observe that for Poisson image editing to work well, the user must carefully draw a boundary on the source image to indicate the region of interest, such that salient structures in source and target images do not conflict with each other along the boundary. To make Poisson image editing more practical and easy to use, we propose a new objective function to compute an optimized boundary condition. A shortest closed-path algorithm is designed to search for the location of the boundary. Moreover, to faithfully preserve the object's fractional boundary, we construct a blended guidance field to incorporate the object's alpha matte.
Poisson Matting
 Abstract: We formulate the problem of natural image matting as one of solving Poisson equations with the matte gradient field. Our approach, which we call Poisson matting, has the following advantages. First, the matte is directly reconstructed from a continuous matte gradient field by solving Poisson equations using boundary information from a user-supplied trimap. Second, by interactively manipulating the matte gradient field using a number of filtering tools, the user can further improve Poisson matting results locally until he or she is satisfied. The modified local result is seamlessly integrated into the final result.

Abstract: A robust synthesis method is proposed to automatically infer missing color and texture information from a damaged 2D image by ND tensor voting (N>3). The same approach is generalized to range and 3D data in the presence of occlusion, missing data and noise. Our method translates texture information into an adaptive ND tensor, followed by a voting process that infers non-iteratively the optimal color values in the ND texture space. A two-step method is proposed.
Abstract: This paper presents a complete system capable of synthesizing a large number of pixels that are missing due to occlusion or damage in an uncalibrated input video. These missing pixels may correspond to the static background or cyclicmotions of the captured scene. Our system employs user-assisted video layer segmentation, while themain processing in video repair is fully automatic. The input video is first decomposed into the color and illumination videos. The necessary temporal consistency is maintained by tensor voting in the spatio-temporal domain.

Abstract: Under dimly lit condition, it is difficult to take a satisfactory image in long exposure time with a hand-held camera. Despite the use of a tripod, moving objects in the scene still generate ghosting and blurring effect. In this paper, we propose a novel approach to recover a high-quality image by exploiting the tradeoff between exposure time and motion blur, which considers color statistics and spatial constraints simultaneously, by using only two defective input images. A Bayesian framework is adopted to incorporate the factors to generate an optimal color mapping function. No estimation of PSF is performed.


Editorial Boards

 

Papers Committees

  • Area Chair, IEEE International Conference on Computer Vision (ICCV) 2011
  • Program Committee member, IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2011
  • Program Committee member, Pacific Graphics 2010, 2011
  • Program Committee member, 3DPVT 2010 and 3DIMPVT 2011
  • Program Committee member, IEEE International Conference on Computer Vision (ICCV) 2009
  • Program Committee member, European Conference on Computer Vision (ECCV) 2006, 2008, 2010
  • Program Committee member, IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2008, 2009, 2010
  • Organization Chair, Workshop on Interactive Computer Vision (In conjunction with ICCV) 2007
  • Program Committee member, IEEE International Conference on Computer Vision (ICCV) 2007
  • Program Committee member, Asian Conference on Computer Vision (ACCV) 2006
  • Publication Chair, ACM VRCIA 2006
  • Program Committee member, IEEE International Conference on Computer Vision (ICCV) 2005

 

Grants

  • Co-Investigator, Techniques for Complicated Scene Modeling and Super-high Resolution Rendering, NSFC key project, 2012-2017
  • Principal Investigator, Intelligent Video Revampment and Upsampling, Innovation and Technology Fund (ITF), 2011-2012.
  • Principal Investigator, High-quality Bilayer Segmentation with Motion/Depth Estimation, RGC General Research Fund (GRF), 2011-2013.
  • Principal Investigator, GPU Based Video Editing with Depth Inference, RGC Direct Grant, 2010-2011.
  • Principal Investigator, Single Image Motion Deblurring in a Local-Structure-Oriented Framework, RGC General Research Fund (GRF), 2008-2010.
  • Principal Investigator, Optimizing General Image Stitching, RGC General Research Fund (GRF), 2007-2009.
  • Co-Investigator, Image/Video based Soft Fabric Redressing on Objects/Environment for Interactive E-Commence Applications, RGC General Research Fund (GRF), 2007-2009.
  • Principal Investigator, High Dynamic Range Image and Video Rendering, Compression, and Display, SHIAE Research Grant, 2007-2009.
  • Principal Investigator, Soft Color Segmentation Using Alternative Optimization, RGC Earmarked Grant (GRF), 2006-2008.
  • Principal Investigator, Texture Synthesis and Image Completion on Programmable Graphics Hardware (GPU), Microsoft Research Grant, 2006-2008.
  • Principal Investigator, Enhanced image and video completion, RGC Direct Grant, 2004-2006.

 

Patents
  • Luminance correction, U.S. Patent 7317843
  • Poisson matting for images, U.S. Patent 7636128
  • Digital cameras with luminance correction, U.S. Patent 7463296