1. Color image tracking and Segmentation

By using mixture models in color space, we are able to segment and track an object in a panorama. The figure below illustrate the concept of segmentation using panorama.

In the figure above, a background panorama is first obtained as the background mosaic. A simple camera will then capture images as the source frames on left. Together with the background mosaic, the foreground object can then be segmented out and then transmitted to the remote site. Reconstruction is then performed by selecting the appropriate portion from the background frame and overlay with the foreground.

The video below illustrate the first stage output (raw output through individual pixel classification, without using bounding box assistance)

and the result

Significant compression can be achieved with this technique. The details can be referred from our paper below.

Publications:

S.H. Or, K.H. Wong, K.S. Lee, T.K. Lao, "Panoramic Video Segmentation Using Color Mixture Models", ICPR'2000, the 15th international conference on pattern recognition conference sponsored by the International Association for Pattern Recognition, Sept 3-8 2000, Barcelona, Spain.  pdf

2. Building reconstruction

Given a camera which is placed at a point inside a building with the restricted freedom of rotating about that point, can one estimate the structure surrounding that point with just the images captured?  In this project,  we use a novel technique to solve the above problem, in particular recovering the 3D line structure from a panorama at a point. We use an optimization-based approach, which used a minimum standard deviation of angles  MSDA as the cost function, to recover the structure from the line drawing representation of the scene. The observation that the constraints at a particular viewpoint can propagate to other views through the use of panorama is our motivation behind using a cylindrical formulation in our algorithm. We demonstrate the effectiveness of our algorithm with both synthetic and real data. The proposed technique is suitable for creating the virtual environment of the interior of an existing building.

Input panorama :

Reconstructed model at different viewing angles (the model is rendered using Crystal Space):

Publications:

S.H.Or & T.T. Wong, Recovering the interior of a building from a panorama” accepted for presentation at Vision, Modeling & Visualization 2002, Erlangen, Germany. pdf