In this paper, we present a hybrid representation of image-based models combining the textured planes and the hierarchical points. Taking a set of depth images as input, our method starts from classifying input pixels into two categories, indicating the planar and non-planar surfaces respectively. For the planar surfaces, the geometric coefficients are reconstructed to form the uniformly sampled textures. For nearly planar surfaces, some textured planes, called lumiproxies, are constructed to represent the equivalent visual appearance. The Hough transform is used to find the positions of these textured planes, and optic flow measures are used to determine their textures. For remaining pixels corresponding to the non-planar geometries, the point primitive is applied, reorganized as the OBB-tree structure. Then, texture mapping and point splatting are employed together to render the novel views, with the hardware acceleration. © 2009 Springer.