Lines indicate structure information of objects. However, the general line detectors cannot give enough clear information with many short or discontinuous line segments. This study presents a new fast three-phase line segment clustering algorithm. Firstly, Hough transform or LSD algorithm is used to attain initial line set; and then these lines are grouped into different sets according to direction; and then each direction set is further subdivided into different sub-sets according to their relative distances; finally the lines are merged or split on the basis of their neighborhood relations to form the final groups. Compared to previous work, the present method is more efficient and easier to implement. More importantly, the clustered line segments can fully indicate the structure information of targets in the image which is verified by the experiments. © 2013 Asian Network for Scientific Information.