A method of detection and localization in lip- reading system based on least square fitting and two stage ellipse fitting was proposed in this paper. Firstly, color space transformation in simple background facial images was made to simply and fast detect skin regions and get threshold for segmenting lip area; secondly, according to two stage ellipse fitting based on least square, lip image are gained. Results show that this method is more effective in fitting effect than algebraic distance fitting. And the computational efficiency is improved compared with geometric distance fitting, and the goal of recognition in lip-reading system could also be archived. Experimental set was limited in sample space of specific speaker and in image sequence features of 403 speech video files, the recognition rate can get up to 90.9%.