Robotics is one important domain of artificial intelligence applications. Proliferation of robotics makes our lives more convenient and colorful. Emotion recognition is one important task for a robot to communicate with human beings. Although different works have been proposed to complete the task of emotion recognition, conventional methods often require a lot of time and memory resources. Since a robot is often required to react timely and equipped with limited resources, we propose an enhanced Siamese network algorithm to resolve such a problem. We combine Siamese network with broad learning system while further improving mechanism of similarity metric. Experiment results show that our method can achieve a comparable performance to conventional deep learning method while reducing consumption of computing time and memory resources.