This article investigates the real-time semantic segmentation in robot engineering applications based on the Broad Learning System (BLS), and a novel Multi-level Enhancement Layers Network (MELNet) based on BLS framework is proposed for real-time vision tasks in a complex street scene on the unmanned mobile robot. This network mainly solves two problems: (1) mitigating the contradiction between accuracy and speed while maintaining low model complexity, and (2) accurately describing objects based on their shape despite their different sizes. Firstly, the BLS architecture is expanded to the deep network with trainable parameters. This trainable network could adjust its weights in a complex environment, and mitigate the adverse impact of the environment on the complex tasks. Secondly, enhancement layers with the extended enhancement layers could extract both detailed information and semantic information. Moreover, an Upsampling Atrous Spatial Pyramid Pooling (UPASPP) is designed to fuse detail and semantic information to describe object features properly. Finally, in the case of the MNIST dataset and Cityscapes dataset, we get high accuracy with 8.01M parameters and quicker inference speed on a single GTX 1070 Ti card. At the same time, the unmanned mobile robot (BIT-NAZA) is employed to evaluate semantic performance in real-world situations. This reveals that MELNet could be run adequately on the embedded device and effectively operate in the real-robot system.