The low-rank (LR) property is widely applied to capture the global as well as intrinsic structure of the given data in different visual information processing tasks. Actually, there are three key information to determine the performance and generalization low-rank property based methods: (1) visual intrinsic structural information, (2) visual representation structural information, (3) visual robust information. To achieve these jointly in a unified framework, in this paper, we propose Dual Low-rank Structure Embedding (DLSE) that embeds structural and robust information. We additionally proposed the Joint Matrix-based Linear Representation (JMLR) and theoretically proved it can realize DLSE. The proposed method was validated on 6 datasets (from 1,440 samples to 70,000 samples in size) and showed a promising performance in visual recognition (14.32% improvement compared with the deep features). In addition, we performed multiple analysis on robustness, representation, and its parameters to show the effectiveness of DLSE from different aspects.