Combining Matrix Factorization (MF) with Network Embedding (NE) has been a promising solution to social recommender systems. However, in most of the current combined schemes, the user-specific linking proportions learned by NE are fed to the downstream MF, but not reverse, which is sub-optimal as the rating information is not utilized to discover the linking features for users. Furthermore, the existing combined models mainly focus on enhancing the representation learning for users by exploiting user-user network, yet ignore the representation improvement for items. In this paper, we propose a novel social recommendation scheme, called MF with dual-network collaborative embedding (MF-decoding), which jointly optimizes an integrated objective function of MF and NE, in which both MF and NE tasks can be mutually reinforced in a unified learning process. In particular, the explicit user-user network and the implicit item-item network are collaboratively used by MF-decoding to enhance the representation learning for users and items simultaneously. Our encouraging experimental results on three benchmarks validate the superiority of the proposed MF-decoding model over state-of-the-art social recommendation methods.