Androgen receptor (AR) plays a central role in driving prostate cancer (PCa) progression. How AR promotes this process is still not completely clear. Herein, we used single-cell transcriptome analysis to reconstruct the transcriptional network of AR in PCa. Our work shows AR directly regulates a set of signature genes in the ER-to-Golgi protein vesicle-mediated transport pathway. The expression of these genes is required for maximum androgen-dependent ER-to-Golgi trafficking, cell growth, and survival. Our analyses also reveal the signature genes are associated with PCa progression and prognosis. Moreover, we find inhibition of the ER-to-Golgi transport process with a small molecule enhanced antiandrogen-mediated tumor suppression of hormone-sensitive and insensitive PCa. Finally, we demonstrate AR collaborates with CREB3L2 in mediating ER-to-Golgi trafficking in PCa. In summary, our findings uncover a critical role for dysregulation of ER-to-Golgi trafficking expression and function in PCa progression, provide detailed mechanistic insights for how AR tightly controls this process, and highlight the prospect of targeting the ER-to-Golgi pathway as a therapeutic strategy for advanced PCa.