Natural language processing technologies, such as topic models, have been proven to be effective for scholarly recommendation tasks with the ability to deal with content information. Recently, venue recommendation is becoming an increasingly important research task due to the unprecedented number of publication venues. However, traditional methods focus on either the author's local network or author-venue similarity, where the multiple relationships between scholars and venues are overlooked, especially the venue-venue interaction. To solve this problem, we propose an author topic model-enhanced joint graph modeling approach that consists of venue topic modeling, venue-specific topic influence modeling, and scholar preference modeling. We first model the venue topic with Latent Dirichlet Allocation. Then, we model the venue-specific topic influence in an asymmetric and low-dimensional way by considering the topic similarity between venues, the top-influence of venues, and the top-susceptibility of venues. The top-influence characterizes venues' capacity of exerting topic influence on other venues. The top-susceptibility captures venues' propensity of being topically influenced by other venues. Extensive experiments on two real-world datasets show that our proposed joint graph modeling approach outperforms the state-of-The-Art methods.