Understanding and predicting each individual's real-time travel destination given the origin information in urban public transportation systems is crucial for personalized traveler recommendation, targeted demand management, dynamic traffic operations and so on. Existing methods are often based on modeling the regular travel patterns through analyzing the long-term personal travel information. They are suitable for destination prediction of individual regular trips with regular travel patterns, but may not work well for occasional trips with strong randomness and uncertainty, especially for the individuals with a few historical travel data. In this paper, we focus on more challenging issue about destination prediction of occasional trips. We design a general Multi-View Deep Learning Framework (MDLF) based on the data-driven insight that a location where a user will destine to is not only related to the user's own travel preference to the location, but also influenced by crowd's travel preference and the region's characteristics of the location under certain spatiotemporal contexts. The destination of an individual's occasional trip can be predicted by combining all these complementary influencing factors. The novelty of MDLF is mainly reflected in two aspects. The first is the effective feature extraction from multiple and complementary views. The second is that a CNN (Recurrent Neural Network) based deep learning component for predicting each occasional trip's destination by calculating a moving trend score for each possible destination. We evaluate the MDLF based on two real-world smart card datasets collected by AFC (Automatic Fare Collection) Systems. The experimental results demonstrate the superiority of MDLF against other competitors.