Most of the price movements of stocks are based on certain patterns, and machine learning techniques can be very good at learning these data models for prediction purposes. This paper provides a comprehensive review and comparison of four stock forecasting models including Autoregressive Integrated Moving Average (ARIMA), Recurrent Neural Network (RNN), Long Short-Term Memory (LSTM) and Transformer. The objective of the study is to gain insights into the methodology, strengths, and limitations of these models in financial market forecasting and to provide recommendations for future research and applications. In the analysis of ARIMA model, this paper highlights its statistical approach based on time series, modeling both smooth and non-smooth series through a combination of autoregression, differencing and moving average. And it emphasizes the advantages of ARIMA in capturing trends and periodicity, but also points out its limitations for nonlinear and complex dynamics. For neural network modeling, this paper delves into the recurrent structure of RNN and LSTM, as well as Transformer's self-attention mechanism. These models excel in recognizing and analyzing long-term trends, managing non-linear dynamics, and adjusting to intricate patterns. However, they also encounter obstacles such as complex training processes and significant demands on computational resources. In a thorough comparison of the four models, the paper underscores their limitations in terms of how easily they can be interpreted and applied. It also offers guidance on choosing the most suitable models or combining different models in various contexts to enhance the accuracy and reliability of predictions.