In this study, the exponential-polynomial-closure (EPC) method is applied to determine and study the non-stationary probability density function (PDF) of the nonlinear system simultaneously stimulated by parametric harmonic and modulated stochastic excitations. The non-stationary probabilistic responses are represented as a time-related exponential polynomial function with unknown variables. Integrating the weighted residual error can result in a group of first-order differential equations. Those equations can be solved by numerical techniques to obtain the undefined variables. Four examples are given to show that the non-stationary behaviors of the presented oscillators can be addressed by the presented EPC procedure effectively, especially in the probabilistic tails which are of particular importance in reliability analysis. Moreover, the outcomes show that the response means equal to zero and the response PDFs are symmetrical with respect to the zero means. In particular, the presented EPC procedure can heavily decrease the running effort with no sacrifice to the solution correctness when compared with MCS.