This paper introduces a novel method for real-time simultaneous input-state-parameter estimation with modulated colored noise excitation. In this method, the unknown inputs are modeled as modulated colored noise, which is realistic for real environmental excitations. By considering this input model, the typical problem of low-frequency drifts in the inputs and displacement responses estimation can be eliminated. The proposed approach provides a joint input-state-parameter estimation algorithm in a real-time manner by introducing the corresponding modeling equation for the unknown inputs into the extended Kalman filter. A Bayesian methodology is utilized to recursively update the noise covariance matrices in the filtering so stationarity of the input/response is not required for the proposed algorithm. Examples using truss and bridge models under different stationary scenarios are investigated. Compared with other existing methods that suffer from severe low-frequency drifts using only noisy acceleration measurements, the presented approach achieves substantially stable and accurate estimation for inputs, structural states and parameters.