Stacked magnetostrictive actuator (SMA) has the advantages of high energy density and high bandwidth, but the output stroke is relatively small and accompanied by strong hysteresis nonlinearity. Introducing the radial-nested stacked configuration, the stroke of a SMA can be increased without deteriorating its bandwidth. However, this configuration consists of three magnetostrictive rods of different shapes which brings more serious asymmetric hysteresis nonlinearity and poses a great challenge on the theoretical modeling of the actuator. In this paper, a magnetic equivalent circuit (MEC) model is established to describe the magnetic characteristic of radial-nested stack. Then, a nonlinear dynamic magnetization model is proposed with the combination of the MEC model and the Jiles-Atherton model. Finally, by considering the multi-degree-of-freedom (MDOF) mechanical dynamic system, a multiphysics comprehensive dynamic (MCD) model is established. What’s more, a prototype of radial-nested stacked Terfenol-D actuator (RSTA) is fabricated, a series of simulations and experiments are conducted to evaluate the proposed models. The parameters that cannot be calculated or measured in the model are identified by employing the multi-island genetic algorithm. Results show that: (a) the MEC model can accurately calculate the magnetic distribution of the RSTA with an error less than 0.2% compared with a finite element model; (b) the MCD model can accurately describe the RSTA output hysteresis nonlinearity under different operating frequencies and amplitudes with a root-mean-square (RMS) error less than 1.1 μ m (1.76%).