The vibration caused by resonance modes frequently occurs during acceleration and decel-eration of the modular joint integrated with flexible harmonic drive. The conventional equivalent rigid-body velocity method with observer can suppress the residual vibration induced by resonant frequency but has poor robustness to model uncertainties and external disturbances. Moreover, it cannot eliminate the torque ripple caused by the harmonic drive during low-speed uniform motion, reducing the velocity tracking accuracy. Hence, a velocity controller with a rigid-body state observer and an adjustable damper is designed to improve the robust performance and velocity tracking accuracy. The designed rigid-body state observer allows a higher gain so that the bandwidth of the observer can increase, and the equivalent rigid-body velocity can be acquired more accurately. Notably, the high gain observer reduces the sensitivity to model uncertainties and exotic distur-bances, especially near the resonant frequency. In addition, the observer combined with an adjustable damper can suppress the residual vibration and torque ripple simultaneously. The proposed method is compared experimentally with a PI method and two other rigid-body velocity methods, such as the conventional equivalent rigid-body observer method and the self-resonance cancellation method, to verify its advantages.