This invention introduces a method and device for fine-tuning multi-layer Transformer, which is a typical application of deep transfer learning. Multi-layer Transformer is a popular hierarchical architecture widely adopted in deep learning.  Specifically, this invention introduces a novel and effective regularization method to improve the fine-tuning process, referred to as layer-wise noise stability regularization. It adds random noise to the input and get the output deviations w.r.t. each layer of the multi-layer Transformer. These deviations are constrained during fine-tuning as a penalty term that will be minimized, in addition to the original ERM loss.