Understanding and Improving Encoder Layer Fusion in Sequence-to-Sequence Learning

UM > Faculty of Science and Technology > DEPARTMENT OF COMPUTER AND INFORMATION SCIENCE

Status	已發表Published
	Understanding and Improving Encoder Layer Fusion in Sequence-to-Sequence Learning
	Liu, X.; Wang, L.; Wong, F.; Ding, L.; Chao, L.; Tu, Z.
	2021-05-03
Source Publication	The Ninth International Conference on Learning Representations
Abstract	Encoder layer fusion (EncoderFusion) is a technique to fuse all the encoder layers (instead of the uppermost layer) for sequence-to-sequence (Seq2Seq) models, which has proven effective on various NLP tasks. However, it is still not entirely clear why and when EncoderFusion should work. In this paper, our main contribution is to take a step further in understanding EncoderFusion. Many previous studies believe that the success of EncoderFusion comes from exploiting surface and syntactic information embedded in lower encoder layers. Unlike them, we find that the encoder embedding layer is more important than other intermediate encoder layers. In addition, the uppermost decoder layer consistently pays more attention to the encoder embedding layer across NLP tasks. Based on this observation, we propose a simple fusion method, SurfaceFusion, by fusing only the encoder embedding layer for the softmax layer. Experimental results show that SurfaceFusion outperforms EncoderFusion on several NLP benchmarks, including machine translation, text summarization, and grammatical error correction. It obtains the state-of-the-art performance on WMT16 Romanian-English and WMT14 English-French translation tasks. Extensive analyses reveal that SurfaceFusion learns more expressive bilingual word embeddings by building a closer relationship between relevant source and target embeddings. Source code is freely available at https://github.com/SunbowLiu/SurfaceFusion.
Keyword	Sequence-to-Sequence Neural Machine Translation
Language	英語English
The Source to Article	PB_Publication
PUB ID	58017
Document Type	Conference paper
Collection	DEPARTMENT OF COMPUTER AND INFORMATION SCIENCE
Recommended Citation GB/T 7714	Liu, X.,Wang, L.,Wong, F.,et al. Understanding and Improving Encoder Layer Fusion in Sequence-to-Sequence Learning[C], 2021.
APA	Liu, X.., Wang, L.., Wong, F.., Ding, L.., Chao, L.., & Tu, Z. (2021). Understanding and Improving Encoder Layer Fusion in Sequence-to-Sequence Learning. The Ninth International Conference on Learning Representations.

Files in This Item:
There are no files associated with this item.

If you have any objections to this item, please fill out the form below and the administrator will contact you as soon as possible.
Content:
Email：	*
Affiliation No.
Verification Code:	Refresh

Any comments and suggestions are welcomed.
Title:	*
Content:
Email：	*
Verification Code:	Refresh