Artificial intelligence and machine learning
Research Article
Decomposition of construction method for a language encoder
Igor Vladimirovich Trofimov
| Ailamazyan Program Systems Institute of RAS, Ves'kovo, Russia | |
|
itrofimov@gmail.com |
Abstract. An encoder as part of a language model is a mechanism for converting text information into an effective numerical representation which is suitable for solving a wide range of text processing tasks by means of neural network methods. This paper suggests a way of decomposing of the learning process for a language encoder. The author considers the issues of expediency of such decomposition taking into account reduction of computational costs, quality control at intermediate training stages, provision of the interpretability of the results on each stage. The quality evaluation of the encoder is given. (In Russian).
Keywords: natural language processing, neural networks, language model, encoder, context-sensitive representations, lexical ambiguity resolution
MSC-2020
68T07; 68T50
For citation: Igor V. Trofimov. Decomposition of construction method for a language encoder. Program Systems: Theory and Applications, 2023, 14:1, pp. 31–54. (In Russ.). https://psta.psiras.ru/2023/1_31-54.
Full text of article (PDF): https://psta.psiras.ru/read/psta2023_1_31-54.pdf.
The article was submitted 13.11.2022; approved after reviewing 17.01.2023; accepted for publication 09.02.2023; published online 19.02.2023.
