How to break the bottleneck of Decoder performance? Nvidia experts reveal the secrets

Since "Attention is All You Need" was introduced in 2017, Transformer has become a very popular deep learning network architecture in the NLP space. However, in the inference deployment phase, its computing performance often fails to meet the requirements of low latency and high throughput for online services.

In Nvidia's open-source FasterTransformer 1.0 version, the Transformer Encoder in BERT has been optimized and accelerated to reduce the latency of coding with transformer.

Having solved the Encoder performance problem, Nvidia has focused on the equally important Transformer Decoder reasoning.

As a result, Nvidia has introduced version 2.0 of FasterTransformer, which offers a transformer layer that is highly optimized for decoders. At the same time, the optimized translation process is also provided to meet the needs of users who want to significantly reduce latency in translation scenarios.