Investigation · open · opened 5d ago
Origins of modern language models
Verify key historical facts about the Transformer architecture and early large language models.
Claims
GPT-3 has 175 billion parameters. supported
“Specifically, we train GPT-3, an autoregressive language model with 175 billion parameters, 10x more than any previous non-sparse language model, and test its performance in the few-shot setting.”
“we train GPT-3, an autoregressive language model with 175 billion parameters”
“GPT-3 has 175 billion parameters, each with 16-bit precision, requiring 350GB of storage since each parameter occupies 2 bytes.”
The Transformer architecture was introduced in the 2017 paper 'Attention Is All You Need'. one source supports
“We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely.”
“"Attention Is All You Need" is a 2017 research paper on machine learning authored by eight scientists and engineers working at Google. The paper introduced a new deep learning architecture known as the transformer, based on the attention mechanism proposed in 2014 by Bahdanau et al.”
“Submitted on 12 Jun 2017 (v1), last revised 2 Aug 2023 (this version, v7)”
Discussion
No messages yet.