Artificial Intelligence: A Decade of Development

The past decade has been an exciting and eventful journey for the field of artificial intelligence (AI). The exploration of the potential of deep learning has led to an explosive growth of a new field, including recommendation systems in e-commerce, object detection in autonomous vehicles, and generative models that can create realistic images and coherent text.

2013 was widely regarded as the year when deep learning “came of age,” sparked by significant advancements in computer vision. VAEs are generative models that can learn to represent and generate data such as images and sounds. They work by learning a compressed representation of the input data, or the latent space, in a lower-dimensional space. This allows them to generate new data by sampling from the learned latent space. Later, VAEs were proven to have opened up new avenues for generative modeling and data generation, with widespread applications in fields such as art, design, and gaming.

In June 2014, the field of deep learning saw another major breakthrough with the introduction of GANs, a neural network that can generate new data samples that are similar to the training set. Essentially, two networks are trained simultaneously: (1) the generator network generates fake samples, and (2) the discriminator network evaluates their authenticity. This training is conducted in a game-like environment, where the generator tries to create samples that can deceive the discriminator, while the discriminator tries to correctly judge the fake samples. At the time, GANs represented a powerful and novel data generation tool not only for generating images and videos, but also for music and art. They also promoted the development of unsupervised learning, which was generally considered underdeveloped and challenging, as they demonstrated the possibility of generating high-quality data samples without explicit labels.

In 2015, significant progress was made in artificial intelligence in computer vision and natural language processing (NLP). ResNets is an architecture that allows information to flow more easily through the network by adding shortcuts. Unlike conventional neural networks, in ResNet, additional residual connections are added, skipping one or more layers and directly connecting to deeper layers in the network. Therefore, ResNets are able to solve the problem of gradient vanishing, making it possible to train deeper neural networks beyond what was expected at the time. This in turn significantly improved image classification and object recognition tasks.

Meanwhile, significant progress was made in the development of recurrent neural networks (RNNs) and long short-term memory (LSTM) models. Despite the existence of these models since the 1990s, they did not receive much attention until around 2015, mainly due to the following reasons: (1) the availability of larger and more diverse datasets for training, (2) improvements in computing power and hardware that enabled the training of deeper, more complex models, and (3) modifications along the way, such as more complex gate mechanisms.
Therefore, these architectures enabled language models to better understand the context and meaning of text, leading to significant improvements in tasks such as language translation, text generation, and sentiment analysis. The success of RNNs and LSTMs paved the way for the development of large language models (LLMs) as we see today.

The human vs. machine Go match in 2016 shocked the gaming world: Google’s AlphaGo defeated Go world champion Lee Sedol. AlphaGo used a combination of deep reinforcement learning and Monte Carlo tree search to analyze millions of game states from previous matches and evaluate the best move – a strategy that far surpassed human decision-making capabilities in this context.

2017 was a crucial year for the breakthrough of generative artificial intelligence. The Transformer model consists of two basic components: an encoder and a decoder. The encoder is responsible for encoding the input data, such as a word sequence. It receives an input sequence and applies multiple self-attention layers and feedforward neural networks to capture relationships and features within sentences and learn meaningful representations. The self-attention mechanism enables the model to understand the relationships between different words in a sentence. Unlike traditional models, which process words in a fixed order, the Transformer model actually considers all words simultaneously. It assigns a value called an attention score based on the relevance of a word to other words in the sentence. On the other hand, the decoder receives the encoded representation from the encoder and generates an output sequence. In tasks such as machine translation or text generation, the decoder generates a translation sequence based on the input received from the encoder. Similar to the encoder, the decoder is also composed of multiple self-attention layers and feedforward neural networks. However, the decoder also includes an additional attention mechanism that allows it to focus on the output of the encoder. This way, the decoder can consider relevant information from the input sequence when generating the output.

The Transformer architecture has become a key component in the development of large language models (LLMs) and has achieved significant improvements in natural language processing tasks such as machine translation, language modeling, and question-answering systems. OpenAI introduced the Generative Pre-trained Transformer (GPT-1) in June 2018, which effectively captures long-range dependencies in text using the Transformer architecture. GPT-1 was one of the first models to demonstrate unsupervised pre-training and fine-tuning for specific NLP tasks. Google also leveraged this relatively new Transformer architecture, releasing and open-sourcing their own pre-training method, called Bidirectional Encoder Representations from Transformers (BERT), in late 2018.

2019 was a year of several notable advancements in the field of generative models, particularly the introduction of GPT-2. This model achieved state-of-the-art performance on many natural language processing tasks and was able to generate highly realistic text, which gave us a sense of what might be to come. Other improvements in the field included DeepMind’s BigGAN, which generated high-quality images that were almost indistinguishable from real images, and NVIDIA’s StyleGAN, which allowed for better control over the appearance of generated images.

In 2020, another model emerged, becoming a household name even outside the tech community: GPT-3. This model achieved a significant leap in scale and functionality for large language models. For example, GPT-1 had only 117 million parameters. GPT-2 increased the number of parameters to 1.5 billion, and GPT-3 reached an astounding 175 billion. Such a vast parameter space allows GPT-3 to generate very coherent text in a wide range of prompts and tasks. It has demonstrated impressive performance in various natural language processing tasks, such as text completion, question-answering, and even creative writing. GPT-3 once again highlights the potential of self-supervised learning, which enables models to be trained on large amounts of unlabeled data. This has the advantage that these models can acquire a broad understanding of language without the need for extensive training on specific tasks, making them more economical and efficient.

In 2021, AlphaFold 2 was hailed as the long-awaited solution to the protein folding problem that had persisted for decades. DeepMind researchers expanded the Transformer architecture to create evoformer blocks, which use evolutionary strategies to optimize the model, thereby building a model that can predict the 3D structure of proteins based on their 1D amino acid sequences. This breakthrough has huge potential to fundamentally change drug discovery, biological engineering, and our understanding of biological systems.

In 2022, artificial intelligence made a breakthrough: OpenAI’s ChatGPT, a chatbot, was launched in November. This tool represents the cutting-edge results in natural language processing and is able to generate coherent and contextually relevant responses to various queries and prompts. 2023: Large Language Models (LLMs) and Robots. This year undoubtedly became the year of LLMs and chatbots. An increasing number of models are being developed and released at a rapid pace. Stanford University researchers released Alpaca, a lightweight language model, which was fine-tuned from LlaMA through an instruction following demonstration. Just a few days later on March 21, Google launched Bard, a competitor to ChatGPT. Google also released its latest LLM, PaLM-2, on May 10th of this year. In the relentless development of this field, it is highly likely that another model has already emerged by the time you read this article.

We also see an increasing number of companies integrating these models into their products. For example, Duolingo announced the launch of Duolingo Max, a new subscription level, designed to provide a customized language course for everyone. Slack also introduced an AI assistant named Slack GPT, which can perform tasks such as drafting responses or summarizing discussion threads. Additionally, Shopify introduced a ChatGPT-based assistant into its Shop app to help customers identify desired products using various prompts.