Beyond Fake News, Artificial Intelligence finally understands the world (a little)

But in recent days, NLP has also caused fear. Working towards AI for humanity, the OpenAI Foundation has published the results of its latest neural network, elegantly named “GPT-2”, which is capable of generating prose of amazing consistency.

GPT-2 in action

Prompted with “Russia has declared war on the United States after Donald Trump accidentally” GPT-2 writes the following article:

« Russia has declared war on the United States after Donald Trump accidentally fired a missile in the air.

Russia said it had “identified the missile’s trajectory and will take necessary measures to ensure the security of the Russian population and the country’s strategic nuclear forces.” The White House said it was “extremely concerned by the Russian violation” of a treaty banning intermediate-range ballistic missiles.

The US and Russia have had an uneasy relationship since 2014, when Moscow annexed Ukraine’s Crimea region and backed separatists in eastern Ukraine. »

And this is one of the shorter examples, you can see longer ones here and here (unicorns!) as well as all the generated text.

AI Transformers: much more than “fake news”

OpenAI decided that the GPT-2 neural network was dangerous and chose to make only a simplified version of the model available to the public. This was followed by enormous media coverage focusing almost exclusively on the possibility of generating “fake news”.

But GPT-2 is much more than that, as it is a “Transformer” type neural network. This leap forward in NLP dates back to Google’s publication of “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding” in October 2018. Transformers do much more than generate text.

Basically Transformers learn a simple task: given a sequence of words predict the next word. It doesn’t sound very impressive; the predictive text on our phone has been doing that for a long time. Pure probability calculation without any intelligence.

The main difference is that Transformers are trained by reading millions (!!) of pages of text and in order to be able to predict the next word in all these situations they must constitute a model of the world of sorts. This model is as much a grammatical model of language as it is a model of the nature of the entities and relationships of this world. Once this model has been created, it can be used for many other language processing tasks, for example:

• Generate a summary of a document
• Perform translations
• Answer questions about the content of documents
• Extract key information
• Have natural interactions
• Understand the subtlety of requests
• Assist in writing
• Create articles from key elements

Implications for the future

Two key facts emerge from this:

• language is being “solved” in the same way as vision has been in recent years. This implies that superhuman performance for well-defined reading and writing tasks is within our reach; and that we will soon see a new generation of tools harnessing it.
• Contrary to what is often read, the most advanced current AIs do develop a form of understanding of the world and a notion of “common sense”. We are witnessing the start of an upheaval of the world as we know it.

We can also observe that Transformers are “simply” an evolution of deep learning and illustrate once again that this evolutionary work already approaches, even exceeds, human performance on a large number of activities that were considered as human-exclusive hunting grounds just 5 years ago. It’s become urgent to anticipate the inevitable evolution of our activities in order to build a beneficial relationship with AI.