UMBC CMSC 471 02 Spring 2021
Introduction to Artificial Intelligence

Homework EX1: Experimenting with Transformers

out Tue 4/27, due Wed 5/21

Assignment repository invitation

This is a simple extra credit homework that asks you to experiment with some pretrained GPT-2 models to generate text from initial prompts.

The transformer model is a neural network model introduced in 2017 that is mostly used for text processing applications like machine translation, text summarization and question answering.

GPT-2 and GPT-3 are large-scale unsupervised transformer language models developed by the company OpenAI that are trained to predict text that might follow an initial sequence of words. The current publicly available GPT-2 network has 1.5B parameters. It was trained on the WebText dataset, which has 40GB of text extracted from the contents of 45 million links posted by users of the ‘Reddit’ social network on or before 2017.

Talk To Transformer is online demonstration by Adam Daniel King that was built with GPT-2 in mid 2019. It asks for the beginning of some text and automatically extends it with additional sentences. Write with a Transformer is another web-based demo supported by Hugging Face that uses GPT-2.

You can easily experiment with the Hugging Face GPT-2 based system in Python using its Transformers package

GPT-3 was released in the summer of 2020. It has a larger network with 175B parameters. Microsoft has recently licensed "exclusive" use of GPT-3 and control of its source code. Others can still use the public API to receive output after applying to get an API key, but its not easy to get one. A paper on GPT-3, Language Models are Few-Shot Learners, received a best-paper award from the prestigious NeurIPS conference just last week. There are many popular articles with examples of what GPT-3 can and cannot do and these will no doubt continue in the coming year.

Haim is another interesting system that you can try online. It uses a model similar to GPT-2 that was trained with a public subset of OpenAI's WebText data. What's novel and interesting about Haim is that it uses what they call an "interpolating language model"

"Given a human-written beginning (prefix) and human-written ending (suffix), HAIM generates synthetic text (body) that fits between them with a desired length. Thus, HAIM introduces two new “knobs” for tuning its output: the suffix, for keeping the generated text on topic, and the length, for controlling the amount of text inserted between the prefix and the suffix."

As an example, we gave Haim the first and last sentence of this paragraph and it filled in the rest.

Double Indemnity is one of the most influential films ever made. Not only is it considered to be the best of the genre, but arguably the best of all time, period. It was the subject of numerous publications and was ranked on many "best of" lists in the early 40's. It was nominated for the Oscars, winning three times. The film's influence has spread through the decades, and it is one of the highest grossing films of all time in the US, has been remade, and is in the process of being a remake. It remains as compelling as ever. The film was made in 1951 and set just years after the 1929 Great Chicago Fire, in a forgotten suburb. It helped define Film Noir and will remain an important film for years to come.

What to do

After reading the items linked from this page, experiment with the two systems.

• HuggingFace's Transformer model using the notebook in your repository or on colab. This can run on your own system if you have the packages installed or on Google Colab. If you run it on Colab, we recommend you use their servers with GPU support. This will allow them to run several times faster. To enable this, first navigate to Edit→Notebook Settings and then select GPU from the Hardware Accelerator drop-down . On my relatively high-end macbook, it takes about 70-90 seconds to generate a ~300 word text and about 30-40 seconds on Colab with GPU support.
  
  
• OpenAI's HAIM demonstration model on the web site

For each produce a series of medium-length paragraphs on different three topics by providing an initial sentence (or two) and, for HAIM, a final sentence (or two). The topics should be varied and might including things like computers, AI, sports, politics, philosophy, or popular music.

Edit the file TransformerExperiments.md file in your repository to record your results, using the the format of our example, and answering the questions at the end.

For each of the two systems show at least three interesting examples for each topic, ranging from good to bad and provide a one sentence assessment of each. Feel free to do more than three topics and use more than three examples.

Conclude with an overall assessment of a paragraph or two on Huggingface's Transformers and OpenAI's Haim. You might address questions like the following: Is it just an interesting example? Might it have utility as it is? What might make it better? Do you think a version trained on a different collection of text (e.g., cybersecurity reports) might be useful?