Conditioning Transformers¶

Providing more conditioning gives more control. This principle might be observed in neuron firing, desicion theory, and process control.

Go to Runtime → Run all (Ctrl.+F9) to experiment yourself. Have fun! 😊

Setup¶

Essentials but not the point of research

In [ ]:
%tensorflow_version 2.x

!pip install git+https://github.com/huggingface/transformers

import tensorflow as tf
import transformers

Collecting git+https://github.com/huggingface/transformers
  Cloning https://github.com/huggingface/transformers to /tmp/pip-req-build-tffp1yf3
  Running command git clone -q https://github.com/huggingface/transformers /tmp/pip-req-build-tffp1yf3
Requirement already satisfied (use --upgrade to upgrade): transformers==2.9.1 from git+https://github.com/huggingface/transformers in /usr/local/lib/python3.6/dist-packages
Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from transformers==2.9.1) (1.18.4)
Requirement already satisfied: tokenizers==0.7.0 in /usr/local/lib/python3.6/dist-packages (from transformers==2.9.1) (0.7.0)
Requirement already satisfied: filelock in /usr/local/lib/python3.6/dist-packages (from transformers==2.9.1) (3.0.12)
Requirement already satisfied: requests in /usr/local/lib/python3.6/dist-packages (from transformers==2.9.1) (2.23.0)
Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.6/dist-packages (from transformers==2.9.1) (4.41.1)
Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.6/dist-packages (from transformers==2.9.1) (2019.12.20)
Requirement already satisfied: sentencepiece in /usr/local/lib/python3.6/dist-packages (from transformers==2.9.1) (0.1.90)
Requirement already satisfied: sacremoses in /usr/local/lib/python3.6/dist-packages (from transformers==2.9.1) (0.0.43)
Requirement already satisfied: dataclasses in /usr/local/lib/python3.6/dist-packages (from transformers==2.9.1) (0.7)
Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests->transformers==2.9.1) (3.0.4)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests->transformers==2.9.1) (1.24.3)
Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests->transformers==2.9.1) (2.9)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests->transformers==2.9.1) (2020.4.5.1)
Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from sacremoses->transformers==2.9.1) (1.12.0)
Requirement already satisfied: click in /usr/local/lib/python3.6/dist-packages (from sacremoses->transformers==2.9.1) (7.1.2)
Requirement already satisfied: joblib in /usr/local/lib/python3.6/dist-packages (from sacremoses->transformers==2.9.1) (0.15.0)
Building wheels for collected packages: transformers
  Building wheel for transformers (setup.py) ... done
  Created wheel for transformers: filename=transformers-2.9.1-cp36-none-any.whl size=652120 sha256=98c783f4edbf0f1a45c982d4bb884fe363cffbacd0c9db54f65a1d6e49198e65
  Stored in directory: /tmp/pip-ephem-wheel-cache-fmjddmul/wheels/70/d3/52/b3fa4f8b8ef04167ac62e5bb2accb62ae764db2a378247490e
Successfully built transformers
In [ ]:
import logging
logging.getLogger('transformers.tokenization_utils').setLevel(logging.ERROR)
logging.getLogger('transformers.tokenization_utils').disabled = True
In [ ]:
tokenizer = transformers.GPT2Tokenizer.from_pretrained('gpt2')
transformer = transformers.TFGPT2LMHeadModel.from_pretrained('gpt2')
In [ ]:
def speak(text):
    inp_tokens = tokenizer.encode(text, return_tensors='tf')
    out_tokens = transformer.generate(
        inp_tokens,
        max_length=50,
        top_k=50,
        top_p=0.7,
        do_sample=True,
        temperature=0.75,
        repetition_penalty=10)
    return tokenizer.decode(out_tokens[0])

Experiment¶

I predict that sentences will appear more similar as conditioning length increases.

In [ ]:
def experiment(text):
    """condition gpt-2 generation on progressively longer starting sequences
    args:
        text: string language input
    returns:
        gpt-2 outputs
    """

    words = text.split(' ')
    for conditioning_len, _ in enumerate(words, start=1):
        conditioning_seq = ' '.join(words[0:conditioning_len])
        print('='*32+'\r',f'{conditioning_len}:', speak(conditioning_seq))
In [ ]:
experiment('AI engineering demands a rigorous examination of every variable involved.')

Setting `pad_token_id` to 50256 (first `eos_token_id`) to generate sequence
Setting `pad_token_id` to 50256 (first `eos_token_id`) to generate sequence

================================
 1: AI and a few others, who are also involved in the project.
A lot of what we do here is not just about making money; it's trying to make sure that you get things done."<|endoftext|>

Setting `pad_token_id` to 50256 (first `eos_token_id`) to generate sequence

================================
 2: AI engineering, and a number of other projects.
It is not clear whether the company will take on any additional commitments in its current financial year or if it has already begun to pay down debt with interest rates that are likely lower than those currently scheduled

Setting `pad_token_id` to 50256 (first `eos_token_id`) to generate sequence

================================
 3: AI engineering demands a large number of workers, and so the problem is that there are few people who can do this job. It's not easy to get hired for it," says Bhattacharya Rao (retd), an engineer from New Delhi

Setting `pad_token_id` to 50256 (first `eos_token_id`) to generate sequence

================================
 4: AI engineering demands a new way of thinking about the future.


<|endoftext|>

Setting `pad_token_id` to 50256 (first `eos_token_id`) to generate sequence

================================
 5: AI engineering demands a rigorous, scientific approach to building the next generation of autonomous vehicles. The company's new robot-based prototype is designed to be able "to drive itself on its own," says Richard DeGroot at Carnegie Mellon University in Pittsburgh and

Setting `pad_token_id` to 50256 (first `eos_token_id`) to generate sequence

================================
 6: AI engineering demands a rigorous examination of the system, and it is essential that we provide an adequate background on its development.
The company has been in business since 1985 as part-owner/operator for over 20 years (in addition to being one or

Setting `pad_token_id` to 50256 (first `eos_token_id`) to generate sequence

================================
 7: AI engineering demands a rigorous examination of the technical aspects and use cases. The objective is to assess whether there are any significant problems with each project that need addressing in order for it be considered as an appropriate solution, or if they could have been avoided altogether

Setting `pad_token_id` to 50256 (first `eos_token_id`) to generate sequence

================================
 8: AI engineering demands a rigorous examination of every aspect, including the design and construction process. The work is done by experts from several disciplines: engineers with experience in designing large scale applications; computer scientists who are involved primarily to understand systems architecture such as hardware acceleration

Setting `pad_token_id` to 50256 (first `eos_token_id`) to generate sequence

 9: AI engineering demands a rigorous examination of every variable in the design and construction process, from structural details to materials. The research team has been working on this for years with some success since its inception by using multiple different techniques including high-speed mechanical analysis (
 10: AI engineering demands a rigorous examination of every variable involved. It is important to note that the only way for us (or any other person) or anyone else in our team, who has ever been trained by someone with this kind background and experience on their
In [ ]:
experiment('Old McDonald had a farm')

Setting `pad_token_id` to 50256 (first `eos_token_id`) to generate sequence
Setting `pad_token_id` to 50256 (first `eos_token_id`) to generate sequence

================================
 1: Old, a man who lives in the neighborhood and has been living there for more than three years.


 (Photo: Facebook) Story Highlights The house was on fire as of Tuesday morning; it's now down to one story at Stoney Creek

Setting `pad_token_id` to 50256 (first `eos_token_id`) to generate sequence

================================
 2: Old McDonald's is still in business. The chain has closed its doors to customers since May, and it will be shuttered by next year after a court battle over the company buying out some of their stores for $2 billion last month.




Setting `pad_token_id` to 50256 (first `eos_token_id`) to generate sequence

================================
 3: Old McDonald had to go, and I didn't know what else.
The next morning at the hospital was a big surprise: The first thing he saw when they arrived on scene were his eyes! They looked like black holes in space – with only

Setting `pad_token_id` to 50256 (first `eos_token_id`) to generate sequence

 4: Old McDonald had a good run, but it was his last chance to get on the field. He got hurt and missed all of 2011 with an ankle injury that cost him time at times in 2012 before he played for Cleveland's second team this year as
 5: Old McDonald had a farm in North Carolina. He was an avid farmer and used to bring his chickens from the fields down, so he always brought them home with him for dinner or just when we were on our way back out of town," she said

Conclusions¶

Being a random machine, I cannot precisely predict the transformer's output. However, the next generated token has high mutual information with previous tokens. If I exercise more control over the previous tokens, then I exercise control ove the following tokens. This is control by narrowing options, as reasonable conclusions decrease in set size with increase priors. Finally, this is comparable to the behavior exhibited in intelligent beings. Although internal dynamics are usually unknown, their predicted behavior is further refined with time. Like gpt-2, longtime friends can even model each other's sentences!