Improving Our Text Generator

If we repeat our earlier experiment and generate some random text with our improved word selection, we might get results similar to the following:

console command you need to select
console kernel class you may wish
console commands provide any changes to

Our generated text is starting to feel better, but we can still make improvements. One way we can do this is to use more than one word when constructing our dictionary. If we think about this idea, it makes sense.

Our text generation algorithm generates subsequent words based on how often they appeared in the original text. Presumably, this original text was written by a human, so they ultimately determined the frequency of those words. If we increase the number of words we reuse when predicting our text, more human influence will come through in the produced text. This, however, only works to a certain extent.

If we reuse too many words from the original text, the generated text will become less and less original and just appear as copy-pasted chunks from the original text.

We can update our existing implementation to support multiword sequences like so: