You have to set top_p to 1 and temperature to 0.8. This was hard-won experience, and no one seems to believe me when I say it. :)

Try it again; I’ll bet you a dollar you’ll be more impressed. Defaults really matter.

temperature 1.0 is the worst possible default. It’s no surprise it returns gibberish; the sweet spot is between 0.7 and 0.8. When I first got into ML, this was a big surprise. But it’s the difference between flying a plane with flaps all the way down (a crazy default) and normal settings (0.8 or so).

Samples with the settings metioned above (tl;dr - results are significantly better with sillysaurusx's suggestions):

Using the default promp:

-----RESULT (DEFAULT SETTINGS)-----

In a shocking finding, scientist discovered a herd of unicorns living in a remote, previously unexplored valley, in the Andes Mountains. Even more surprising to the researchers was the fact that the unicorns spoke perfect English. The scientist confirmed his findings by being at the farm.

Wild dingo

The wild dog, also known as a canid is the feral domestic dog subspecies with the wild ancestor of the domestic dog. The Australian Dingo is native to Australia, and...

// it continues to list animals and small facts about them

-----RESULT (SILLYSAURUSX'S SETTINGS)-----

In a shocking finding, scientist discovered a herd of unicorns living in a remote, previously unexplored valley, in the Andes Mountains. Even more surprising to the researchers was the fact that the unicorns spoke perfect English.

The surprising discovery of English-speaking unicorns was made by an international team of scientists led by Dr. Steven Wise, from the University of California, Irvine (UCI). According to the article, they stumbled upon the unexpected herd in the remote Aucayacu Valley, in Peru. During their field expedition of the location, the team discovered a herd of five unicorns.

Wise told Scientific American that he was fascinated by the discovery and the questions it posed. He said that the unicorns had no problem understanding basic questions, such as which way to go to a certain location, or what time it was. They were also able to understand and respond to basic commands such as “hello,” “good-bye” and “trot.”

The scientists explained that unicorns are unicorns because they possess a pair of horns. The horns at the front of their heads are not covered in hair, but rather are made of a material that is hard and smooth.

Dr. Wise said that unicorns have a different kind of horn than any other living mammal. He added that although people often refer to the pair of horns as “horns”, they are actually not.

On the other hand its infinitely more Open than anything from OpenAI :)

Anecdotally from playing around with both, GPT-J seems to struggle compared to GPT-3 when it comes to regurgitating remembered details from the prompt. For example when trying to generate short stories I've found that GPT-J will randomly genderswap characters from sentence to sentence, which I never had happen with GPT-3.