Recently, I used node.js and tensorflow.js to train an LSTM (Long Short Term Memory Network) on the life works of Shakespeare. I documented it in a youtube video here Needless to say, it generated some pretty interesting stuff. Below, I have text samples from its training - as it learned to write english words, construct scripts, and even produce something bordering on slightly comprehensible.
My LSTM had 3 layers of 512 cells, and I let it see ten characters into the past. It generated the output text character-by-character, so it had to learn english words all on its own.
Part 1
Epoch 1:
After a single epoch of training, the LSTM has figured out, well, not much. It knos that characters are grouped together, and are occasionally separated by spaces:
RnDj.Rbecdti o-it oenzn t
rhittlt ,ete a oiet , a sottt ahitetoa te n n uio r struuoul l in
Epoch 5:
By epoch five, it has picked up on the idea of words, and has begun to use newlines effectively. The words are still nonsensical.
F.$k
ahoetot o rt
t e e tn h
oa md oe i
eta
s e p a t
nm e ehsic ds e oe oer d e tu de
Epoch 9:
By epoch 9, the neural net produces the most common vowel sounds, most of which are pronounceable.
vght
ste lo uat rie hoe en ha ae tu uare oo eo oe eio te ott oi hhe te hon. Cuo to ae tor ha whn bw
Epoch 10:
Most of what it generates in epoch 10 is nonsense, but it generates the first remotely english-looking sentence:
I thate hove moitee ou oa totte
Epoch 15:
By epoch 15, the AI is doing much better. It is producing a few english-ish words:
X3A' ants rios we and yon heme on the bat in the pomite to renroe and his thered af the buth weth a
Epoch 20:
By epoch 20, the AI has learned the "script" structure of Shakespeare, and its improvement is quite significant. Even more english words appear:
COMINIUS:
Whe sons on thes fon on the bares
Than sot the comshes thes thon on ann he the thas os than to sast he dan theirs,
And anod.
COMINIUS:
The gowes the snades the ols with the fegins,
The getes burse the saurs bemoses thoy your than to son! Af the sorst me bud af you the paress.
MARsI:
MU thes than the an surn core these the secenss than they bathing hat the sirs that hears, and bures he as wert the peerss,
Ant nor then souts that wall the comtertet that wither.
Epoch 50:
After Epoch 20, improvement is gradual. However, by Epoch 50, the majority of words it is producing are english, and it has picked up on some grammatical structures:
CLARENCER:
IURo:
I will no with the scrace his is the hore the stand his noble had enery the is Lore and he shall your gellow, he will 't: and you this are make think he wall bear him in my loved with his lrester the with his some.
CORIOLANUS:
Hear hand this a must that that that should have be him the made his mide and, and partion to the house that be lord with the would speak the world that is the generals and say besent me that I think me that are for the people
We have we is chacter this wifans,
A peace the should be mine stat with the meself the say the may your betord the might him that her in my with all the consareit.
CORIOLANUS:
Come, I know that his of here deserving, and his thou hath to cheass, and we did helcome be and made this noble dear the mare the wreed the mes the wide
And I think are the faunterang to must, and that no more the stard that he hast the shall me shall shall not be us mone that that this gase that that shall make I have me bear me maked his
Epoch 100:
By epoch 100, it has learned many english words, and the script it generates might look like Shakespeare at a glance.
MENENIUS:
Where is a matter; the people
Make the senator:
Whose the gods mother's tread the sons to the sourt to give his poppetions and the people, your solders the state the canter him be your wife the ploper to reash of him intent to should not so home and surperity him to might with him be the forter the come.
Second Conspirator:
And he was sach a soldier.
SICINIUS:
What, if the man if you the son,
The follows
fierte the gods to the people natery frame his reperity and the gods the gods the possed him voice and the markes and the cause to his fierter of more is Rome,
And, I do say the people the can son the state
As the condemned to be so the mother you
Part 2
After training the LSTM with a memory length of ten characters, I expanded its memory to twenty characters and trained it again. The results weren't too different. So I expanded the memory to fifty characters, and continued trainig the model. The scripts it generated were significantly better:
Epoch 1:
At first, there is no noticeable difference - you might noticed it has learned bigger words though - like counterfeit:
Jraarfsaoft me was to sorrow not stay and not such as it
sounce:
The weeping met, the counterfeit an
Epoch 10:
By Epoch 10, the script is looking a lot better. With its bigger memory, the LSTM can write shorter lines, keep some consistent themes, and even keep track of who is in the scene.
It also generates some hilarious fake words - like "voomest".
FARCIUNIA:
I would they shall have now let me it is not the weard,
She would have not the stand on thee of the markys the preverent is your night.
CAREIA:
What are with will the friends, stay send our stranger'd the matter.
VALERIA:
Then they shall have been you quittle. I am a more with his dear forth
One our royal country on his grace of would proy,
There very in his dear, beoo the sun and the collery is his country.
CAREIA:
I will poor the voomest the warrant the house.
MARCIUS:
I am not, when it is not we present to die all I have been keep him forth so slive the lies in more them bear uson stir.
Second Senator:
I have for the stay madam, and I will be my son sho
Epoch 30:
By Epoch 30, the script really sounds like a Shakespeare one. Virtually all of the words are English, and the LSTM impressively remembers that CORIOLANUS is in the scene, long after he has stopped talking.
Second Citizen:
Then soul, which make you may a peace my death with your son the stebict them.
MENENIUS:
Which common the office the hols, and then not in his horse are the bold with the people, though you would leaves
be sprong him to be at the regarve the desire their better his hand to the people.
CORIOLANUS:
Ay, the then the part them: and first the sontent,
The sun with our worthy he shall should the consul.
First Senator:
I shall be man, the matters to the country may be the thing man be do their our with thee.
VOLUMNIA:
I have should you come, good mark us the man, whom
That is the common to the consul men't to be with not a sport.
CORIOLANUS:
O, it is not they shall as he hath a man's blood
With the proud. Where he content him stoble to gentles to be the sone to the
Epoch 50:
While the generated text here is similar to that of Epoch 30, the LSTM has learned how to generate monologues, in addition to quick lines of dialogue.
COMINIUS:
Come, the bear come, the tribunes
I'ld be not be vanience to the common home the people.
CORIOLANUS:
Must of the show'd them.
SICINIUS:
My some house, what some and feel you me the common and to your
some own the meating, therefore a bear of thee be so head to breast
Than you see the war be so man a bear them
The dead to hear me see the some consider,
And the proved deven by the streech a common,
Come the mother whose mother?
CORIOLANUS:
SICINIUS:
I have speak be so shill for whose head,
The prithee see him they be not some with
the sent to take a speak them them my shall
I have precend so that you shall be continues
The voices with the people, they come,
And but he that the power hath we me here the stroke my man the people
That whoo's stand of the words, and now
The noble house thee with me this hither.
CORIOLANUS:
I know the selt him the strength.
CORIOLANUS:
I would be consul
The one but a come, and
Epoch 100:
After Epoch 50, the performance of the LSTM leveled off. The "Murderer" script it generated is here for closure (and it is also a pretty decent script - ha ha):
First Murderer:
I have done to him, then the man have much in the counsel,
When thou shalt not sent to the queen,
And the curse to have down the angel and the world
see the sun of the reasons so sent to see the remompy.
First Murderer:
Mary man!
CLARENCE:
Alas the mother to the air,
I have done to the death of shines, and his lawful.
First Murderer:
I will not shall not the business with a warrant the man
The such a dayn from him his absence for thy disconeen and offence
To the presence and still the reason to give
The world of did of heart me of men to the world
To see you in the hatchens to the stabbed for my world of clarge.
First Murderer:
To friends to meaning that thou art thou hadst ender peace.
Second Murderer:
So will not have saw to medness so he honour
To entertain with me on the sword, and the heaven.
One Last Script...
One particularly amusing script that was generated by the LSTM talks about "the people"... a lot:
MENENIUS:
Sir, comes when the people!
MENENIUS:
Do you a persons
That have not sword in the people end
That shall for the people.
COMINIUS:
What cause the people vail to the people
What you will be consul, the people.
SICINIUS:
Hear me death him to make the people!
MENENIUS:
No, no, be consul.
COMINIUS:
Nay, you shall therefore, sir, the people,
Which you should the one that the gracing for the wise the sun
Of the march of the people in the people.
COMINIUS:
You are to the people,
When he will the varture of the people his senate
That is the consul of your best
And sure of the honours they worthy little that is the death.
MENENIUS:
Therefore so are in the noble worthy request
The ground was take the people though the lattle and bench
That you shall to make the consent of the voices
What shall many in their proud to cold prosperous
True his found of the people,
Be content of the people.
And I'll leave you with that amazing work of literature. I hope you enjoyed looking at the hilarious results of my LSTM. Thanks for reading!
Top comments (11)
I have always tried to implement something like this in
javascriptbecause I don't likepythonsyntax much but I have never been able to do so. Are you thinking of posting the source code on Github?Yes, the source code is here: gist.github.com/N8python/93139a670...
Want some fun? Train it off Finnegan's Wake.
This is the initial result with 10 characters of memory... more training is needed:
It's perfect.
It does a bit better with 50 characters of memory:
Here's a github repo of all the Finnigan's Wake stuff it generated:
github.com/N8python/finnigansWakeGen
I just looked at Finnegan's Wake... it is even less comprehensible than Shakespeare. I probably will.
I'm beginning to get very interested in NLP haha
Text generating models are very fun! I'd totally recommend you get into them!
This is awesome!
Thanks!