DEV Community

Cover image for Raku: other implementations, archeology and glorious details
Tib
Tib

Posted on

Raku: other implementations, archeology and glorious details

Raku past implementations

(This is a translated/improved version of the french article Raku en 2020 from the same author)

Thank you to Elizabeth Mattijsen who helped to proofread and with advices.

Table Of Contents

The history of Raku

The history of Raku interpreter and virtual machine is really not straightforward... If you've heard about "Parrot", "PIR", "PASM", "Pugs", "Rakudo", "MoarVM", "NQP", "Rakudo on JVM", "Rakudo on Parrot", "Mildew", "Ponie" but you're lost with all these implementations then this blog post is made for you!

I made researches 🔍 to clarify all historical and technical (!) details and gathered all the results here, with several homemade and original sketches to make it crystal clear for you.

Lot of original sketches

The birth of Raku

Raku is born on 18 July 2000 from a famous event where John Orwant threw coffee mugs against a wall during a meeting.

The renaming process : Perl 6 → Raku

A programming language that changes its name... it does not happens often!

Recently, Raku celebrated its second year (or its 21 years). Here is how the renaming happened:

  1. Long time ago (around 2005), people discussed about an alias but it never led to anything
  2. while answering a question, Larry Wall proposed an alias
  3. alias requested by some people from Perl 6 community
  4. and validated by Larry
  5. later, one member of the core team (Elizabeth Mattijsen) opened an "issue" on the official GitHub of Raku to ask for a renaming: Issue from Elizabeth
  6. Elizabeth then opened a merge request towards the Git official repository of Raku (requesting maintainers to vote): Pull Request from Elizabeth
  7. people voted for (or abstained), then came the "blessing" from Larry Wall: Comment from Larry Wall
  8. renaming done, please welcome Raku!

The renaming process in links:

Compilers, frameworks and virtual machines

In this second part, I will detail the past implementations of the Raku language. Get ready, acronyms are coming!

Scrabble letter

Which "official" implementation?

Short answer: Rakudo and MoarVM.

(the details about Rakudo/MoarVM are in the previous blog post of the serie)

Initially it was decided that no implementation would be defined as "official" for Raku but rather that "Raku is something that passes the official test suite". In fact, since 2012 there is an official de facto implementation which is Rakudo + MoarVM.

Before that, there was the Parrot framework (2002-2012) as well as Pugs (2005-2007) and various other compilers and runtime environments (Mildew, Niecza, Yapsi…). This technical and historical part will be discussed at the end of this dispatch. But let's go back to talk about the beginnings.

The very first announcement of the creation of Raku dates back to July 19, 2000. Larry Wall's idea was to make it an ultra-community project right down to the language design, so he would start by asking to contribute to the specifications. It resulted in 361 RFCs, , from which Larry Wall produced apocalypses, which where then explained by Damian Conway in the exegesis , and then later further detailed in the synopses.

RFC Apocalypses Synopses

Links for historians or the curious:

Parrot

Created in 2002, the framework and the Parrot virtual machine have long been central to Raku's implementation (this is no longer the case today). Parrot was born from an April Fool's Day announcing the unification of Perl and Python and even the upcoming release of a book on the subject:

Announcement of the unification of Perl and Python

Why build a new VM?

The JVM was studied in 2001, but considered poorly suited to dynamic languages ​​(it still wasn't in 2010, quid of 2021?).

Compilation

Parrot offers much more than a virtual machine, Parrot is a language creation, compilation and execution framework. This framework makes it possible to build high level languages ​​(HLL) easily and quickly. It has been put into practice by Rakudo for a long time.

PCT

PCT stands for Parrot Compiler Tools. Its purpose is to enable the creation of a compiler and a high level language execution environment (HLL).

PCT

PCT is written in PIR (see assembly section below).

PAST

PAST stands for Parrot Abstract Syntax Tree: internal representation of a syntax tree.

HLL

HLL stands for High Level Language.

PGE

Originally called P6GE, PGE stands for Perl Grammar Engine. Based on HLLCompiler. A PGE rule looks like this:

rule term   { <number> | \( <expr> \) }
rule number { \d+ }
rule expr   { <term> ( '+' <term> )* }
Enter fullscreen mode Exit fullscreen mode

PGE was used by Pugs, another Raku compiler (see below).

TGE

TGE stands for Tree Grammar Engine.

PACT

PACT stands for Parrot Alternate Compiler Toolkit. An alternative to PCT because PCT was considered too limited.

Assembly

PASM

PASM stands for Parrot assembly language (PASM). The low level code ready to be converted and executed by Parrot VM. PASM code looks like this:

  set     I1, 1
REDO:
  gt      I1, 10, END
  print   I1
  print   " "
  inc     I1
  branch  REDO
END:
  print "\n"
  end
Enter fullscreen mode Exit fullscreen mode

References:

IMCC

IMCC stands for Intermediate Code Compiler. It is an alternative tool for creating and running Parrot bytecode

IMCC brings its own language, commonly known as Parrot Intermediate Language (PIR). IMCC embeds the Parrot runtime environment, so IMCC can compile from PIR to PASM, then from PASM to PBC, then execute this bytecode. IMCC can also do optimizations, although it doesn't by default.

PIR

First called IMC, PIR is an overlay of PASM without being a language of high level. Files containing PIR code have the suffix ".imc"

This is what PIR looks like:

.sub loopy
        .local int counter
        counter = 0
LOOP:   if counter > 10 goto DONE
        print counter
        print " "
        inc counter
        goto LOOP
DONE:
        print "\n"
        end
.end
Enter fullscreen mode Exit fullscreen mode

You can learn more from this IMCC intro or this article on how to write PIR. See here PIR examples or even the detailed documentation of FR mongueurs on PIR.

Execution

NCI

NCI stands for Native Call Interface.

PMC

PMC stands for PolyMorphic Container or Parrot Magic Cookies (the way of representing data in the virtual machine).

PBC

PBC stands for Parrot Byte Code

The Parrot VM

The Parrot VM is a registry-based virtual machine, which is not the "standard" (eg JVM).

Here are some features of the Parrot VM:

  • able to manage static and dynamic languages
  • Copy On Write
  • manages continuations and closures
  • can be embedded in C code
  • PolyMorphic Container (flexible type storage)

It's easy to include Parrot in C code or call C in Parrot (Native Call Interface).

Raku implementation based on Parrot (2002-2012)

PCT IMCC Parrot VM

Or with a merged IMCC and Parrot stage (later because IMCC contains Parrot):
PCT IMCC

Below is an even more detailed/technical diagram (taken from Parrot documentation):
How Parrot works

Many details are available in the Parrot Design Document

Problems with Parrot

If Parrot was initially at the center of Raku, this has changed over time:

  • 2008: "While Parrot is the most solid solution to deploy Perl 6 in the long term, on the other hand, the challenges Parrot has accepted had proven to consume more time and resources than previously expected." (Perl foundation news)
  • 2013: "The latest news is that Parrot Virtual Machine is no longer the only one enjoying Rakudo's exclusivity." (all about Perl6)
  • communication problems
  • Parrot gradually moved away from Perl 6 (by becoming more generalist)

Little by little, the Rakudo compiler has freed itself from Parrot and is now targeting a new virtual machine.

Pugs (2005-2007)

From here we start archeology, the real one 🔍

In his short existence, Pugs explored many architectural choices. Pugs refers to the interpreter, compiler, runtime environment, and test suite. The compiler can compile to Haskell, JavaScript, Perl 5, or PIR (for Parrot, if you remember).

Pugs compilation flow

Pugs is written in Haskell and falls dormant after 2007.

Compilation

Here are the concepts of builds around the Pugs implementation.

PCR

PCR stands for Pugs Compiler Rules. This is a Perl 5 implementation of Raku's regular expression engine. PCR replaced PGE in Pugs.

LREP

LREP evolved into Pugs::Compiler, then, later, MiniPerl6.

Pugs::Compiler

Pugs::Compiler is a set of Perl 5 modules for compiling Raku.

MiniPerl6

Was part of Pugs but in a separate directory. MiniPerl6 became KindaPerl6 then Perlito6.

KindaPerl6

KindaPerl6 or KP6 is a Raku implementation of the Raku grammar with a bootstrap in Perl 5. KindaPerl6 is built from MiniPerl6.

Execution

PIL

PIL stands for Pugs Intermediate Language. PIL is not a human readable format but rather an abstract syntax tree internal to Pugs.

PIL-Run

Pugs PIL Perl

PIL-Run was built on top of a set of Perl 5 modules that read and run PIL. PIL-Run was Pugs' back-end in Perl 5... at the start.

Pugs based implementations

Pugs + various back‑ends

"Pugs compiles and executes"
Pugs compile and run

"Pugs compiles for Parrot (PIR)"
Pugs PIR

The compilation to PIR is done in two stages, small zoom on it:
Pugs PIL PIR

"Pugs compiles (transpiles?) To JavaScript"
Pugs JS

"Perl compiles (transpiles?) To Perl 5"
Pugs Perl 5

v6

v6-pugs then v6-alpha and now v6.pm is a complete rewrite of Pugs, using a mix of Raku (Pugs::Compiler and associated modules) and Perl 5. It reuses code from the runtime environment of PIL-Run and "half of the CPAN" :grining:

Now v6 is released in Perlito6. v6.pm is the front-end for Perlito and Pugs in Perl 5.

Perlito

The Perlito project is led by Flavio S. Glock. The Perlito project actually contains multiple compilers for multiple languages ​​in multiple languages ​​targeting various back‑ends:

  • compile Perl 5 to Java source code
  • run Perl5 directly in the JVM
  • compile Perl 5 to JavaScript source code, run Perl5 directly in the browser or Node.js
  • compile Raku to JavaScript source code, run Perl6 directly in the browser or Node.js
  • compile from Perl 5 to Perl 5
  • compile from Raku to Perl 5
  • compile from Raku to Python 2.6
  • compile from Perl 5 to Perl 6 (in progress)
  • compile from Raku to Ruby 1.9 (in progress)
  • compile from Raku to Go (in progress)
  • compile from Raku to Common Lisp (SBCL) (in progress)

One thing he can't do is translate from Perl 5 to Raku.

MiniPerl6 and KindaPerl6 form Perlito6. Perlito5 is a port of Perlito6 to Perl5. It is "Perl 5 implemented in Perl 5". Perlito5 itself is written in Perl. Perlito6, on the other hand, is written in Raku.

Perlito only implements a subset of Raku called the "useful subset" which:

  • has no list context
  • has no inheritance
  • has no laziness
  • has closures but no coroutines or continuations
  • has no multis.

This "useful subset" is kind of similar to NQP (see previous blog post for details about NQP).

Other compilers, runtime environments and virtual machines

In this part, I will continue my tour of initiatives around compilers, runtime environments, and legacy virtual machines.

STD and viv

STD and viv are works of Larry Wall.

We find in STD the grammar of Raku… written in Raku (which did not have a compiler at the time! 😏 It's the beginning of the bootstrapping problems 😀

VIV or literally in Roman numerals "VI → V" (and therefore "6 to 5") was a module capable of converting from Perl 6 to Perl 5.

SMOP

First called YAP6, SMOP stands for Simple Meta Object Programming

SMOP is a compiler and runtime environment written in C for Raku, but we sometimes say SMOP to refer only to the runtime environment (which may be targeted by other compilers).

SMOP YAP6

SMOP is not a virtual machine but a runtime environment that resembles that of Perl 5, while having functionality to manage Raku.

Mildew

Mildew was an STD to SMOP compiler.

Mildew SMOP

It directly uses the STD grammar.

Elf

Elf was a compiler written in Raku with a Ruby grammar. It could issue Perl 5 or Lisp. It was never completed. It was expected that Elf could target SMOP.

Ponie

Ponie stands for Perl On a New Internal Engine. This involved running Perl 5 in the Parrot virtual machine. Stopped in 2006.

Announcement Jesse Vincent

Ponie

Punie

Punie (direct reference to Ponie) was a Perl 1 compiler to Parrot.

Yapsi

Yapsi for Yet Another Perl Six Implementation is an implementation of the compiler and its Raku runtime environments and virtual machines in Raku.

Yapsi

SIC means (S??) Instruction Code, this is Yapsi's own byte code.

Niecza

Niecza is a Raku compiler written in C# that targets CLR (Mono).

Niecza Mono

Topaz

Topaz was a Perl 5 core rewrite in C++ started before Perl 6, finally abandoned.

Sapphire

Sapphire was another rewrite of Perl 5 internals in September 2000, shortly after the announcement of Perl 6. More experimentation than something else.

Phew! It was the last one, we're done. 😀

Conclusion

There are many interesting concepts to be found in Raku as well as in its history and the history of its implementations (although most of them are part of a vanished world, they well deserved a little closer look).

I hope that this long blog post has been able to lift the fog around the twists and turns of Raku's implementations, its history and to highlight the concepts of the language.

Discussion (0)