Announcing the Pen programming language v0.4

The Pen programming language is a new parallel, concurrent, statically typed, functional programming language. I'm excited to announce its v0.4 release here!

I've been working on this programming language project for almost a year. Recently, we've released its new version with new syntax constructs, standard packages, Rust FFI (Foreign Function Interface) and complementary tools like formatter and documentation generator.

In this post, I would like to introduce the Pen programming language, and describe its current status and new features included in the latest release.

Install

To try out he Pen, you can use Homebrew to install it on Linux, macOS, and WSL on Windows.

brew install pen-lang/pen/pen

For more information on how to write and build programs in Pen, see Getting started.

Introduction

The Pen programming language is a general-purpose, strongly and statically typed, functional programming language with automatic precise memory management based on ownerships. And it's also made for scalable software development. It aims to make development of large-scale applications easy and scalable by focusing on software maintainability and portability. Programs written in Pen should be simple, testable, and flexible against changes in the real world.

The language that influenced Pen the most is Go. You can also call Pen a functional descendant of Go with focus on application programming. Pen shares the same goal of Go like simplicity, reliability, efficiency, etc. Though, Pen explicitly excludes system programming from its domain to pursue further simplicity, portability, and developer productivity.

Current status

Currently, the language is at the stage of syntax finalization for v1. The last missing piece of the syntax constructs is generic built-in functions similar to Go's. They are expected to be called directly and behave similarly to built-in operators rather than normal functions. By introducing them, Pen can implement more built-in operations without increasing the language's complexity due to addition of syntax constructs. Also, that makes Pen more flexible as a language against requirement changes in the future.

Changes in v0.4

Since a v0.4 release of the language is dependent on and had been blocked by a release of LLVM 14, it includes many new features in the language and complementary tools. Here are major ones of them.

The Perceus reference counting GC

One of the biggest changes in a compiler is implementation of the Perceus reference counting GC (Garbage Collection.) It is one of the state-of-the-art GC algorithms for functional programming languages. Compared with non-reference-counting GCs which are more popular in other functional programming languages, such as OCaml and Haskell, its implementation is relatively simple even though it performs comparably to them as described in the paper. Also, adoption of such a reference counting GC makes programs written in Pen more portable even into WASM without any modification. This is because other GC methods commonly require full view of memory to track which memory locations are still reachable. Checking the entire memory including stacks at runtime requires target-specific logic or is impossible on some targets like WASM. I described more details on implementation and performance benchmarks of the Perceus reference counting algorithm in another post.

Rust FFI

A pen-ffi crate is a Rust FFI library for Pen. Thanks to both languages' semantics based on ownerships, they can interoperate with each other very easily. The new standard packages included in this release like Http and Sql packages are actually simple wrappers of third-party crates in Rust. Using the Rust FFI library, you can also write your own packages in Pen while utilizing existing resources written in Rust and benefiting from its growing ecosystem.

go expression

It's still experimental but we've introduced the first piece of parallel computation primitives, go expression. By using the go expression which originates from Go, you can delegate heavy computation, slow I/O, or any other computation you want to perform concurrently to other execution contexts like threads.

future = go \() number {
  # Some heavy computation...
}

Formatter

Pen is now equipped with its official formatter command, pen format. This command formats each source file or all in a package at once. It can also receive source from standard input and emit formatted one to standard output so that it integrates with editors and IDEs easily.

Similarly to Go's gofmt and differently from some other formatters like rustfmt in Rust, the source formatter of Pen does not define any canonical form of source codes given the same sequences of tokens. But it rather tries to align indents, and add and remove spaces and newlines given hints extracted from the original source codes so that developers can still control their source codes to make them look readable in their own contexts. For more information, see pen format --help.

Documentation generator

The new release also includes a new pen document command that generates documentation files of packages in Markdown. To generate the documentation, you can simply run pen document with some options, such as a package name and URL, in a package directory. For more information, see pen document --help.

Http and Sql standard packages

In addition to new functions and types in the Core and Os standard packages, we've added Http and Sql standard packages. As their names suggest, the Http package provides HTTP server and client logic, and the Sql package provides Sql client logic.

Because those packages depend on the third-party crates (hyper and sqlx respectively) in Rust, they are currently planned not to be included the default installation bundle of the language. But they are likely to be separated into different Git repositories.

LLVM upgrade to 14

As I mentioned first, LLVM was upgraded to 14 in the new release finally, which fixed many bugs including the ones related to tail call optimization. Examples of fixed bugs are:

• Certain operations on the built-in map type led to segmentation faults.
• pen test command failed on macOS when running multiple tests.
• macOS with M1 chips could not run binaries compiled by the compiler.

Now, most programs written in Pen should work also on macOS with M1 chips as well as on x86-64 chips. There is a plan to support macOS on M1 chips as the first tier platform although we can't guarantee that Pen works properly there due to limitation of our CI infrastructure currently.

What's next?

There are quite a few features planned for the next version of Pen including:

• Generic built-in functions
• Proper implementation of the C calling convention for FFI
• More functionalities in standard packages

The C calling convention is already implemented. However, the current naive implementation is inefficient and sometimes requires heap allocations. By implementing it properly, Pen can call C/Rust functions without unnecessary overheads.

Conclusion

We've hit the great v0.4 milestone of Pen! It contains new syntax, standard packages, and other complementary functionalities. Those features include:

• The state-of-the-art reference counting GC
• Parallel computation primitive (go expression)
• Http and Sql standard packages

Thank you for reading this article! And if you are interested in Pen, please install, try it out, and give some feedback!