Will McGugan is among the most well-known Python developers. He's the author of Rich, a library for formatting output in the terminal. It's used, among others, by pip, and has more than 40K stars on GitHub. In 2021, Will started building Textual, a TUI (text user interface) framework based on Rich. At the end of the year, he founded the company Textualize.
In this conversation, we discussed both technical and non-technical parts of this journey. He tells about the transitions between various phases: from side project to a popular open source library to an open core business. He also shares his thoughts on designing APIs, maintainability, and speeding up your Python code.
Building Rich in Public
Reka: Let's go back in time. How did Rich start? As I understand, it wasn't a linear story.
Will: Well, it's kind of a long story. It was in the back of my head for a few years. I was working on another project where I wrote some things to the terminal and hacked together a class. It did various formattings with ANSI escape sequences. Making text bold or magenta, wrap it.
It was a bit messy, but I realized there's potential to extract that to a library. So about three years ago, I started doing that. Based on the original code that I had written several years before, I built something. A bit tidier, a bit more elegant than that original class.
And it came together fairly quickly. There was a first version in just a couple of months of development. And it kinda took off very quickly. I pushed it in the usual channels, and it got a lot of stars.
People seemed to really appreciate it. Various formatting libraries existed that time, but they didn't work very well together. So you could colorize text but couldn't wrap it. You could create a table but couldn't colorize the text inside it.
So I figured I'd create a system where you can build all these things and have them nicely integrated. So that was pretty much the driving force behind Rich.
R: How did you begin creating this library? Did you extract code function by function from this old application?
W: I didn't use any code from that application. I only took the core idea from there. This core idea has two main parts. A Console object, which represents the output channel. And a protocol, which I can add to any object, to define how the output for this specific object looks like. So to print a table, you can construct the table, and then you can call console.print(table).
So everything follows this same pattern. I think that made a lot of sense to people. It just kind of clicked, because it's similar to printing. You don't call string.print(), you call print(string).
This pattern also enabled me to build various kinds of things which could be printed. Tables, panels, rich text, and progress bars. But the API is the same for all of them: You construct the object, and the Console can print it.
R: I definitely agree that the Rich API is very consistent and discoverable. What else did go into this besides the consistent pattern?
W: Rich uses SemVer. And I've always been ready to bump the major version number to improve the API. So, it's in version 12.6.0 now. And each one of those major version bumps was because of a change in the API, just to make it a little bit more elegant.
I think developers are prepared to accept quite a lot to do something they wanna do. They are prepared to accept clumsy, awkward APIs, if it does something that they need.
But I've been determined to make this API elegant. So, I've iterated over it quite a bit.
R: Have you felt any drawbacks of these iterations? Complaints because of breaking changes?
W: Not really, because I was very upfront about it.
Well, actually, early on, there was such a story. I made a breaking change, and I didn't increase the major version number. And then I got complaints. And I was quite surprised that there were so many people using it. And they cared enough to complain about it.
But after that, I've been consistent: I increased the major version for any breaking change. And I documented what changed, and people have been happy with that.
Basically, what developers don't want is surprises. They don't want things to just stop working suddenly. Which I can understand. But you can document all these changes, show where it's broken. And maybe explain why you changed it. So, people are happy that they can upgrade at their convenience. And also, they can be comfortable because they know it won't just break at some point in the future, seemingly randomly.
R: Switching gears, how did you manage the logistics of a full-time job and a demanding side project?
W: I was a Python contractor and working from home, which allowed me a little bit of flexibility and spare time. So I did my day job and then maybe an hour for Rich at lunch and maybe a little bit in the evening before dinner.
And it keeps interesting. Because you got a day job, which pays the bills. But I often find my hobby projects more interesting. So, I found the time for it.
I mean, I wasn't that consistent. There were gaps. Sometimes, there was a month where I didn't do anything, because I thought I'd finished it. And then I got a number of people asking for a particular feature. And I thought that's a good idea. Then I spent the next few weeks kinda working quite heavily on it.
So, yeah, I just kind of fit it into the slack time. Where I'd done the day job and didn't have anything else to do.
R: You said that Rich took off quite quickly. How quick was that?
W: It was released two and a half years ago. Now we're above 40K stars. If you look at the star history, it's been almost a straight line. There have been a few bumps, mainly due to social media. But generally, the growth returns to the same level.
R: Rich was also your first project that you've been building in public. How do you like that experience?
W: That worked really well, partly because Rich is quite visual.
When I got something working, I was pleased and posted it on Twitter with a few comments. I did this just for my own amusement, really. But people started following along and I built up a bit of a following. And I'm sure that contributed to the popularity of Rich. Because people would share it.
Rather than waiting, say, a year until the API was very solid before I announced it, I announced it early on. And just kept releasing new content, whenever I had a new feature or fixed a bug. Or sometimes even when there was a fail, when something went wrong, I would post about that.
So people could follow along with the development. And it was beneficial to me. This helped my motivation. Knowing that there were people out there, who were genuinely interested in what we're doing.
Because development can be a very solitary activity. You just tend to be on your own, hacking away. It's nice to know that there are other people that care about the thing that you're working on.
R: What kind of reaction did you get? How much did the feedback influence it?
W: A fair bit. So in the beginning, I didn't have many followers on Twitter. So, it was just me posting something. But occasionally, I'd get a star or a like or a comment. But as it went on, I got more and more comments, more and more feedback.
And that did influence the direction that Rich went. People would ask for features. At several points, I thought, oh, it's finished now. But people would ask for something and I realized that the library would be better, if I implemented that. So as time went on, there was more and more feedback from people. And a lot of that feedback went into the library. So it did very much influence the direction that it took.
And it's quite a positive thing. If I was just building for myself, I might just build a very lean core set of features that I needed. But then its use would be limited to just myself or to people who are doing exactly what I'm doing. But by listening to feedback, I built something which was broadly useful.
Now, Rich has become kind of standard for doing something more sophisticated at terminals. And it's because of listening to all that feedback I've got for the last two years.
R: Do you have some specific examples in mind? Features that you didn't plan to add, but the community asked for?
W: Yeah, quite a lot of it. Progress bars are a good example. I had imagined that Rich would just purely be writing static content, nothing animated. But progress bars were requested quite a few times, and I looked into that.
I figured out how to do the mechanics, and I enjoyed the technical challenge. So I built the whole progress bar thing, even though I had no use for it. And now, it's one of the most loved features.
R: I definitely like the progress bar, as well, so thank you. 😀 Regarding the feedback, have you developed some strategies how to evaluate it? When to accept it and when to ignore it?
W: One individual feedback on its own probably wouldn't make that much an impression with me. When multiple people give the same feedback or request the same feature, that's a sign where I should focus my efforts on.
It was almost like polling the community. The most asked for feature is the feature that I would implement next.
It's quite a strange thing because when you start project like this, it's purely for your own amusement. And you're the boss, it's not like a day job. No one tells you how to write code. You write whatever you want and do it however you want. And when it gets more popular, something shifts. You start working for yourself and you start working for other people.
So rather than being your own boss, you've now got ten thousand bosses. But that makes it more useful for more people, and so it's beneficial.
R: Do I recall correctly that Textual also came from the input from the community?
W: Yeah. That's right. Some friends on Twitter took Rich and built a dashboard, which shows live events happening on GitHub. Pull requests and various things. They created a nice table and it had emoji icons. It was kind of cool. They were using Rich under the hood, but they've added the interactive stuff themselves.
And then I thought such apps could be really useful, but there wasn't a nice way of creating them. So that's when the idea of Textual came. Could I take Rich, add interactivity on top of it, and create an application framework?
And I started doing that again. Just as a hobby project. I had no thoughts of turning it into any kind of business. But after a few months, it kinda came together.
I had a strong tech demo. And I kinda convinced myself that, yeah, this is gonna work. That this could be the, like, the beginning of something, which could be very successful. Fingers crossed.
Photo by Textualize
From Side Project to a Company
R: When did this happen? When did you decide to focus on Textual?
W: Towards the end of 2021. I figured that I could take this application framework and build a web service around it. Which I could then sell as a service.
It was only the very beginning of an idea but I was prepared to take a year off. I was going to live on savings and implement this. And at the end of that year, I'd either have a business or I could get funding for it. In the worst case scenario, I would just have to go back to the day job.
That didn't last very long, because the opportunity for funding came along. And I took that, which is probably smarter than burning through a year of savings. So at the beginning of this year, we founded Textualize. Since then, we've been working on Textual and Rich full time.
R: What motivated your decision to create a separate project? Instead of adding all this new functionality to Rich, which was already a well-known brand that time.
W: There's quite a nice division of labor between the two libraries. Rich is more for static content, for writing stuff to the terminal. It has a few dynamic things, like the progress bars. But I figured it was better for Rich to just focus on static content, as it's reason for being.
Because it's already quite a large library. And if I added too much to it, it runs the risk of being too big. People would just look at it and think: Well, it does way more than I could possibly need. Plus, it would have a lot of other dependencies.
So, I figured that I would then separate it into two libraries. I would build something new for the dynamic, interactive stuff and that would have Rich as a dependency.
So, if you don't need a full application framework, you won't need to pull everything in. You can just use the bits that you need.
I think that was a good move, because it means that there's a nice division of labor. There's Textual for the dynamic stuff and Rich for the rendering.
R: You mentioned in an interview that Textual's first prototype was a bit messy and you cleaned it up afterwards. Can you tell more about that?
W: Sure. When I started Textual, it was kind of just exploring what could be done. So first, I just built a prototype using asyncio and Rich and see if it would work, first of all.
And a lot of that was built over a number of weekends where I was drinking lots of coffee. And it came together. But the code was not as tidy as I'd want it to be.
So when I got the funding, I could just purely focus on that. I could apply all the best practices and try to make the code as neat and tidy as possible and address some technical debt.
I could also be more ambitious. Because if it was like a hobby project, working on the evenings and weekends, I'd have to be quite pragmatic. Because there's only so much a single developer can do.
But once I got the funding, we can do a lot more because I've got employees to work on it with me. Very smart people and we can do some work in parallel.
So, we took the Textual source. We didn't restart. There was lots of good stuff in there. We just took the original tech demo, built on that, and used that as a platform for the features. And now, we've tidied it up. Most of the code, I'm quite proud of: It's quite neat and tidy.
And I encourage people to look inside the code. And now, it's a much better platform going forward. So we're gonna make lots of releases quite rapidly in the future.
R: Which release cadence do you strive for?
W: We released 0.2.0 recently, which had the CSS support. That was on a feature branch for well over six months. This is not how I like to work.
I like having rapid releases. Now, we're gonna aim for something like a week to two for each new release.
So that if you're using it, you can keep following the latest fashion. That means you can identify if there's any problem early on.
R: Why was CSS support a special case?
W: There was very little to build on. No library to do CSS that I could just import. First, I had to write a CSS parser. Then I had to write a CSS renderer. Then we had to integrate CSS all the way through the project. These all took a while.
Here and there, I ported back some fixes to the main branch. So that I wouldn't leave early adopters completely in the dark.
But once the CSS stuff was done, now we can just keep iterating. Which is a nicer approach for everyone involved.
R: How was the transition from a side project to a company?
W: It took a lot of adapting. I'd been sort of working freelance, from home for the previous twelve years. Now, I'm working in an office. And that was quite hard to adapt to: having to get up and go into work every day. Given I hadn't done that for ten plus years.
The other difficulty was giving up control. I suffered from wanting to do everything myself. Even though I've got great, talented guys working for me. So I had to learn to delegate and allow other people to work on it. Because up to that point, it was just my project. But now, I'm just a member of the team.
So that was quite hard to adapt to. But other than that, it's not been too bad. It's like I can have a bit more control over my life. I don't have someone telling me exactly where to work. I can make those kind of higher level decisions, which I prefer. So my day-to-day life hasn't changed all that much, really.
R: What helped you with giving up control?
W: I think it's just acknowledging that when you're not the only person working on it, things will go faster and smoother. And also getting feedback from other people is always beneficial.
We'll probably be distributed going forward. But for now, we've got a small office in Edinburgh, and it's been quite useful to bounce ideas backwards and forwards. So when you get used to that, you realize that ultimately, it's gonna be better for the project.
Textual is gonna be like the work of many talented people rather than just one individual. So I think it's acknowledging that it's better for the project.
R: How did you choose your team? Which roles to hire for and who to hire?
W: My first employee's just sitting over there. I knew of him via GitHub. He's got several other projects, one of which was a testing framework. So, we sort of corresponded on GitHub a few times, and I was impressed by his work. And he just, by coincidence, happened to be in Edinburgh. So I just reached out. And I convinced him to join me. So that is employee number one.
At the moment, we have three developers including myself. I'll have a fourth by the end of the year. Actually just in a couple of weeks.Next year, we're probably hiring again, probably in the middle of the year.
Last but not least, I've got a PA and bookkeeper and she's also my wife. Very overqualified for what she's doing currently, but she's a member of the team.
R: What came into the office vs remote decision?
W: It's hard to say. Initially, I thought we would be just distributed. But when I talked to Darren, he was asking about an office, and I started thinking about an office.
And there are some benefits. I think it feels more like work. If you have to get up and physically go to a new location, it kinda puts your head in a different place.
And it has been beneficial in that if you got people around you where you don't have to arrange Zoom calls. You can just lean over and talk to them. I think that has been very beneficial in bouncing ideas backwards and forwards. And creating quick iterations.
I think working remotely can work very well. And as we will have more and more employees in the future, we'll get distributed. But I would like to keep a core team in the same office, because I think that's a very good way of just gathering feedback very quickly.
R: How was the technical part of this transition from side project to a company?
W: I always tried to write code, even if it's just for myself, that other people could pick up and read. But when you are working with other people, that has to be like first and foremost in your mind.
You have to add docstrings and you have to name things very carefully. You have to make it, so that someone else could contribute. I find, a good rule of thumb is: If you can explain it to someone easily, then it's good. If you struggle to explain something or if you feel the need to justify some code you've written, then that probably means that it's not such a good code.
With good code, there should be very little that you have to defend and justify. It should be obvious that this is a nice algorithm, a nice way of doing it.
R: Were there surprises? Stuff that you thought was obvious but was difficult to grasp for your new teammates?
W: Yes, there were. Sometimes, you can lose sight of what's obvious and what's not. Because everything you write builds on something you've written in the past.
So, for you, it might feel obvious because you've got this last two months of work in your mind.
I think, what you have to keep in mind is that when someone comes to the project completely fresh, they don't necessarily have that background. In their mind, they haven't got that mental model.
That's why you have to structure your code in a way that makes it independent of previous work. So that you don't have to, again, justify the code that you've written based on the last three months work. It should stand on its own.
I think, keeping that in mind is important when you're working with other people.
Technical Lessons
R: You also wrote a blog post about 7 things you've learned building Textual. One of them was: "Immutable is the best". Has this been your preference longer or was it a lesson from Textual?
W: I've been thinking that way for a while. But working on Rich and Textual made it clear to me that immutable objects in Python really simplify a lot of things.
One of the problems with Python is, it's a bit too slow. It's reasonable to acknowledge that.
But if you cache things, you can make it a lot quicker. And things are easier to cache, if they're immutable. If objects can't change, you can just stick them in a cache. So, that's one benefit of immutability.
Another benefit is testability. If you have pure functions, which take immutable objects and return an immutable object, it makes testing much easier. Immutability also makes code easier to reason about.
So, I can understand why in some languages everything's immutable by default. In Python, nothing's truly immutable. I think, it's still beneficial to work with immutable type objects. These are, for all intents and purposes, immutable. So, I try to do that whenever I can.
Usually, the exceptions are large objects. These would be more expensive to copy. Bigger objects also tend to have more state and change it more frequently.
Also, there are things which can't be conceptually immutable: input, output or files, etc. Those can't be immutable, because the actual data can change underneath.
For me, immutable is the default idea. I only make objects mutable if I absolutely need to.
R: You've mentioned Python's slowness. Which brings up the question: Why did you choose Python?
W: I chose Python, because it's the language I prefer to work with. I love the elegance of it and the fact that it's very readable, and you can iterate very quickly.
But I've always known it was slow. It's not a compiled language. It doesn't generate machine code. At the machine code level, assigning an attribute is almost instant. It's like less than one cycle. But in Python, you gotta look up a dictionary and execute a bunch of code. I always knew it was slower, but generally it's not a problem.
When it came to Textual, there was a number of optimization issues.It got faster and faster. And most of those improvements were kind of algorithmic. But even once you got the algorithm going as fast as you can, there were a number of micro-optimizations. I think, most code can be halved in the time taken through micro-optimizations. So, in general, the code got faster as I worked on it.
And also, each Python version gets faster. I mean, the newest version 3.11 is, on average, 30% faster. And they're aiming for five times faster.
That's encouraging. I think, in the future, Python won't be seen as a slow language. Similar to JavaScript. If JavaScript can be fast, Python can be fast.
So, I think, in the in the future, speed will be much less of an issue. At the moment, it takes some work. Especially for things which are doing a lot of number crunching, a lot of processing. You might have to really focus on that code to make it fast enough. But, I think, in the future, it might just go fast enough by default. Which I'm really encouraged by.
R: Did I understand that correctly? The execution time can be halved by micro-optimizations for a lot of code?
W: Yeah. I think, that's an average.
If you take any loop, that's more than a couple of lines, you can apply a bunch of micro-optimizations. Just reduce number of operations that have been executed. Doing things like hoisting attribute access outside the loops. I think that most code can benefit from that, and it can go, like, twice as fast.
That's just my estimate. I haven't done any grand tests or anything. It's not an academic thing. But I found that almost all code can be sped up by two times.
R: How did you approach these optimizations? Did you start with profiling?
W: The general advice is that you should profile the code before you optimize. That's not bad advice. But I've found that most of the time, the slow code is really obvious.
Inner loops, loops doing the most iterations and doing the most work are the usual suspects. And it tends to be a piece of code that does more than one thing. It tends to be doing a lot of things. So, I think, often, the critical paths that slow your code down are really easy to identify.
If you understand the code, you have an idea where things can be slow. But you do get to a point eventually where you're not sure. Then you do have to profile and it tells you where to focus your efforts.
But, I think, if you know your code well, it's generally fairly obvious. At least from the outset, which bits are quite slow and which bits are fast.
R: Can you mention an example of a speed-up?
W: Yeah. So when you render things in Textual, it starts off with a high level representation of byte code segments. Which is a piece of text, plus a style. And then Rich will transform that into ANSI escape sequences.
And I could tell that was doing a lot of work: transforming this Style object and the ANSI escape sequences. And this will be done for, like, every piece of text. So, probably every word, we'd have to do this kind of work. So, that was a point where it was kinda clear that was slow, and I spent a lot of time optimizing that.
And one of the ways I did that was make the Style object immutable. I love immutable things, and then I could cache that. So, when you're generating ANSI escape sequences, you don't have to calculate the Style objects again. You can just look them up in a dictionary. And that made it a lot faster.
So that's kind of cool. I think, the new version is twenty times faster than the original code.
Terminal as Platform
R: How is Textual used currently?
W: We have a gallery, showcasing Textual applications. They seem to be more developer focused at the moment. People also like it for cryptocurrency, because they can display tables of numbers that update very quickly.
There's a few productivity tools, but I think it'll be interesting going forward to see further usages. Including apps intended for non-technical users.
The CSS branch was merged just two weeks ago. So I'm really excited to see what will happen in the near future.
R: Which further usages do you expect?
W: There's quite a broad number of use cases. In essence, it's any application that fits in that kind of terminal form factor.
But I think initially, it'll be for internal tools. Managing configuration, managing servers, editing files, browsing files, that kind of thing. Things that work with your local file system or network.
And because it's terminal based, it means you can go from the command line, go into a Textual app, and then go back to the command line. And it's kind of like seamless.
It also means much less distraction than opening a web browser. The terminal can keep you in this frame of mind where you're working and interacting with your computer and doing productivity tasks.
So, I think initially internal tools or things like network automation. But after that, I would like to break out into almost anything. Because if you're a developer or someone quite technical, you are home in the terminal. Most developers open a browser and a terminal at the beginning of the day, and they'll have them open all day.
So I think people would prefer terminal apps to web apps if they were feature complete. Because it just integrates better with our workflow.
Ultimately, there are very few restrictions. Terminals can't do videos or sophisticated graphics. But for everything else, I think, there's a fairly large set of people who would actually prefer content to be in a terminal.
Which is quite an odd thing because I'm running this in a MacBook with M1. But my preferred way of interacting with the computer is with a grid of text in a in a little box. And I'm not alone, so I think, there's probably a fairly sizable market for it.
R: How does the open core business model of textualize.io look like? What will be the paid product?
W: The paid product will be a web service. You can take a Textual app and turn it into a web application running in a browser.
That means you can distribute it to other people more easily. One downside of terminal apps is that they are generally used only by technical people. Which is quite strange to me, because terminals are installed on virtually every desktop, which has ever been shipped.
But they're only used by a relatively small number of people. So, I think, to make Textual truly useful, you have to be able to build apps for everyone, not just for developers. So that's where this web service comes in. If you build an app just for yourself or a small group, maybe it's just in a terminal.
But if you want other people to use it, you can use our service. Create an account, in the command line, run textual serve providing a Python file, and you get a URL. You can host it anywhere on a virtual private server, and you can run it in the browser.
Roadmap
R: How does the roadmap look like? What are the short and middle term plans?
W: Short term is we're building more widgets for Textual. At the moment, it's got quite a small set of widgets.
We've got a checkbox, a button, scrollable panels. But we want more sophisticated widgets. We want a better tree control, markdown viewer, and also the list control, that kind of thing.
We're trying to build a nice big library of built-in widgets. So that if you're building an application, it's generally got what you need.
In the midterm, we're kinda moving away from development to developer relations. So, we want to evangelize Textual by building tutorials and videos. We can distribute them via YouTube and even TikTok and go to conferences and things.
R: So, you're currently focused on widgets. Is it an area where you also expect contributions from the community?
W: Definitely. We would love people to start building widgets. So, you're just going to PyPI, you'd search for "textual-", and then you get like a number of third party libraries.
I think that's very important because, ultimately, we can only build a subset. We can't think of or implement everything. So, we want to allow people to build and distribute their own widgets. That's why we're documenting that quite thoroughly.
R: It seems that you've generally put quite a lot of emphasis on documentation.
W: Yeah, we've put a lot of effort into the documentation. That's one of the reasons why we delayed the CSS branch. This feature needs a thorough documentation.
Just giving someone the code and saying you can use CSS wouldn't be very useful. It needs a bit of introduction and explanation. Once folks have seen the reference documentation, it's quite straightforward. But without documentation, it wouldn't be useful.
So, we've spent a lot time on the documentation website, and I think it's quite nice. We've got nice diagrams and screenshots. We've got reference pages and API level documentation. So we're quite pleased with it.
R: Yes, we encourage people to check out the docs and build something with Textual. Thanks a lot for the conversation.
Resources
Textual
Rich
Further
- 7 things I've learned building a modern TUI framework blog post
- 7 lessons from building a modern TUI framework interview with Will at the Talk Python To Me podcast
- The terminal as a platform interview with Will at The Changelog podcast
Top comments (0)