loading...
Cover image for Directions and roadmaps
Darklang

Directions and roadmaps

paulbiggar profile image Paul Biggar Originally published at blog.darklang.com ・5 min read

There was a wonderful post just now that I saw on HN as I was procrastinating writing this post. It was by Tim Bray, and it covered my favorite topic, which is just how batshit cloud services have gotten, this time Traffic Director on GCP. I don't have anything more to say about this that you haven't heard before, blah blah blah, complexity bad, blah blah, simplicity good.

Which leads me quite nicely into some clarity I've gotten over the last few weeks. After reducing the team down to just me, I spent the last month doing a lot of thinking, really trying to get my head around where Dark is, what I should be doing, priorities, etc.

One idea that I raised in an earlier post is that perhaps Dark should try to do less. Rather than being the be-all-and-end-all of how to build apps in the cloud, I speculated that maybe aim a little shorter and build perhaps Zapier-but-with-a-programming-language, or perhaps a really nice forms-library-with-a-programming-language, or whatever.

I talked to users, contributors, investors, friends, etc, and spent a lot of time thinking about this, and basically concluded "fuck that". Almost everyone involved with Dark, me included, gets excited about revolutionizing backend programming, and I concluded that taking a detour from there is not interesting to anyone.

Positioning

That isn't to say that there isn't value in better-zapier or better-google-forms or whatever, but that these are nice use cases of Dark, rather than the actual goal of Dark.

The root issue here is that Dark is hard to position, in a marketing sense. Startups and projects tend to work best when they start targeting a small niche, like "retail", or "low-budget fast fashion retail", or even better "low-budget fast fashion retail targeting urban women aged 24-32". So everyone wants to know "what niche is Dark really good at?" And unfortunately the answer to that is far less specific than I would like, as a lot of tech is designed to let you do whatever you can come up with. "Programming language but for retail" makes about as much sense as "Kafka but for retail" or "computers but for retail".

We actually tried to do something like this with Slackbots, you may remember. We tried to get 300 Slackbots written in a month or so at one point, and I think we got around 15 written. The problem is that people just don't have that many Slackbots to write in a particular week or month. And when they do, they have the same problems that other people have, which is that the thing you want to connect your Slackbot to can be just about anything.

So we've struggled with positioning, and one of our big problems has been telling people what it is that we do.

Current State of Dark

Currently, I believe that Dark is a pretty effective MVP, but is actually further from product-market fit than I initially thought. The reason for this is that features that Dark needs to be successful are often hidden below either product or technical debt. We've gotten some very effective proof points that Dark is on the right track, in particular that the core things that I think make Dark great (Deployless, Trace-driven development, and the lack of infrastructure) are things that people love.

Roadmap

The connection between the MVP and the positioning might not be immediately obvious, so let me spell it out. Our positioning tells people that Dark is a particular thing, namely the easiest way to code backends. And then they try Dark and discover that that is not the case. And then they wander away.

I think both positioning and the product are missing the same thing, which is a sense of where Dark is going. To illustrate what I mean, here's an extremely rough draft of how I think I should position Dark:

Dark is trying to be the easiest way to build an API. It's experimental, shiny, with interesting features, and very very incomplete. Here's a list of what's interesting. Here's a list of what you can do with it. And here's a list of what sucks about it.

Then the website would, for each feature/use-case discussed, include the relevant parts of the roadmap right there. For example "Dark is currently good for teams of two or fewer. To handle teams of 10, we'll need Operational Transforms, global feature flags, a history of changes made, and code review".

Not coincidentally, this same problem (the lack of where we are going) has come up in the two contributor meetups we've had. People are eager to help, but unclear of how exactly to do so, beyond the fairly chore-like issues in the issue tracker. So a roadmap should help there too. Unsurprisingly, this was also a problem for the engineering team before.

Next steps

So I'm going to try to produce a roadmap over the next few months. I think a good way to approach this is to try to write some app in DarkV2, without actually implementing any of it, just essentially writing out the code in a doc. Then, as I "use" the DarkV2, write the spec explaining what exactly these features are, how they work, etc, focusing especially on getting enough detail to outline the difficult bits.

The aim is to say where Dark is going, so that we can say how we are going to get there, and present that in a useful form for users, contributors, etc.

The difficult bits here refers specifically to the things that we've learned in Dark so far. Like, how the HTTP "framework" is inflexible, or that it sucks that you can't write your own types, or that Dark doesn't have GraphQL built in, or that we're relying on traces in places where a type-system would be much more useful.

And in the meantime, I'm going to be doing a bunch of learning about various technologies (eg GraphQL, Rust) - as I'll be replacing large chunks of the product, now is as good a time as any to think about their implementation.


You can sign up for Dark here, and check out our progress in these features in our contributor Slack or by watching our GitHub repo. Comment here or on Twitter.

Posted on by:

paulbiggar profile

Paul Biggar

@paulbiggar

CTO of Darklang (https://darklang.com). Founder of CircleCI.

Darklang

Dark is a holistic programming language, editor, and infrastructure.

Discussion

markdown guide
 

Almost everyone involved with Dark, me included, gets excited about revolutionizing backend programming, and I concluded that taking a detour from there is not interesting to anyone.

Glad to hear you say this as I was worried we'd lose such an interesting future in backend development.

 

Hi Paul,

I signed up for the beta and tried out the tutorial. I also had a look at the create a blog tutorial canvas. Here are a few thoughts:

  1. Signing up required an additional click to Reset password flow. Is it possible to send the reset password link in the welcome mail itself?
  2. Really liked the documentation and the getting started tutorial.
  3. When I create an entity in DB e.g A (id int, val text) there can be an option to auto-generate the CRUD methods for it
  4. I liked the Copy as CURL option for the non-GET methods. Is it possible to fill it (for POST body json) with either randomly generated dummy data or a user-specified default data?
  5. This is mostly a personal preference, but I generally approach an API design in a data-first manner i.e I play around with the schema (if using relational DB) before I start work on the endpoints. So the trace driven development was a new thing for me.
    • Is it possible to create a "span" of traces e.g (from an e-commerce perspective) - user creates a product, uploads product image, updates a collection (e.g "Trending") with that product, uploads a deal (buy one get one) for the product. This kind of span where 1st request creates a product-id and later requests use them is especially useful for quick end-to-end testing
    • Building on the e-commerce example from previous point, I noticed that we cannot yet create a user defined sum type e.g Product state of ( Uploaded | Inventory Int | OutOfStock | Deleted ). is there any other way to manage the product state machine in a type safe way?
  6. I did not understand the gray "v5" in REPL for making the HTTP request. Is that an internal versioning of the Standard library? I initially thought the function itself had a v5 suffix and it was being suggested by the auto complete.
  7. Is there a comparison with some of the low code / no code tools like Hasura?
 

Thanks for the thoughtful writeup! Planning on fixing 1, 3, 4 and 5. I'll take a look at how to simplify 6 also!

I really like the idea of a span of traces. Also, will be adding enums in the future, those are really important for exactly the reason you state!

There isn't really a comparison with low code, as it's not really intended for the same audience. Will think about it though.