When you’re writing code, you have business requirements—also known as “features”—such as “the customer must be able to search for books by word count.” You also have “scaffolding” code that has to be written, but that the customer (or manager) doesn’t really notice, such as configuring a web framework and writing exception classes.

Particularly in systems with large amounts of technical debt, developers spend a lot of time refactoring the scaffolding, shoring it up, moving it around, and so on. The more time you spend fixing your scaffolding, the less time you have to develop your features.

What if you could spend less time writing the scaffolding code and more time focusing on the business requirements? Here’s how our company made that happen.

Anyone who’s taken a beginning database course learns about the critical importance of database transactions. For example, consider the following code:

sub transfer_money ($self, $to, $amount) {
    if ( $self->balance < $amount ) {
        Exception::InsufficientFunds->throw(...);
    }

    $self->dec_balance($amount);
    $to->inc_balance($amount);
}

The code is a grotesque oversimplification, but let’s just look at one little line:

    $self->dec_balance($amount);

What happens if that fails and throws an exception? If it fails before the money is withdrawn, that might not be disastrous. If it happens after the money is withdrawn, we withdraw the money, but don’t deposit it to the new account. Our customers will be very unhappy.

So we fix that with a database transaction. It might look like this:

sub transfer_money ($self, $to, $amount) {
    if ( $self->balance < $amount ) {
        Exception::InsufficientFunds->throw(...);
    }

    $self->db->do_transaction(sub {
        $self->dec_balance($amount);
        $to->inc_balance($amount);
    });
}

Now if dec_balance or inc_balance throws an exception, we’re guaranteed that whatever money was withdrawn will still be in the original account.

But it still has a massive bug. What if some other code withdrew money after we checked the balance, but before we withdrew it? Oops. It’s called a “race condition”, because the behavior of your code depends on whether it finishes before or after some other code.

In this case, the behavior depends on whether the system allows negative balances, whether the account allows negative balances, and other factors. We don’t want to risk this bug, so let’s expand the transaction scope.

sub transfer_money ($self, $to, $amount) {
    $self->db->do_transaction(sub {
        if ( $self->balance < $amount ) {
            Exception::InsufficientFunds->throw(...);
        }

        $self->dec_balance($amount);
        $to->inc_balance($amount);
    });
}

So now we check the transaction before we check the balance. Good to go, right?

No. ORMs (object-relational mapper) tend to hold their data independently from the underlying storage mechanism. The data might be stale and you need to refresh it. But you can’t just do a SELECT, you need a SELECT ... FOR UPDATE to ensure that the row is locked in the transaction.

sub transfer_money ($self, $to, $amount) {
    $self->db->do_transaction(sub {
        $self->requery_for_update;
        $to->requery_for_update;

        if ( $self->balance < $amount ) {
            Exception::InsufficientFunds->throw(...);
        }

        $self->dec_balance($amount);
        $to->inc_balance($amount);
    });
}

We still have more work we need to do, but already we had a very simple method that is starting to get complex just to make sure it’s safe! We didn’t add any functionality. And that’s what we want to fix. The “functionality” is what the customer experiences. All that extra code is scaffolding and it’s easy to forget.

All you wanted to do was move a bit of money from one bank account to another. When your code gets seriously complex, it can be hard to track down bugs caused by race conditions and not having proper scope on transactions.

Which brings me to the real point of this article:

The more things you must remember, the more things you will forget.

That is something that developers often gloss over. “You have to know your system.” “It’s your fault that you weren’t paying attention.” “Be better!”

No. Don’t write code with the expectation that people will always remember exactly how things are wired together. The Tau Station MMORPG is a half million lines of code. Even devs who’ve worked on it for years find dusty corners they don’t know about.

Making This Simple

So, what would the above look like in the Tau Station code? Currently the game doesn’t allow players to have multiple bank accounts, but they can move money from their wallet to their bank account and conceptually that’s the same thing. It uses an “Exchange” system we’ve built and it looks like this:

my $exchange = $self->new_exchange(
    Preamble( deposit => { amount => $amount } ),
    Steps(
        Wallet(      $self => remove_credits => $amount ),
        BankAccount( $self => add_credits    => $amount ),
    ),
);

The “Preamble” is a section that declares what success or failure messages are going to be displayed to the player and what information, if any, to use for those messages. The “Steps”, however, are what we’re actually trying to accomplish. In other words, with our exchanges, we write declarative code that describes the desired behavior. All of the “scaffolding” code is hidden away behind the scenes.

For example, note the code we didn’t have to write. The exchange system handles the following automatically:

• Exception handling
• Transactions
• Success messages
• Error messages
• Logging
• Tax calculations
• Sufficient funds

And for other exchanges, we have things such as:

• Job queues for asynchronous work
• Email
• Message popups
• … and many other details I’m omitting from this

In fact, the declarative nature of the above code means that we can even take this “exchange” and cut-n-paste it for our non-technical narrative designers and they understand what it means!

And guess what? We can reuse this. Here’s another example, reusing the BankAccount.add_credits code, for a relatively complex action of buying an item from another player (a bit simplified).

But first, imagine how you would write the code for buying an item from a vendor. You need to be in the same area, right? You need to have enough money, it needs to be for sale, and so on. Think of all the code you would have to write to check for that and all of the other issues, along with ensuring the database remains consistent.

Here’s what it looks like declaratively:

Steps(
  Location(           $self => is_in_station    => $station ),
  PlayerMarketItem(   $self => is_not_seller_of => $item ),
  PlayerMarketItem(   $self => can_buy          => $item ),
  BankAccount(        $self => remove_credits   => $amount ),
  BankAccount(      $vendor => add_credits      => $amount ),
  Inventory(          $self => add_item         => $item ),
  PlayerMarketItem( $vendor => remove_item      => $item ),
)

Did you think of all those steps? How much code would you have had to write to implement all those steps? And would you have remembered all the possible exceptions, the transactions, the SELECT ... FOR UPDATE, and so on? Would you have remembered or cared about all the success or failure messages?

By writing code in such a declarative style, we have taken incredibly complex behavior and not only made it simpler, but more correct.

Here’s another example. In Tau Station, you can save your progress by gestating a new clone. If you die, you respawn into your latest clone. What does clone gestation look like? It used to look like this mess:

sub gestate ( $self, $character ) {
    # check their area
    croak( … ) unless $character->area->slug eq 'clonevat';

    # make sure they can afford the clone
    my $price = $self->price_per_character($character);
    if ( $character->wallet < $price ) {
        $character->add_message( … );
        return;
    }

    my $guard = $self->result_source->schema->txn_scope_guard;

    # pay for the clone
    $character->dec_wallet($price);
    $character->update;

    # spawn a new clone
    my $clone = $self->find_or_new(
        character_id    => $character->character_id,
        station_area_id => $character->station_area->station_area_id,
    );
    $clone->$_( $character->$_ ) for $character->physical_stats;
    my $now = DateTime->now;
    $clone->clone_date($now);
    $clone->gestation_date(
    $now->clone->add( seconds => $self->gestation_delay($character) ) );
    $clone->update_or_insert;
    $character->add_message( … );
    $guard->commit;
    return $clone;
}

And that was a bucket of bugs. And hard to follow. Now it looks like this:

Steps(
  Location( $self => is_in_area => 'clonevat' ),
  Wallet(   $self => pay        => $price ),
  Clone(    $self => gestate    => $self->station_area ),
)

Honestly, which of those would you rather write?

Declarative Exchanges

So how does this magic work?

The Behavior

When you create a new exchange, the first thing it does is go through your steps and figure out what objects might be updated. Then we:

1. Start a transaction
2. Refresh all assets via SELECT...FOR UPDATE
3. Run all the steps
4. Commit on success and rollback on failure
5. Notify the player(s) (if appropriate)

It sounds simple, but there’s a lot more going on under the hood. For example, if you are a member of a syndicate and you earn income, you may have to pay “tax” to that syndicate. Thus, the exchange needs to know that it must fetch the syndicate object, lock it, and send taxes to it. As a developer writing the code, you almost never have to pay attention to this. It’s all automatic.

The Structure

Each exchange step looks very similar:

Location( $self => is_in_area => 'clonevat' )

In exchange parlance, that’s:

Topic( subject => verb => object )

Everything follows a linguistic SVO (subject-verb-object) pattern. To create a new Topic for the exchanges, you create a class called Veure::Economy::Asset::Topic (there are legacy reasons for the name) and have it inherit from Veure::Economy::Asset. We have another system that automatically finds and loads all these classes and ensures that the asset name is exportable on demand. You just write the code, there’s no need to wire it together because that’s done for you.

Each of these classes takes a subject (the first argument) and implementing a verb is merely a matter of writing a method. The object (in linguistic terms) becomes the argument(s) to the method. A simple is_in_area check might look like this:

sub is_in_area ( $self, $area_slug ) {
    if ( $self->station_area->area_slug eq $area_slug  ) {
        return $self->new_outcome( success => 1 );
    }

    # give them a nice failure message
    return $self->new_outcome(
        success => 0,
        message => ...
    );
}

Simple, eh? And now we can reuse this to our heart’s content.

Failure

Aside from the fact that the exchanges allow us to write incredibly complex code very quickly, one of my favorite parts is the fact that even though it’s declarative on the surface, underneath it’s objects all the way down. That means we get the full power of OO introspection where we need it. For example, what happens if I’m running the test suite and an exchange throws an exception?

Well, of course, we get a stack trace. And at the top of that trace, we get a stringified version of the exchange, along with its data. In this example, it’s for refueling a spaceship:

character('ovid')->new_exchange(
    slug            => 'refuel-current-ship',
    success_message => 'Your ship is now being refueled.',
    failure_message => 'Unable to refuel ship.',
    Steps(
        Area( character('ovid'), is_in, docks ),
        Ship( ship('ovid', 'bootlegger'), is_docked_on, tau-station ),
        Ship( ship('ovid', 'bootlegger'), needs_refueling ),
        Character( character('ovid'), not_onboard_ship, 
          ship('ovid', 'bootlegger') 
        ),
        Money( character('ovid'), pay, -15.00 ),
        Character( character('ovid'), refuel_ship, ship('ovid', 'bootlegger') ),
        Character( character('ovid'), set_cooldown, 
          {ship => ship('ovid', 'bootlegger'),
          cooldown_type => 'ship_refuel',period => 1000} ),
    )
);

In this case, we got an exception because there’s a serious bug: somehow the character has been asked to pay negative credits. This stringified exchange shows this very clearly here:

Money( character('ovid'), pay, -15.00 ),

So it’s dead simple to recreate conditions that cause failures in incredibly complex behavior. In this case, we knew our exchange system was fine, but something was sending it bad data.

Regrets

If there is one regret I have about the exchange system, it’s that it’s not open source. We would release this if we could, but when we started on it, it wasn’t clear what the end result would be. Thus, it’s deeply tied to the game Tau Station . If we find ourselves with the time—or a contract—for this, we will release this for everyone.

As an aside, Ruby has something rather similar, named Trailblazer. The main difference in exchanges is that we’re not tied to MVC or the web. Trailblazer documents itself as “A New Architecture For Rails”, suggesting a tight coupling. That being said, it has fantastic testimonials which match our internal experience with exchanges. I expect we might see more of this type of code in the future.

Hire Us!

If you want this sort of software quality for your code, hire us. The above examples are in Perl because that's what the backend of Tau Station is written in, but we work with many different languages.

Here’s a case study of some emergency work we did.