loading...
Cover image for Stateless Service Gotchas

Stateless Service Gotchas

kspeakman profile image Kasey Speakman Updated on ・7 min read

Why stateless services?

The primary benefit of stateless services is scalability. In particular, Functions-as-a-Service -- AWS Lambda, Azure Functions, Google Cloud Functions, etc. -- should only host stateless services. Because each function execution might run on a different machine and multiple executions may run in parallel on different machines.

What is state?

State is just data. But in this context it means data kept in memory between executions, and needed in order to service requests. A stateless service should not be required to keep anything between executions, but it may depend on other parts of the architecture with "memory" like databases.

How To

So this is easy right? Just shove everything in a database instead of keeping it in memory. Nearly that. Below I have outlined some specific gotchas that can make a service stateful unintentionally. The first couple of the gotchas below may already be disallowed by Functions-as-a-service providers. But I'll mention them anyway, because statelessness can be useful even if not using FaaS.

Gotcha: resource handles

This is pretty well-understood nowadays, but it bears mentioning that open files and network connections have state. To be stateless, you cannot assume a resource handle is still open between invocations of the service. You have to assume the service may be shutdown and restarted on a different node at any time. In general, it is best to open a resource only when you need it and close it immediately afterward.

Gotcha: session data

Many platforms offer built-in session handling and default to keeping it in-memory. If you want to use session data, move it off of being handled by the service, probably into a database.

Gotcha: ordering

You cannot assume that two requests issued asynchronously will be executed in the order they are sent. It could be that the first request gets delayed in processing on one node. Then the second request runs on a different node and finishes first. If the order of processing is important, wait for the first request to finish before issuing the second.

Gotcha: serial identity

The common type of serial identity is the auto-increment primary key. Since many copies of a service may be executing in parallel, you cannot make assumptions about the next-assigned ID or guarantee a gapless ID sequence due to write conflicts (discussed below). In fact, the whole concept of serial identity can become a bottleneck to scalability unless due considerations are made. There are other identity strategies that fit better for scalability, including ones that are (mostly) ordered.

Gotcha: write conflicts

It can happen that two users are trying to make changes to the same entity at the same time. Classic example: An account has $100 balance. Two simultaneous requests are issued to withdraw $100. In a naive multi-threaded or multi-node system, neither request knows about the other, so both would likely succeed. You would rather this kind of situation be detected as a write conflict. (In the real world, banks see this as an opportunity for profit instead of a problem. But either way they have to handle it.)

Every API potentially has this problem. But a lot of APIs never take write conflicts into consideration, and might not need to. In low-volume systems or systems where users can't change shared resources, concurrency conflicts are rare. And the worst case is typically an every-blue-moon support call: "Hey, I just made changes to this item, but now my changes are missing." Two simultaneous changes to an item resulted in one of the changes overwriting the other (aka Last Writer Wins). Support replies "Have you tried turning it off and on again?" and everybody moves on with their lives.

Write conflicts are where you have to start thinking about concurrency control. The usual suspects being pessimistic or optimistic concurrency.

Pessimistic concurrency

In general pessimistic concurrency will get slower as you add more nodes. You can think of it as having to obtain an exclusive lock on the data before you start running business logic. More nodes means longer waiting times for a lock. Since the point is being scalable, pessimistic concurrency is not a great fit.

Optimistic concurrency

Optimistic concurrency says I'll go ahead execute the request and prepare all the changes. But before I commit my changes, I'll make sure that the entity is exactly the same as when I started... nobody else changed it in the meanwhile.

The most common way to implement optimistic concurrency is by adding a "version" to each entity. It could be, for example, an integer or an etag. When I start executing the request I can read the version of the entity from the database or let the client provide it. (After all, the client has made the decision to change the entity based on the version they had at the time. So why not let the client tell me that?) Then if the version still matches when I'm ready to commit, I bump the version with the commit.

If the version does not match, then there are decisions to make. Do I stop and error? Do I reread the entity and retry the request? Do I commit the changes anyway? Fortunately these are not my decisions to make. I have to ask my business experts: what should I do if two people try to make changes to the same order at the same time? It might even come down to very specific circumstances like: Bob is trying to cancel the order but meanwhile Pete just added $10,000 more in sales to it. They will probably tell you to chuck Bob's request in the trash.

Yep, scaling will make you think about problems you never considered before. Both technical and business-oriented.

Should all services be stateless?

No.

...

Oh, you want more? Ok, well sometimes it is not worth the trade-off. For example internal APIs with a low number of requests per second. Save yourself some time there by skipping the extra-cost scalability features.

Doesn't this just move the scalability problem to the database?

Yes, great observation! With your services stateless, you can deploy as many copies as you need to handle your load. But your storage tech may not be able to handle it. That is a topic for another post.

:)


Addendum - Examining the Scalability of Authentication and Authorization

Authentication means proving yourself to be a valid user. "This Van Gogh painting has been authenticated as genuine." Authorization means checking whether a valid user has permission to perform an operation. "You are authorized to access floors 1-3, but floors 4 and above are off-limits." When referring to both I usually write "auth".

Not that I've done a formal survey, but most auth frameworks should already work with stateless services. Users, permissions, and the link between them are likely stored in a database anyway. Traditional auth frameworks store the list of active logins in the database too. Then your service needs to lookup that active login session on every request to make sure it hasn't been terminated (logged off by either the user or an admin) or had its authorizations changed. That means a network round-trip to ask the database on every request. Even in stateful services, this can cause unwanted latency. So auth frameworks frequently introduce in-memory caching to reduce the number of database reads.

The problem here is that in-memory caching has limited effectiveness on stateless services, if the FaaS provider even supports it. Because each service invocation could run on a different service instance, there will be a lot more cache misses than in stateful services. And consequently a much larger percentage of requests will need to make a round-trip to the auth database. Thus adding latency and database load which may hinder scalability. To make caching effective again, you would need to use an external cache like Redis. That is another integration point, another system to care for and feed.

The newer alternative is to use a cryptographic signature in lieu of centralized database lookups. The common implementation is JWT. I have an LI5 post that does a decent job comparing JWT with traditional authorization. JWT allows for self authorization. That is, the service can check for itself (using CPU instead of a DB round trip) whether the token is valid. Once validated, the token's data (e.g. roles and permissions) can be trusted.

But JWT has a couple of downsides that come to mind. The first one is that it has a learning curve. Like most things that use cryptography, it is best to let a trusted provider or library take care of most of the gory details. Configuring it can still feel other-worldly. But don't panic. Others have done it, and you can too.

Second, permission changes are not immediate. Remember above how I said that authorization is self-service? Without checking some central list of valid logins, I do not have a way to know that a particular token has been administratively revoked. As far as my service is concerned the token will continue to be valid -- letting the user make requests -- until it expires. The common way around this is using Refresh Tokens. This boils down to forcing a reauthentication every so often (e.g. every 5 minutes). But the reauthentication happens behind-the-scenes with no user interaction. That way at least every 5 minutes (or whatever time you set) you get updates on permission changes. This outcome is eerily similar to traditional auth lookups with 5-minute caching. ;)

Personally, I have been using JWT for newer development. I haven't found that I needed refresh tokens so far. It is a good thing too, because most sources say that refresh tokens should not be exposed to the browser. What fun.

Posted on Oct 3 '17 by:

kspeakman profile

Kasey Speakman

@kspeakman

collector of ideas. no one of consequence.

Discussion

markdown guide
 

I really appreciate this article's content but I am wondering...

Why this image ?

What does it add to the article, except its existence ?

 

Thanks for your comment. Not sure why, but it cracked me up.

If you find an image which is a better fit, I'm open to changing it. My thought process is that services are commonly represented as gears in architectural diagrams. It is a great metaphor for scalable services because a gear is a pretty simple, copyable piece in a larger system. I looked for a pile of gears to indicate multiple copies of a service. But most of the ones I found were behind pay walls. I landed on this one because it has multiples of the same gear as part of a larger system of gears, and it was CC0 licensed.

 

Well it's not a criticism and I really don't have any advice about putting this or that image, but it was more of a general question.

I see more and more blogs putting sort-of stock photos at top of articles, which doesn't really add anything except some media to the article, and I was wondering about why do such things.

It came into my mind because, where I live, I have a dead slow internet and, when I want to read an article, I generally appreciate having access to an article's content and optionnaly images giving informations like diagrams and such and, when a page takes more than 10 seconds to load, I close the page and put the website in my "heavy list".

Sometimes, headline photos are so heavy I completely disable remote content loading to get the plain article, and as I don't see any interest in those images, I was really wondering why such a thing was more and more common.

Ah, I am with you now. I didn't give the size of the image a second thought, but I will in the future. Thanks for the comment!

 
[deleted]
 

Great point. I left out auth because of the depth of the topic, and also because many auth frameworks are usable from stateless services by default. IOW, they already use a database instead of keeping the data in memory. And that can be a good starting point depending on your experience.

However, it is an entirely valid point that: using a database lookup for every authorization will eventually become a scalability bottleneck, and it will add latency to every request (round trip to auth database). JWT is a much better alternative which alleviates the bottleneck and avoids the latency. On the downside, it can have a learning curve and permission changes do not take effect immediately.

I think you are right that it bears mentioning, so I will add a section for it.