Databases: Why Does It Matter What We Choose?

As we all know, database in terms of a project is its vital part, but its vitality depends on decision we make throughout whole development cycle. Databases integrated our life in with such meanings as 'relational' and 'hierarchical' models, but, since those times we have had a dozen databases, and each suggests the solution we need, but in the end our project starts 'suffering' from our decisions related to database we choose and approaches we make.

Traditional DBMS, NoSQL, NewSQL, - we do not lack them, but we do lack the solution and make even wrong one, afterwards having issues on networks or query performance, without saying about obvious business needs. You could solve all these problems by having DBA, but they're not a silver bullet; they can solve your project issues on db, but as far as it's possible.

So, the question "How to choose and design database for project needs and not to stuck with that?" should concern us at first place. Constantly, the path to it shouldn't be as hard as it seems be first time. This is why this article will try to explain (not to give your a proper path) how you could build a pattern due to your databases problems.

What problems can we have?

In the means of 'problems', there can be identified three main large factors: technology, human, economy.

Technologies

Here, your project can be troubled with its current stage of development: rather it's MVP or PoC, or even production-ready. While PoC suggests maneuvers and lacks data certainty, MVP already has its real data for database which is chosen during first iteration, but on the other hand, MVP can have economiс uncertainty that leads up to tech debt in possible future.

Meanwhile, production-ready project has all we need: database, data, code, integrated -ops cultures to be better, nevertheless it can't avoid possibility to have serious issues with databases or its replica sets across multiple servers.

Humans

With technologies, one of project critical part is human factor. Data misconception due to final product subtlety or either misunderstanding between client and team can lead to tech debt or performance issue in production environment and cause mistrust in a team towards each other as a consequence.

In addition to what was said above, there also should be mentioned that chaotic management process and current project bad engineering culture can accelerate databases issues.

Economy

Once in a while, mostly economy isn't the strong side of the project. Occasionally, customer doesn't understand how much it will cost to make desirable data available; team doesn't know how much they're going to spend on database cluster. In a few cases, team or their management can't decide whether it's better and cheaper to support database cluster with their own DBA or buy support on other side.

OK, but what about examples?

Yet, there haven’t been shown any examples so far. Let's answer the questions from our point:

• When was the last time you issued problems with database concerning technological side of the project?

• When did you last argue on issues involving databases?

• Were your calculation made right about database maintenance?

If you answered all these questions above, - you can skip this chapter. If not, -  let's see the examples, though you should understand that all following situations may never occur.

Example 1

A team is not satisfied with current database performance and want as well to increase it by 30-40%, but they find out that not only speed was one of their concerns, however it was also about stability. The question was: "How can we improve performance to be sure that 99.99% of our queries will reach our SLA goal?". As the result of their desperate search next conclusions are made: sharding, upgrade or either it's the table.

Even so, all conclusions are worrisome. If they pick sharding, there can be network or read/write problems; in case of upgrades team can encounter financial risks or downgrade. The final conclusion - the table itself or database - waste of time that they lack.

Example 2
Meet Marlene and Yevhen. Marlene is a database expert whereas Yevhen is a developer. Although they’re different, they work in a team so far; their teammates know Marlene and Yevhen can dispute occasionally when it comes up to a database. This time wasn't an exclusion.

Marlene was concerned that Yevhen's task solution might bring troubles in the future and could unexpectedly lead database to such mistake as OOM and database might end up dead. Yevhen kept on saying quite the opposite and tried to reassure team that his solution could solve client's trouble.

How R&D could solve it?

As much as we'd like to satisfy our needs and customers as well, we should understand the importance of engineering culture, or precisely the innovation part, which is likely to be considered one of the factors of company survival. Indeed, it can also become a burden nonetheless. But that's not the point.

Right now let's see how r&d could help us bring more transparency due to database. We could divide our research and development cycle into four steps: theorize, explore, design & develop with further tests implementation, implement our solution itself and improve.

Let us seek how it can be scaled up to databases' problems below.

Theorize (and synthesize)

In means of theorization, we should provide information on research object and exploration's direction. Clearly, we should form here the research object along with known SRE terms: SLA, SLO, SLI. Although, we should remember about trade-offs by two points of view - customer's and team's. While customer defines kind of information he needs to see on final stage, engineering team should present liability to support client's demands regarding needs (from technological aspect up to economy).

Looking back to our examples, we can see that team in Example 1 mostly made a few assumptions (sharding, hardware upgrade and database schema refactoring), whilst Marlene and Yevhen have disagreements due to indicators and agreement with client (Marlene is sure that Yevhen's decision will affect SLA; Yevhen's opinion is all about SLI).

Explore (hypothesize and clarify)

We're trying to make all our theoretical beliefs real here, approving or rejecting client's needs. The desired result is feasibility study along with pilot study; these papers will answer such questions as "Did we get what we wanted? Are there any risks during development or after implementing client needs?" Mostly, it's our first attempt to implement and develop features for database and client's data, - it can be either database schema integration or adding more capabilities to almost existing cluster.

It's okay to identify non-standard situations and experiment with them. As far as we go deeper into details, we’ll make sure that further implementation and design process will be definitely worth the low cost.

In examples above, the team from Example 1 will make experiments to solve all the puzzles in test environment, from tracing query and explain plans till database schema refactoring; for Example 2 Marlene and Yevhen, who are suffering from lack of communication, will sit and absorb their knowledge using 'pair programming'.

Design, develop and test

The necessity of domain knowledge is important nowadays besides tech skills. So to say, each team member should also understand that willy-nilly he can be replaced at any time. Client won't catch the thing you're saying if you're gonna explain to him technical aspects of feature implementation; he is concerned about his business demands and expects from you the same understanding of what he sees by output based on data he provided you with.

The most part of r&d staff is here, in this step. We develop feature and make sure that it will meet SLA we made while building theories. Hereby, we should contribute to documentation and project knowledge base for future team members to work along with us or instead of us; quality assurance should mean the same as development, besides, QA will make sure that our database enhancement or data meets business requirements.

The team didn’t expect that the problem was so simple: the index that caused so much trouble and some optimizations the made a while ago, lead to their current problem. They decided to rollback some of their doings and made a post mortem study, in which they told that it was not only the database's problem but also their misknowledge of its internals. Marlene and Yevhen made out a solution that met their both expectations, in addition to team's client was excited about modifications included to output and approved it.

Implement and improve

The conclusion is successful implementation and data output to client. Nonetheless, the improvements will still be made, but not as fast as it possible, but as fat as it suits us and our client. Of course, it's also about minor bug and fixes to be released after our database changes: from migration and up to altering value type (if it's possible of course).

Conclusion

If you build the system, you should remember that it's not only the code and performance matters. By making approach to r&d, you can improve current team's engineering culture and build up a system where database won't be turn into a catastrophe, but will be controlled and data will meet client's needs.