Rocket Tutorial 04: Data Persistency and Rocket (with MongoDB)

[Photo by Luigi Pozzoli on Unsplash, modified (cropped)]

We will continue to build a CRUD API with Rocket.
This installment: persistency with a DB!

The code for this tutorial can be found in this repository: github.com/davidedelpapa/rocket-tut, and has been tagged for your convenience:

git clone https://github.com/davidedelpapa/rocket-tut.git
cd rocket-tut
git checkout tags/tut4

We need to secure the data.

Right now we have everything saved in memory. This is not an ideal situation, because we need persistency. Even if we could make sure that the instance of the server will never fail (which is impossible), the first time we would need to shut the server down, all data would be gone.

We can easily test for this (yes TDD, Test Driven Development, is a thing! Although I don't use it too often :-) ). Le't create a new tests/persistency_test.rs:

use rocket::local::Client;
use rocket_tut::rocket_builder;
use rocket::http::{ContentType, Status};
use rocket_tut::data::db::ResponseUser;
use serde_json;

#[test]
fn create_and_persist_test(){
    // We make sure that client1 gets properly disposed of
    {
        let client1 = Client::new(rocket_builder()).expect("Valid Rocket instance");
        let mut response = client1.post("/api/users")
            .header(ContentType::JSON)
            .body(r##"{
                "name": "John Doe",
                "email": "jd@m.com",
                "password": "123456"
            }"##)
            .dispatch();
        assert_eq!(response.status(), Status::Ok);
        assert_eq!(response.content_type(), Some(ContentType::JSON));
        let response_body = response.body_string().expect("Response Body");
        let user: ResponseUser = serde_json::from_str(&response_body.as_str()).expect("Valid User Response");
        assert_eq!(user.name, "John Doe");
        assert_eq!(user.email, "jd@m.com");
    }

    // Let's create a new client and ask for info there using the email
    let client2 = Client::new(rocket_builder()).expect("Valid Rocket instance");
    let mut response = client2.get(format!("/api/users/{}", "jd@m.com")).dispatch();
    let response_body = response.body_string().expect("Response Body");
    let user: ResponseUser = serde_json::from_str(&response_body.as_str()).expect("Valid User Response");
    assert_eq!(response.status(), Status::Ok);
    assert_eq!(response.content_type(), Some(ContentType::JSON));
    assert_eq!(user.name, "John Doe");
    assert_eq!(user.email, "jd@m.com");
}

We can now test this:

cargo test -- --nocapture create_and_persist_test

[.. skip ... skip ..]

thread 'create_and_persist_test' panicked at 'Valid User Response: Error("invalid type: string \"user j.doe@m.com not found\", expected struct ResponseUser", line: 1, column: 28)', tests/persistency_test.rs:31:76
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
test create_and_persist_test ... FAILED

(we cut much of the output to focus on the problem)

The test failed because the API could not find the user: invalid type: string \"user j.doe@m.com not found\".
Thus, serde_json could not construct a valid ResponseUser and it failed.

Since we did not use the same server, but we re-created it using the same rocket constructor, the new server did not have the data that indeed we sent to the first one.

Well this is a problem, because in real life failure does happen, and recovery must be a feature (besides scaling: I mean, if two instances of the server cannot communicate with one another, at least by passing data to each other... what a mess!)

MongoDB to the rescue

We need a way to store persistently rocket's state. Enters DBMS: a DataBase management System.

If you have it not installed on your system, you can use an online service (Atlas, for example. There's a free tier to start testing it).
Otherwise:

# in the following substitute "focal" in "ubuntu focal" with "bionic" or "xenial"...
echo "deb [ arch=amd64,arm64 ] https://repo.mongodb.org/apt/ubuntu focal/mongodb-org/4.4 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-4.4.list
sudo apt-get update
sudo apt-get install -y mongodb-org

(Please, check the official installation page for other systems).

Now we need to start it.

sudo systemctl start mongod

Once everything is in order, let's start to use it in our code:

cargo add r2d2 r2d2-mongodb dotenv

We added few crates:

dotenv makes it easier to use secrets in our project.

r2d2 r2d2-mongodb are needed to pool the mongodb, that is, to create a pool of connections in order to speed up the communications

As database, we are using MongoDB, which is maybe the most known noSQL database. Notice we are not using the current crate (mongodb), but that which is exposed in r2d2-mongodb. More on that later.

This database is not relational: it is not a series of tables, but instead a series of collections of documents. Such documents can be also different one from the other: that is, they need not to be homogeneous.

In practice this means that if we add or vary some fields on Users in the future, the old users can remain the same (we need still to provide support for them, though), while the new one can use the new features. Also, if we have different kind of users, that need different fields, we do not need to keep them in two different tables, or add all fields to all users, as in a relational DB, but we could have different kinds of users in the same collection at the same time.

In sum, sometimes a noSQL DB is more fit to the purpose, as in this case, so we'll take advantage of this.

Besides, MongoDB uses a format for documents, BSON, which is compiled (binary) JSON: changing from one to the other is a matter of milliseconds, so we can say that if you are already dealing with JSON documents, to store them in MongoDB is a naural choice.

At last, a CAVEAT: using mongodb in Rust means matching out for a lot of Result, Option, and Result<Option>, because the connection to the DB is usually over the internet (MongoDB has its own database-as-a-service as well). As we know, net connection is more prone to fail than, say, a connection over a local socket to an instance of MySQL running on the same machine as the server. That is why the unwrapping of the responses can take looooot of code.

That said, there are two ways to go about to check for many Result's and Option's: either match everything, or using combinators: ok(), unwrap_or(), and_then(), etc, etc... I prefer to match and follow strict indentation, so that it is clearer at the end what we are matching for. We use indentation to help understanding the patterns of Ok(_)/Err(_) and Some(_)/None.

Maybe it's matters of style: being also proficient in Python means I can reason about code using indentation much easier than follow a pattern of dot-combinator (.and_then(), .ok() etc...). Those who prefer combinators will forgive me, I hope.

A note on mongodb and rocket_contib

Before the last release there was a direct support of mongodb in the databases section of rocket_contrib. Now the support has been removed. I tried even with an older version, but I could not manage to let the two connect.

It is a pity because there is a good support for the diesel ecosystem instead, which is geared towards relational databases (MySQL, Postgres, SQLite), but the support of the NoSQL databases is heavily lacking... Anyway, let's start coding our way around in MondoDB.

Implementing it in code

Let's first modify src/data/mod.rs:

pub mod db;
pub mod mongo_connection;

Now we need of course a src/data/mongo_connection.rs:

use std::ops::Deref;
use std::env;
use dotenv::dotenv;
use r2d2::PooledConnection;
use r2d2_mongodb::{ConnectionOptions, MongodbConnectionManager};
use rocket::http::Status;
use rocket::request::{self, FromRequest};
use rocket::{Outcome, Request, State};

First thing to notice: dotenv has to be used in connection with std::env.
The method it uses is as follows: dotenv reads the variables from a .env file, and assigns them to the environment variables. Then, we can use std::env in order to retrieve them.

We are using std::ops::Deref because we will need to impl the connection to the mongodb pool later on.

We declare two types for later use:

type Pool = r2d2::Pool<MongodbConnectionManager>;
type PooledConn = PooledConnection<MongodbConnectionManager>;

The type Pool refers to the pooled database, and the PooledConn to the pooled connection to that DB.

pub struct Conn(pub ConnType);

That is the connection we will use.

impl<'a, 'r> FromRequest<'a, 'r> for Conn {
    type Error = ();

    fn from_request(request: &'a Request<'r>) -> request::Outcome<Conn, ()> {
        let pool = request.guard::<State<Pool>>()?;
        match pool.get() {
            Ok(database) => Outcome::Success(Conn(database)),
            Err(_) => Outcome::Failure((Status::ServiceUnavailable, ())),
        }
    }
}
impl Deref for Conn {
    type Target = PooledConn;
    fn deref(&self) -> &Self::Target {
        &self.0
    }
}

We impl for our Conn a FromRequest so that Rocket can use it as a guard, and Deref, so that we can access the underlying MongodbConnectionManager and transfer it to the PooledConn.

Now, let's create a init_pool() function to fire up the pooled connection

pub fn init_pool() -> Pool {
    dotenv().ok();
    let mongodb_address = env::var("MONGODB_ADDRESS").expect("MONGODB_ADDRESS missing");
    let mongodb_port = env::var("MONGODB_PORT").expect("MONGODB_PORT missing");
    let database = env::var("MONGODB_DATABASE").expect("MONGODB_DATABASE missing");
    //let mongodb_user = env::var("MONGODB_USER").expect("MONGODB_USER missing");
    //let mongodb_password = env::var("MONGODB_PASSWORD").expect("MONGODB_PASSWORD missing");

dotenv.ok() is all we need to transfer the variables from our .env file to the environment.

Then we retrieve each variable with env::var(). We use expect() here, because in this way the server will panic at start up, sending us the message with which variable was not found.

Notice two things: the part that is commented out is relative to the MongoDB user and password: in a local development they are not needed, but they will be necessary if we use the Atlas service, for example, or if we want to secure it for online access (spinning up our server in a virtual server somewhere, and the MongoDB instance in another server, elsewhere, as is good practice).

Second thing: the .env file! (to be kept possibly at the root of the project)

MONGODB_ADDRESS=127.0.0.1
MONGODB_PORT=27017
MONGODB_DATABASE=users
MONGODB_USER=me
MONGODB_PASSWORD=password

This is just an example, for use in a local setting. However, it is good practice to add the file to the .gitignore. That is why you will not find this file in the repo.

Make sure to adapt the .env file with your info, if you are using a database service.

Back to our src/data/mongo_connection.rs:

    let manager = MongodbConnectionManager::new(
        ConnectionOptions::builder()
            .with_host(&mongodb_address, mongodb_port.parse::<u16>().unwrap())
            .with_db(&database)
            //.with_auth(mongodb_user, mongodb_password)
            .build(),
    );
    match Pool::builder().max_size(64).build(manager) {
        Ok(pool) => pool,
        Err(e) => panic!("Error: failed to create database pool {}", e),
    }
}

At the end all this is all boilerplate. Notice the authorization part commented out here as well.

Let's see how src/lib.rs has changed since last time:

#[macro_use] use rocket::*;
use rocket_contrib::serve::StaticFiles;
use rocket_contrib::helmet::SpaceHelmet;
pub mod routes;
pub mod data;

pub fn rocket_builder() -> rocket::Rocket {
    rocket::ignite().attach(SpaceHelmet::default())
    .mount("/", routes![routes::ping::ping_fn])
    .mount("/api", routes![
        routes::user::user_list_rt,
        routes::user::new_user_rt,
        routes::user::info_user_rt,
        routes::user::update_user_rt,
        routes::user::delete_user_rt,
        routes::user::patch_user_rt,
        routes::user::id_user_rt
    ])
    .mount("/files", StaticFiles::from("static/"))
    .manage(data::mongo_connection::init_pool())
}

This is the whole file! As you can see, we do not need anymore the Users struct, as in order to save our users we are employing not a shared state, but a proper DB.

Therefore, we took out .manage(Users::new()) and in its stead we are managing a Pool object, by init'ing it with data::mongo_connection::init_pool().

And really, that is all we need.

Connect the routes to MongoDB

Now let's work on our routes inside src/routes/user.rs.

Here we could go about with me explaining every little choice in the code, and why... But I feel this will not be profitable for your learning. You have the whole code in the repository, so I feel confident you could just copy&paste it!

However, I think that you are reading this tutorial in order to learn something useful to apply to your projects, not in order to learn to copy the code. Isn't it?

So let's start to see our use section:

use rocket::*;
use rocket_contrib::json::{Json, JsonValue};
use rocket_contrib::json;
use rocket_contrib::uuid::Uuid;
use rocket::response;
use rocket::http::{ContentType, Status};
use rocket::response::{Responder, Response};

use r2d2_mongodb::mongodb::bson as bson;
use r2d2_mongodb::mongodb as mongodb;

use bson::{doc, Bson};
use mongodb::db::ThreadedDatabase;
use mongodb::coll::options::{ReturnDocument, FindOneAndUpdateOptions};

use crate::data::db::{User, InsertableUser, ResponseUser, UserPassword};
use crate::data::mongo_connection::Conn;

We do not need anymore use crate::Users, neither the rocket::State. In fact our Conn already implements a FromRequest.

What we need in addition to what was already there, is:

• doc! macro and Bson type, from bson; however, we are not using the bson functions from the bson crate, but those exposed by r2d2_mongodb::mongodb::bson, because the r2d2_mongodb version is not up to date with the bson crate, and can give problems. Same thing as mongo, we are using the crate version exposed by r2d2_mongodb.

• As we already said, we are using the mongodb crate exported by r2d2_mongodb as well. Thus, we alias this to mongodb, and we get the ThreadedDatabase, which is used by r2d2's pool; we import also some Options types for use in calling the MongoDB main functions.

• In addition, we'll use the crate::data::mongo_connection::Conn recently created.

Next we define a constant for the collection name:

const COLLECTION: &str = "users";

We should get it in each function with the env::var(), and that is the professional way to go about it, but for now let this suffice (feel free to modify your code though, which is a useful exercise).

We impl an additional error for ApiResponse, because we will use it a lot.

pub fn internal_err() -> Self {
    ApiResponse {
        status: Status::InternalServerError,
        message: json!("Internal server error"),
    }
}

Let's face it: if the connection between our server and MongoDB fails, we'd better not let this out to the world, but send a "stupid" 500 "Internal server error". Better this than explain, and in doing so, expose info that could potentially lead to security breaches.

Let's start with the first route:

#[get("/users")]
pub fn user_list_rt(connection: Conn) -> ApiResponse {
    let user_coll = &connection.collection(COLLECTION);
    match  user_coll.count(None, None) {
        Ok(res) => ApiResponse::ok(json!([res])),
        Err(_) => ApiResponse::internal_err(),
    }   
}

We get connection: Conn as parameter. Then we select the collection with &connection.collection(COLLECTION) and assign it to user_coll.

Then we need to count how many documents (user records) are there in the collection with .count(). The function takes two Options to which we pass None to stick with the defaults.

We match over the Result as we said earlier we would often do, and we either return the count inside a vector, or an "Internal server error", with ApiResponse::internal_err(). The reason is that the .count() method fails mostly because of connection errors, so we cannot even test over it.. However, we are not off the hook because of connection errors: we need to make sure that when it will fail (and someday it will, if the service gets used a lot), we'll fail gracefully. That is why at each possible unforeseen fail you'll see a ApiResponse::internal_err().

count() is one of the many calls we can do over a collection. Check out https://docs.rs/mongodb/0.3.12/mongodb/coll/struct: there is a complete list.

Of all these calls, for our routes we need: insert_one() to insert a user into the collection, find_one() to retrieve a user, either by its id or one of the field's value, find_one_and_delete(), to delete a user.

We'll need also find_one_and_replace() to update info on the user. In fact it's easier to replace a user with an updated version of it, than update it, with find_one_and_update(). The problem is that in order to update info, we use custom functions, such as hashing the password, and update the timestamp.

By the way, we need to modify fn update_password() and fn update_user() a little, in our src/data/db.rs. In fact, we will not simply update the original User, but we need to return the updated User:

pub fn update_password(&mut self, password: &String) -> Self {
    self.hashed_password = hash_password(password, &self.salt);
    self.updated = Utc::now();
    self.to_owned()
}
pub fn update_user(&mut self, name: &String, email: &String) -> Self {
    self.name = name.to_string();
    self.email = email.to_string();
    self.updated = Utc::now();
    self.to_owned()
}

Now it updates itself, and it returns the updated version, so we can use it to replace the older one inside the MongoDB.

Now for example, inside update_user_rt(), that updates name and email, first we search for the user with user_coll.find_one() then, after much match-ing, we parse the user inside the collection back to our struct User with bson::from_bson(Bson::Document(found_user)) (which needs itself other matches).

We check also that the password to authorize the change is correct and at the end of this process we update the User with the found_user.update_user(&user.name, &user.email). Then we can replace the user in the MongoDB with this one, using find_one_and_replace()

Things to notice:

• find_one() returns a Result<Option<Document>>; the Result is needed to check that the connection didn't fail (which we match with an Err(_) => ApiResponse::internal_err()). The Result wraps an Option, because in case the operation did not fail, it can happen that we are searching for a document that does not exist (think of wrong user id, for example); so if there's no such Document, the find_one() returns a None. If that happens, we match the None with ApiResponse::err(json!(format!("id {} not found", id))).
• Notice too, that find_one() accepts a BSON document (created with doc! macro) to specify what's been sought for. In this case the document is a JSON object that sets _id to the id of the user. Since we need the default options, we pass to it also a None as second argument.
• once we found the Document we need to parse it from Document to BSON, and from BSON to our User. In order to do so, we need to give hints to the compiler, especially that we need to parse a User; and we need to match it, because bson::from_bson() can possibly fail (if the document is malformed, again usually because of connection errors). So the hint we give to the compiler is Result<User, _>. We use _ because we don't care here about the type of the Result.
• the inverse function, bson::to_bson() can also fail, so we need to match it as well (well, if this fails it's not a connection error, but probably a bug... but we match over it nevertheless).
• find_one_and_replace() can return either the original document, or the updated one. We need this last one, because we need to return the info, as it was this our API signature. So we construct first a FindOneAndUpdateOptions, and we set its return_document field to ReturnDocument::After (it must be wrapped in an Option).
• once we get back the updated User, we need to parse this again with bson::from_bson, and match everything. After this, we need to return not the User (with all its fields, such as updated, salt, and hashed_password), but a ResponseUser with our trusty from_user().

All this is translated to the following code:

#[put("/users/<id>", format = "json", data = "<user>")]
pub fn update_user_rt(connection: Conn, user: Json<InsertableUser>, id: Uuid) -> ApiResponse {
    let user_coll = &connection.collection(COLLECTION);
    let id =  id.to_string();
    match user_coll.find_one(Some(doc! { "_id": id.clone() }), None) {
        Ok(find_one) => {
            match find_one {
                Some(found_user) => {
                    let found_user_doc: Result<User, _> = bson::from_bson(Bson::Document(found_user));
                    match found_user_doc {
                        Ok(mut found_user) => {
                            if found_user.match_password(&user.password) {
                                let insertable = found_user.update_user(&user.name, &user.email);
                                match bson::to_bson(&insertable) {
                                    Ok(serialized) => {
                                        let document = serialized.as_document().unwrap();
                                        let mut opt = FindOneAndUpdateOptions::new();
                                        opt.return_document = Some(ReturnDocument::After);
                                        match user_coll.find_one_and_replace(
                                            doc! { "_id": id.clone() },
                                            document.to_owned(),
                                            Some(opt)
                                        ) {
                                            Ok(updated_one) => {
                                                match updated_one {
                                                    Some(updated_user) => {
                                                        let updated_user_doc: Result<User, _> = bson::from_bson(Bson::Document(updated_user));
                                                        match updated_user_doc {
                                                            Ok(updated) => ApiResponse::ok(json!(ResponseUser::from_user(&updated))),
                                                            Err(_) => ApiResponse::internal_err(),
                                                        }                                                        
                                                    },
                                                    None => ApiResponse::err(json!(format!("id {} not found",  id))),
                                                }
                                            },                    
                                            Err(_) => ApiResponse::internal_err(),
                                        }
                                    },
                                    Err(_) => ApiResponse::internal_err(),
                                }
                            }
                            else { ApiResponse::err(json!("user not authenticated")) }
                        },
                        Err(_) => ApiResponse::internal_err(),
                    }
                },
                None => ApiResponse::err(json!(format!("id {} not found",  id))),
            }            
        },
        Err(_) => ApiResponse::internal_err(),
    }
}

As you can see, lot of code, with 7 match, and one if to take care of.
See if, by checking the code with the explanation above you can understand what's being done.

The rest of the routes

If you understand the working of the code above, you'll get as a breeze both info_user_rt, which gets info on the user with the id, or id_user_rt, which gets the same info but searching by email instead.
Both of them use find_one(), the first with doc! { "_id": id.clone() }, and the second with doc! { "email": email.clone() }.

As for the other routes, these are the major things to notice:

• new_user_rt() first gets a User from the InsertableUser, then parses it to BSON. Once serialized to BSON, it transforms it to a MongoDB Document, and passes it to the insert_one() collection's method. This method does not return the user just inserted, but a Result - wrapped record containing the inserted document's id as inserted_id (or a None if it was not inserted). At this point, we use the inserted_id to search back the user with find_one(), and we follow from now on, the same pattern we have used before with the other routes that get info on the user (check the update_user_rt above).

• with delete_user_rt() we need first to find_one(), and then match the password (as with update_user_rt). When we have cleared that the password is correct, we use find_one_and_delete() which works just as find_one(), and returns also the recently deleted record At this point we end up following the same pattern as with find_one().

• as for patch_user_rt(), with which we update the password, we match the presence of the new password (otherwise it's useless, and we can abort); then we find_one() and match the original password. Once we are authorized (with User.match_password()), we get an updated user, and we reinsert it with find_one_and_replace() as we did to update the user info. We do not need to return the info on the updated user in this case, but just the message "Password updated".

And this is really all there is.

We can now run cargo test and proudly see all tests passing (if everything went smooth), even the new one in the tests/persistency_test.rs.

Conclusions

It has been a lot of code, I hope you can navigate through the ocean of match to get to what is the core of the functions.

Next time we will improve something more, and upgrade our security a notch.

Stay tuned!