simple static gitlab review apps using nginx

The Motivation

When working on front-end projects for customers I always wanted to be able to show them the current state without having to deploy to the test environment all the time. I'd like to be able to have multiple versions, one per branch, so multiple people could work on different things but have deployed, world-reachable version to show and discuss with colleagues or customers. This would also improve merge requests since one could not only review the code but also review the deployed project in the browser without having to check out the code.

This is exactly what GitLab calls "Review Apps". It is basically using their CI environments feature and showing them in merge requests.

The Implementation

The recommended way seems to be to do this using docker & kubernetes but getting the resources for even a small cluster is not cheap and setting it up just for better code reviews for a small portion of the code is not something I wanted to do. So I went with a much simpler approach. I got a small VM, set up nginx with a wildcard vhost $name.review.example.com and configured this to serve files from /var/www/$name/public. It cannot execute backend code or proxy to other backends, it only serves static files.

The nginx config for this is pretty straightforward:

server {
  listen 443 ssl http2;
  listen [::]:443 ssl http2;
  server_name ~^(?<name>.+?).review.exampe.com$;

  root /var/www/$name/public;

  # insert your TLS config here. https://mozilla.github.io/server-side-tls/ssl-config-generator/ is a great tool!

  location / {
    try_files $uri $uri/ /index.html;
  }

  access_log /var/log/nginx/pages-access.log;
  error_log /var/log/nginx/pages-error.log debug;
}

Configuration for gzip, general TLS stuff, sendfile etc. is done globally in the nginx.conf.

The deployment works via a gitlab shell runner running on the same machine that just rsyncs the code to that target directory. The GitLab CI scripts looks like this:

build_app:                                                                                                                                                                                                                                  
  stage: build                                                                                                                                                                                                                              
  image: node                                                                                                                                                                                                                               
  script:                                                                                                                                                                                                                                   
  - npm install                                                                                                                                                                                                                             
  - npm run build-release                                                                                                                                                                                                                   
  only:                                                                                                                                                                                                                                     
  - branches                                                                                                                                                                                                                                
  artifacts:                                                                                                                                                                                                                                
    paths:                                                                                                                                                                                                                                  
    - public/                                                                                                                                                                                                                               

deploy_review:                                                                                                                                                                                                                              
  stage: review                                                                                                                                                                                                                             
  script:                                                                                                                                                                                                                                   
  - mkdir -p "/var/www/$REVIEW_TAG/public"
  - rsync -drv --delete public/ "/var/www/$REVIEW_TAG/public/"
  environment:
    name: "review/$CI_BUILD_REF_NAME"
    url: https://$CI_PROJECT_ID-$CI_BUILD_REF_SLUG.review.example.com
    on_stop: stop_review
  dependencies:
  - build_app
  only:
  - branches
  tags:
  - review

stop_review:
  stage: review
  script:
  - rm -rf "/var/www/$REVIEW_TAG"
  variables:
    GIT_STRATEGY: none
  when: manual
  environment:
    name: "review/$CI_BUILD_REF_NAME"
    action: stop
  only:
  - branches
  tags:
  - review

The stop_review task can be triggered manually or is used by gitlab when the MR is closed and the branch deleted. It just removes the docroot. The tag review is used to select the correct runner for these jobs.

Authenti-what?

This works great. It builds the code on our internal docker ci runner, puts the result in an archive, downloads the archive onto the review server and rsyncs the content to the document root. The deploy_review job takes about 3-4 seconds for most apps. From there on the public folder is served publicly via https://<project-ID>-<branch name>.review.example.com. Without authentication. That's easy, and may not be a problem for some cases, but it definitely does not feel good when you deploy projects developed for customers there, maybe even containing real-world test data.

So some sort of authentication was required. Since the ease of use was one of the core features on this, hiding it behind a complex SSO system was not an option (also there is no such system currently in use here). HTTP basic auth would be the perfect fit but maintaining .htpasswd files is painful if the number of users gets large and it only allows for authentication. But when we have authentication why stop there? We could add some authorization to the mix so we can allow only a certain set of users to access any given project.

nginx subrequests to the rescue

Thankfully there's an app nginx module for that. The ngx_http_auth_request module allows you to ask a third party to authenticate the user. On the server side, nginx creates a subrequest to that third party and checks the response status code. If it is 200 it continues to serve the clients request, if it is 401 or 403 it returns the same error to the client, passing through the WWW-Authenticate header for 401 results. Every other status code is considered to be an error (HTTP 500 for the client).

The additions to the nginx config look like this.

server {
  # ...
  location / {
    auth_request /_remote_auth_/;
    try_files $uri $uri/ /index.html;
  }

  location = /_remote_auth/ {
    proxy_pass https://auth.example.com/remote_auth;  
    proxy_pass_request_body off;  
    proxy_set_header Content-Length "";  
    internal;
  }
}

We added the auth_request directive to the main location context and added another, internal location context to proxy the subrequest to our authentication backend. Using the proxy here allows us to have some more control over the requests, for example removing the request body (if there is any) and unsetting its content length.

Overengineering is my biggest strength

So with the nginx side solved, we need a backend to answer our authentication requests. I decided to roll a small Symfony application that just answers HTTP requests on a single route and requires HTTP basic auth on it, using users from a database. This setup allows for future extensions, like checking the requested domain name against a list of allowed products or adding different storage backends like LDAP.

Building something like this in PHP using Symfony as a framework can actually be done quite quickly, it involves about 200LoC in PHP, most of what can be generated (bootstrap code, db migration, orm entities) and a few dozen lines of configuration. For convenience, we are using the EasyAdminBundle to provide a simple user interface for administrators to create and modify users.

I choose to run the authentication backend on the same server as the review app since its currently the only service using it. Deployment is easy, since we already have gitlab runner there.

deploy_to_prod:  
  stage: deploy  
  script:  
  - rsync -drv --exclude '.env' --exclude '.*' --exclude 'vendor' --exclude 'var/' --delete . "/var/www/authcenter/"  
  - cd /var/www/authcenter/  
  - curl -O "https://getcomposer.org/download/1.7.2/composer.phar"  
  - php composer.phar install --no-dev  
  - php bin/console doctrine:migrations:migrate --no-interaction  
  variables:  
    APP_ENV: prod
    DATABASE_URL: $PROD_DATABASE_URL  
  environment:  
    name: authcenter prod  
    url: https://auth.example.com
  only:  
  - master  
  tags:  
  - nginx 

The nginx config for this is not complex at all. I just used the template provided by symfony and just tweaked it to fit my PHP-fpm config. I also made use of the fastcgi_param directive to set the APP_ENV and DATABASE_URL environment variables required by the Symfony application.

I can sign in into the application using https://auth.example.com/admin and get the EasyAdmin interface to modify or create users, or use https://auth.example.com/remote_auth and just get an empty HTTP 200 response.

When browsing one of my review apps I'm asked to enter my basic auth credentials on the first request and every following request receives the same credentials because that's how browsers do this stuff.

The Discovery Of Slowness

This works, but I soon recognized that it's slow. Actually, terrible slow for something that's "just serving static files". Requests took up to 1second, sometimes even more, per file. That number should be well below 100ms! I went and tried to debug/profile the PHP application. My first suspicion was that opcache would be missing. And really, that module was not installed. But that did not fix my problem. It was faster, but not fast. I was prepared to get into the dirty work profiling this really small application on an OS level when I recognized that giant "WARNING: performance killer ahead" sign I should have seen when I started this project.
The current implementation worked as follows.

• The user's browser sends a request to the server, containing the credentials in the Authorization header.
• nginx starts subrequest, handing the credentials over to my PHP application.
• PHP queries the database for the username
• PHP verifies the password using bcrypt.
• If the password matches, it responds with HTTP 200, if not with 401.
• nginx either continues serving the file or responds with 401.

Every step of this is really cheap (the db query takes about 1ms) except bcrypt. It is even specifically designed to be slow. It does hard work to be slow. That is the general idea of a good password hashing / key derivation function: you cannot take shortcuts, you have to do the hard math stuff. And while running bcrypt once during login and setting a session cookie after that is not a problem at all, running bcrypt for every file on a stateless basic auth session is actually a terrible idea.

Bad performance? There's Cache!

While there might be a way to make nginx set a cookie for authenticated requests, I chose a solution that does require way less configuration and actually fixes my performance problem for most workloads.
Nginx can cache proxy responses. All I had to do is allow caching for that response in my PHP application and add a proxy cache to the internal location block.
The PHP code looks like this:

    public function index() {
        $response = new JsonResponse([]);
        $response->setCache(['max_age' => 300, 'public' => true]); // 5min
        return $response;
    }

The new nginx config like this:

location = /_remote_auth/ {  
  proxy_pass https://auth.example.com/remote_auth;  
  proxy_pass_request_body off;  
  proxy_set_header Content-Length "";  
  proxy_cache auth_cache;  
  proxy_cache_key $name$remote_user;  
  proxy_cache_valid 5m;  
  internal;  
}

This requires a proxy zone to be configured within the HTTP context in nginx.conf:

http {
  # ...
  proxy_cache_path /var/run/nginx keys_zone=auth_cache:10m;
}

The proxy_cache_key assures that that we cache one response per username & app up to five minutes. That allows us to serve all resources for that app with only one request to the actual auth backend, but we are still checking authentication for every app and every user once.

The Conclusion

I'm currently quite happy with this setup. The performance is not great but acceptable, we got basic authentication with user management being not-terrible and can extend this system pretty easy. There's an option in nginx to automatically revalidate stale caches in the background that might be useful, although the cache might not be the best solution for this performance problem.

Ideas on how to improve this setup or recommendations for alternatives are always welcome!

After writing all that I found (or refound I'd guess) that there is some official documentation for that kind of setup from GitLab: https://gitlab.com/gitlab-examples/review-apps-nginx.