Tournamentmgr.com's SEO performance has been historically poor from conception. When analyzing the number of pages google crawled, it averaged below forty per day.
Despite the fact that Tournament Manager hosts nearly ten thousand tournaments, and respectfully nearly that many unique pages, there is no differential between static pages and dynamic pages. This is due to the fact that the site runs as a single page javascript application, and its dynamic content rely solely on external API calls. After coming across a blog post, I was compelled to conduct research on this topic. The question - how can I build something that's dynamic to search engine bots, not intrusive to user experience, and budget friendly to a serverless based solution - became the center of my research.
In answering the previously stated question, I found many server-based tools, as well as some third party hosted utilities, but they either had an operational cost of maintenance, or an expense to the third party, both of which weren't options. Services like prerender.io, give you the ability to hold cache of up to 250 pages without cost, except with Tournament Manager's scale that's not ideal. Therefore, after going back to the drawing board I wrote a module (more on this later), and then containerized services to be deployed with AWS Fargate. Utilzing event-based and cron style, the services generate sitemaps, render pages, and then upload to an S3 Bucket.
Design
Pyppeteer is a python module that renders full pages, regardless of site speed. Utilizing this, I created two modules, prerender and scraper, of which read in the robots.txt file, discover sitemaps, and then loop through to determine the urls. Once complete, scraper will render these pages and return to you the static html, which can then be placed into an s3 bucket. Below you can see the sample implementation of this works from an invoction perspective.
Following the creation of the cached pages, I followed Jake Wells's blog, mentioned above, and created a simple origin request lambda@Edge function. Borrowing a prerender conditional, credited below, I was able to directionally route traffic to the cached version if originated from a webcrawler, and to the live site if from a typical user. To note: In order for this to work I had to modify CloudFront, which involved whitelisting the header, user-agent, and forwarding query strings.
Implementation
Utilizing Fargate, an AWS containerization clustering service, I created two task definitions. These definitions included an internal service that interpreted the DynamoDB database into valid XML URLs, and a prerender service utilizing the aforementioned Sitemap Prerender application. Finally, I built a serverless framework wrapper for CloudFormation to deploy my resources. As seen below this does several things, including creating the IAM roles, cluster for the services, and the task definitions.
Results
The results speak for themselves. After implementing the sitemap, rendering the dynamic pages, and routing bot-based traffic to the new urls, crawable addresses increased by 900% going from 40 to 400. Another benefit also seen is with open graph. Due to the fact the pages were static, when users would share their tournaments, tournament metadata and images would be lost. With this modification, users can now share their tournaments on any major platforms, and get their own custom icons and information to the users, better connecting them to their tournaments.
Top comments (0)