How we made an advancement site that scales to thousands

by Hivemind Programmers • 03/28/2024 • 4 min read

Introduction

Hi, I'm Polar. I was the primary developer of our Worlds Advancement Site, one of our most prominently known public resources. The goal of this website is to provide easy access to a list of teams which have advanced past their qualifiers and have made it to worlds.

The Why & The How?

To answer the first question, manually updating spreadsheets has been the standard for far too long, its too much effort and can be easily automated with some code.

How?

We used various technologies to make this possible, most of these are standards for the majority of scalable web applications.

• PostgreSQL was used to store the data of teams, regions, and competitions.
• Next.js was used to provide server-side and client-side components which update in real-time
• SWR is a term that means "stale-while-revalidate", it's a strategy that we use to return data quickly.
• Kubernetes & Docker is how we can automatically scale containers based on load

What is PostgreSQL?

PostgreSQL is an object-relational database which we use to store all our data in, it's fast, and light weight. Our schema is very simple. We have a table for advancedTeams, championships, regions, and a counter for how many teams have advanced currently. Inside the advanced teams table for example, we have:

id (int), region (text), team_name (text), award (boolean [true/false]), advanced_off (text [award]), country (text), state (text), city (text), slot_number (int), team_number (int)

PostgreSQL is useful because its fast, but when mixed with Prisma we can make it really easy for anyone to write database calls.

What is Next.js?

Next.js is a react framework which allows us to render stuff on both the client, and the server. It's scalable, ships light, and works like a charm. We also use Next because of it's built in API routing, we don't need to create a seperate API for things like, getting the currently advanced teams for example. This is further enchanced by Nextkit which provides type safety for API responses with Zod.

What is SWR?

SWR is a confusing thing for a lot of people, but in reality it's quite simple, it's a strategy that allows us to send you some cached data, and update it later. The way SWR can be explained best is like this:

GET cached_data -> FETCH /api/... -> UPDATE ui

This allows our application to update automatically without even refreshing the page, as an SWR request is sent everytime a database update is called.

Kubernetes and Docker

Kubernetes and Docker are complex, but simple in concept. Each service we run uses a docker container

• PostgreSQL
• & the website

The way this functions performantly is it allows us to scale our containers based on load, so if we get a ton of visits to the site, we just start up some more site containers. The database does not get scaled in this process, however it is still managed by Kubernetes.

How does the updating logic work?

The updating of the site can be done manually, through editing the database, or automatically, every night at midnight, through a CRON job.

What is a CRON Job?

A CRON is a way to scheduele a task at a certain time, in our case we call our update endpoint every night at midnight.

The logic behind it

• Authorize, we first make sure that the authorization on the request is valid before attempting to update.
• Request, then we make a request to the FTC-API, there is an endpoint with every competition and their slots.
• Parse, then we parse the body, if it fails to parse we return.
• For Each each competition on this response, make a request to FTCScout to get more event details on each competition
• Add the competition to our database, or we can make an update call to mark it as completed.
• For Each again, to add the advanced teams, we make a request to the FTCScout API to get more team information
• Add the team to our database
• Count the teams advanced we can do this simply by doing advanced_teams.count() with prisma.
• Add it to the database so we can update the graph and progress bar on the home page

That's all for now!

And that is how we made our advancement site that scales to thousands of users per day.

If you have any further questions, feel free to ping me in the FTC Discord, @isitpolar.