This post covers design choices, a few implementation details, and problems encountered while creating club9.com.

Edited on 2020-04-16: Club9.com has been sold, so all links to club9.com have been removed. The content on this post may not reflect the current status of club9.com and I am no longer affiliated with the site.

Idea

I heard about the Rust programming language, tried it out, and it immediately became my new favorite programming language. I wanted to become more proficient in the language, and the best way to do that is by making real-world applications. A visual component for my application was something I definitely wanted, and I decided that the web was the way to go. This resulted in needing to become knowledgable in the various technologies surrounding the web such as database, UX, and design.

Once I decided on a Rust-powered web application, I had to come up with a project using all the technologies I wanted to cover. I was a full time college student most of the time (some semesters I had scheduling problems) and I was also working 40 hour work weeks to pay the bills. This meant that managing the scope of the project was of paramount importance.

I eventually settled on creating a web service that monitors site availability. I felt this project would be large enough to cover the aspects I wanted, as well as remain small enough to be able to squeeze into my busy schedule.

Goals

• Learn at least these things:

  • Rust
  • SQL
  • HTML/CSS/Javascript
  • Build systems
  • UX and UI design
  • Application design

• Create a working application

• Create an actual product

Tech Choice: Servers

I wanted to make sure I knew how to manage cloud servers (I've only done on-site previously), so I chose AWS as my provider. There were other options I looked at such as DigitalOcean, but since AWS is used by a huge portion of the internet, I felt it would give me more transferable skills.

Tech Choice: Database

I love open source, so I immediately installed PostgreSQL. I had worked with relational databases in the past, but never really committed to learning SQL, so this was a great opportunity. I avoided using a document or object database because learning SQL was a major goal for this project.

Tech Choice: Backend

All backend programs were to be coded in Rust since learning the language was a major goal.

Tech Choice: Frontend

100% feature parity across all screen sizes is one of my personal goals that I apply to all projects. Nothing irks me more than visiting a website on my phone and half the features decide to take the day off because my screen size isn't accounted for. This is especially crazy considering 54% of web hits are from mobile devices.

I was also aiming for a "native app" experience. I didn't want to replicate how a native application looks and feels, but I did want it to be snappy and responsive without any delays that are typical on a website.

User-facing Application

I spent a considerable amount of time looking at frameworks and other frontend-y things before eventually settling on Vue.js. I felt like Vue sort of fit the way I think in terms of designing an application. Many of the frameworks I looked at have their own individual way of doing things and try to nudge development style into their particular vision. But the Vue framework stays mostly on the sidelines and lets me focus on actually making a product, instead of forcing me to jump through hoops. However, my biggest reason for picking Vue is the simplicity: something that is conceptually simple is also fairly simple to implement in Vue.

Application Visuals

I had a very specific idea in mind as to how my application would be structured and how it would look visually. Chances were that I would just end up making custom everything, but I took a look at some CSS libraries anyway just in case I saw something that would save some time. I checked out Bootstrap, but quickly decided that it would be too much work to integrate my ideas. Bootstrap seemed like a great library, however I was specifically making an application and I didn't want it to look like I used a template. Plus, there is never anything wrong with learning new things.

Opinion: The Good

I spent a decent amount of time planning out database relations. I hadn't made something on this scale before, so I probably over-engineered it a little in some aspects. There are a few small "cache" tables with duplicated data that are dedicated to frequent specific queries. However, the remainder of the tables are all in 4NF.

As mentioned previously, I had only managed on-site servers (including one I had built from the ground up, so I knew it inside and out.) However, this was my first cloud deployment so there was lots of documentation to go over before I felt comfortable launching on AWS. After many hours reading the relevant materials (and making sure I fully understood the billing aspects) I was able to configure everything needed to run the service. This included Route 53, EC2, RDS, and S2 as well as setting up appropriate roles.

Getting my Rust code to run on AWS was trivial. I just set up a Docker container on my development box using the same OS version as the image running in the cloud. I used the Just command runner to automate builds, testing, packaging, and deployment. This tool made the development experience extremely fast and easy.

Flexbox and CSS grid made website creation a breeze. My last production site I authored was prior to HTML5, so everything was delicately floated around. Creating a layout and implementing fully custom designs was near painless thanks to flexbox and grid.

Opinion: Database Queries

There is a query builder for Rust called Diesel, but I opted against using it since I specifically wanted to learn how to write SQL queries. This was the first mistake, but it did not become apparent until later in development. I had to change a few tables and overall architecture at one point, and this caused problems with the hand-written queries. Since I wrote them manually, they were not checked by the Rust compiler, and I had to re-verify that they all worked correctly after refactoring. There were tests for all queries at this point in development, however fully testing a query is non-trivial and I needed to ensure they were working as expected. Had I used the query builder, the compiler could have automatically checked everything for me and I wouldn't need to write a majority of the tests.

The second issue I ran into was the repetitive nature of queries for a web application. The vast majority of the queries were pretty basic CRUD, which aren't at all useful for learning. I should have used the query builder for the simple things and then spent time manually writing the more complicated queries. This would have provided a better learning experience, as well as faster development time. In addition, I wouldn't need to write tests for the simpler queries since the compiler would catch any problems.

Opinion: Build Problems

This is a small issue but I feel it is worth mentioning. I run Arch Linux on my machine and keep it updated regularly. Arch Linux is known for being on the "bleeding edge" in terms of getting software updates out to users very shortly after they are released by upstream developers. The problem with this is Arch Linux would have the hot new version of a package, a Rust package would take a while to follow, and the AWS docker image was on some ancient version. This led to issues with multiple versions conflicting with one another,

I eventually got around this by specifying exact versions of shared libraries to use on every build command for every target. It was a hassle figuring out the correct versions to use, and I broke my own box at one point trying out different versions, but this did fix the issue and it was easy to update when new package versions were released.

Opinion: Frontend

There's no way to sugar-coat this: making a single page web application is a frustrating mess. However, looking at this from a business and implementation point of view, the web is still the way to go:

• A web application will run on anything that runs a web browser.
• Phone and desktop support with only a single codebase.
• Only one version of the application needs to be made, with a few caveats:
  • Multiple screen sizes need to be accounted for
  • Small browser issues
• Easy to make things look nice.
• Custom elements don't require rolling manual drawing code (to an extent)

I'm optimistic that the primary pain point (Javascript) will be greatly alleviated with webassembly. Given how nice flexbox and CSS grid are, I feel that this will complete the transition of the web into a true application platform.

Service Applications

I initially started simple with just a single job server handling the entire service. Once more features were added, the job server was split into several microservices in order to reduce complexity and so I could spin up new servers later if needed. Here is a listing of the applications that run the service (minus the web application):

• Site Probe
  • Checks websites for availability and files site statuses.
• Badge Generator
  • Creates the badges users may put on their site.
• Alerter
  • Reads site statuses filed by the Site Probe and sends an alert if needed.
• Stripe Endpoint
  • Gets billing updates from Stripe when subscriptions are updated.
• Admin Notifier
  • Sends push alerts to my phone for various things I need to know about regarding service status.
• Cleanup Daemon
  • Does nothing at the time of writing, but will eventually archive old data.

Other tools were written to help manage the system:

• DBTool
  • Bare-bones CLI tool for manually managing the service (account adjustments mostly).
• GenKey
  • Creates crypto keys used by services that require them.
• Migrate
  • Migrates the database if any table changes occur. This is mostly an empty CLI application with convenience libraries to make migrating as easy as possible.
• Spec Generator
  • Walks the application output directory and uses a Tera template that gets rendered into an RPM spec.

A few libraries were also created:

• DBQuery
  • All database queries (except for those use by Migrate) are centralized in this library. This helps keep things organized when changes are made + makes it easy to test.
• LibStripe
  • This handles all communication with the Stripe payment processor. There are other existing libraries for this, but I ran into issues with all of them.

Final Thoughts

Working on Club9 was a lot of fun and a great learning experience. The scope of the project probably could have been reduced by not including "make a real product" as a requirement, however I think this was beneficial since it gave me a clearly defined goal to reach instead of just "learn some things". I would have made a few different choices given a second round, but I consider the overall project a success.

Photos/Screenshots