How I run helpyourhighstreet.org for 20p per monthBy Benjamin Howarth on
COVID-19 has forced a lot of us into unemployment, or projects put on hold, but that doesn't mean us programmers can't do our bit during this time.
I'm part of a volunteer group run by Newspeak House called the Election Tech Handbook - it's basically a collection of resources for political activists to use to help run better campaigns and get better engagement from voters.
During COVID-19, this group has been re-tasked with collating and organising resources for anyone adversely affected by the virus (which is pretty much the entire planet), which is now the Coronavirus Tech Handbook.
Whilst helping out with organising the handbook, I discovered a lovely little project called HelpYourHighStreet, which is a directory of small, independent (or small chain) retailers selling gift cards and other services online.
I got in touch with the project owner, Kanishka, and offered up my skillset for contribution, which he accepted.
Q1: why giftcards?
Giftcards are a useful line of credit available to a retailer that doesn't involve them asking the bank for a loan. Instead, customers buy a giftcard which gives the retailer instant cashflow, and the promise to provide goods and/or services later on, when the customers goes to spend the giftcard value.
However, because they're technically a line of credit, if a retailer enters administration, the giftcard becomes worthless. Frankly, I think it's a risk that consumers are willing to take on their local shops that they know and love.
Q2: what's the setup?
Initially, Kanishka and Steve, the main developer, were just sending round a ZIP file containing the site's HTML, CSS & JS, and then uploading it to an Amazon S3 bucket. When I started on the project, the first thing I did was instantly set up a Git repository, established Gitflow, and pushed it into Github, to guarantee that everyone could collaborate together on the site.
When discussing where best to host this, we hit upon Github Pages - after all, the site is just two static HTML files, so I immediately ported the site over to Jekyll, a Ruby-based static site engine that Github uses to run the Pages feature. This also made it easy to create components for the site, such as navigation and footer, which made any changes to the layout easy to re-use.
When working out how to transfer the repository into an organisation, to ensure transparency and continuity in case anything happened to me, we hit a stumbling block that Github Pages is only available on private repositories for paid accounts. With 8 people on the project, this was a no-go, but I found out that Gitlab offers this feature for free - so we shifted to Gitlab.
Whilst hosting the site on Gitlab was ideal, we didn't want to irritate Gitlab's engineers with intense traffic - so we whacked Cloudflare's free plan over the front of the site as well.
Q4: a staging area?
To be able to collectively test out new features, we needed someplace that would host a staging site. Given that we'd previously used S3 to host production, I offered to host the staging site on my Azure account. I also wanted to automate deployments to this area, so I set up an Azure release pipeline, to run releases out every two hours from the develop branch. I also popped Azure CDN in front of this.
Q5: the data feed, part 1
The feed being in two different places wasn't ideal, but with some work, we also discovered that a Google sheet can be viewed as JSON (thanks Clark Ngo!).
To be able to handle map pin locations, we needed to take the address of the business and geocode it into a latitude and longitude. We used some existing code for this, so that when three cells are selected, the leftmost being the address, and the other two being for latitude and longitude, a block of JS would call the Google Maps API to return our latitude and longitude.
Q6: the data feed, part 2
The JSON that Google Sheets was giving us was extremely verbose, and weighed in at about 300kb with roughly 300 entries - less than ideal, cause this could easily extend to several MB if we got to thousands of shops in the directory. I realised that we'd need to modify the JSON going into the site, and host it someplace, ideally not in Git - and our staging environment was perfect for this. We started by using Underscore to map the JSON into a format that was smaller and easier to manage. Now, I wanted to update the data feed on staging on a regular basis, but not use the Azure pipeline for this. Enter a Node.js serverless function! The function simply takes the existing data feed, maps it into a simple POJO using Underscore, randomises the order of the output - nobody pays us to be on page 1 of the results page, so this is the fairest way to ensure that all the businesses listed are getting fairly promoted - and finally, GZIPs the entire JSON and pushes it into Azure BLOB storage (by GZIPing the feed, it reduces that 300kb of data into 107kb (and that's with about a thousand shops in there now!).
Q7: the data feed, part 3, and extra features
We want to launch the project in more countries, and by using Jekyll, we can do i18n pretty easily with a plugin or two and somebody's offered us up a collection of German businesses also doing giftcards. We also want to add Google Places photos to the results - which is an API that Google charges $7 per 1000 requests. Initially we put this on the front-end of the site, which was a HUGE mistake, as we ate about $70 of the $200 free credit Google supplies in two days, which was completely unsustainable. We'll be updating the Node.js function to pull these images into the staging area and hosting them ourselves, that way we'll keep usage of that API as low as we feasibly can.
In the meantime, the actual cost of hosting the site - serverless function, BLOB storage, CDN, and Azure Pipelines - comes to this:
No, that's not a mistake. Halfway into the month and my site hosting has cost £0.10 (about $0.14 USD). And that, folks, is how you run a highly-scaleable site on virtually zero budget.
Dear Reader, what's one of the most interesting tech setups you've worked with lately? Let me know in the comments!