Ask HN: Why can't the UK Gov petition site scale to use?

3 points by sammorrowdrums 5 years ago

So throughout the day, there has been a huge rush on the petition to stop UK exiting the EU, and we've seen a lot of gateway 503 errors (the default nginx page), and "down for maintenance" pages - and I was wondering if anyone could shed any light as to how it has been so hard for them to cope with scaling...

https://petition.parliament.uk/petitions/241584

Article about the crashes:

https://www.independent.co.uk/news/uk/politics/brexit-petition-revoke-article-50-theresa-may-deal-extension-a8832851.html

Traffic has been about 2000 users / minute I believe, and users each get sent an email, and then have to click a confirmation link - and you can see the current number of signatories so I guess they are probably looking at more like 4 - 5000 users peak across a minute so hundreds of thousands an hour roughly.

They also have JSON api that various people are querying `curl https://petition.parliament.uk/petitions/241584/count.json`

What I really want to know is, why could they not just "turn up the juice"?

gtsteve 5 years ago

I struggle to say why not. I just checked and it's hosted on AWS Ireland which should have the capacity to deal with any level of traffic.

I noticed that dig-ing it returns two IP addresses, indicating an elastic/application load balancer is in use as opposed to a CDN. That means every request is being served off their own instances so that's not great.

A CDN would certainly help; you can serve all paths from the CDN cache with a 1s TTL except the pages where you post to the signature page.

But why could they not add more instances to the pool? Possibly the data or cache layer was overwhelmed; I'm sure the application layer can scale just fine.

And why didn't they design it right in the first place? This level of interest isn't normally seen so they probably never did any load testing scenarios of this type. They do have their own development team but I'm not sure what sort of budget they have.

  • sammorrowdrums 5 years ago

    Thanks! That all sounds about right. I suspect it's true that they could have done better but I guess they were just taken off guard.