Workaround for 'too many certificates already issued'
The time when I found out the hardway that you can only renew a LetsEncrypt certificate five times a week.
Rate-limits and LetsEncrypt
I didn’t know this until I hit it but LetsEncrypt has rate-limiting [1] in place to limit how many certificates and renewals a domain/user makes. This is important to know because you can easily accidentally hit these rate limits when setting up some services and if you do… Then you have a painful wait.
This post is a story about me hitting one of those LetsEncrypt limits and how I managed to avoid waiting seven days to continue my work!
The most important thing you need to know though: LetsEncrypt has a staging environment [2] which has reduced rate limitations to allow you to develop with it. You should not experiment with the production service because of those rate limits and the limited resources needed to generate those certificates.
[1] https://letsencrypt.org/docs/rate-limits/
[2] https://letsencrypt.org/docs/staging-environment/
Hitting a Renewals Limit
I was working on a web service that wasn’t a typical webserver set up, it was partially in and out of a container set up and so the default LetsEncrypt set ups didn’t apply. It was also given only a subdomain that was generated for it that was out of my control. I needed to customise a bit the LetsEncrypt certificate auto-renewal process so I was experimenting a bit in production with the subdomain certificate generation and renewal.
It took a few certificate renewals but I had everything set up. For a final test, I deleted all the local certificate data (the private certs) and triggered one final certificate renewal.
Oh no.
Instead of getting my renewed certificate, an error message came back instead:
too many certificates already issued for exact set of domains
Reading the documentation, I found the following:
Renewals are treated specially: they don’t count against your Certificates per Registered Domain limit, but they are subject to a Duplicate Certificate limit of 5 per week. Exceeding the Duplicate Certificate limit is reported with the error message too many certificates already issued for exact set of domains.
Oh no. Only 5 certificate renewals per week! What have I done!
I ask a colleague for advice and he suggests revoking the current certificates to see if that helps. It doesn’t work. Reading further into the documentation I find the following:
Well what if I revoke the certificates and get back to the start? Revoking certificates does not reset rate limits, because the resources used to issue those certificates have already been consumed.
Oh no. I’m stuck.
The Panic Sets In
I start googling around and find help threads like [3]. It looks like the official response to the issue is to tell people to wait seven days for the rate limits to reset and that there is a staging environment for a reason.
This isn’t great for me, this isn’t a personal domain or web service, it’s for work!
The Work-Around
I keep investigating, learning that the rate limit is in place because the certificates are generated using hardware tokens and so there’s limited resources. As I re-read the error message, I realise something.
exact set of domains
The key is in the error message. This error comes back because I’m trying to renew a certain set of domains.
While the domain is not in my control, I did have actually two domains (both subdomains) actually issued and resolving to the webserver. I was originally only trying to issue the certificate for one subdomain.
The thought hit me: what if I change the set of domains by adding the second one to the certificate request?
It works! LetsEncrypt gives me a brand new certificate. It looks like the rate limit is exactly tied to the domains requested for the certificate. And with that I can publish the web service and put it into production.
In the end I learned my lesson. LetsEncrypt has reasonable rate limitations and if you want to play around with it, you need to use the staging environment. The work-around is not one I will ever use again and I hope publishing this post doesn’t lead to it being abused. But I thought it’s too fun to keep to myself.
Note that the staging environment allows for 30,000 certificate renewals per week compared to the 5 in production!