Some things I learned today about Route53, DNS, and Certificates on AWS
I was trying to deploy a stack via Pulumi to AWS. The stack included an SSL certificate managed by ACM. The Pulumi configuration included the Route53 Hosted Zone that the application should use.
When a certificate gets created via ACM, it will try adding a CNAME record to the Hosted Zone. ACM needs to validate that you actually own the domain name. It does this by adding the CNAME record and then tries to resolve it from the internet. If ACM can find the record, it proves that you control the domain.
Key finding: a Hosted Zone is not necessarily an actual domain name. You can create hosted zones for internal services. Like, you can create a hosted zone called mycoolsite.com without being the actual owner of mycoolsite.com. So, a hosted zone is not a domain name registrar. It is a DNS server configuration that manages DNS records for a specific domain, but only works if the domain is properly delegated to point to that hosted zone's nameservers.
If you make a hosted zone with a domain that doesn't exist, the DNS resolvers will return NXDOMAIN. So when ACM does to try to resolve the CNAME record, it is going to fail (or be stuck in PENDING_VALIDATION) unless that domain has proper DNS delegation from its parent domain.
Why would anyone create a hosted zone that isn't accessible to the internet (without a resolvable dns name? For internal AWS services that only need to communicate with eachother within a shared VPC.
The CNAME record was created, but the certificate was stuck in PENDING_VALIDATION status. I checked the events on the Kubernetes cluster and noticed this warning,
Error syncing load balancer: failed to ensure load balancer:
error creating load balancer listener: "UnsupportedCertificate:
The certificate must have a fully-qualified domain name,
a supported signature, and a supported key size."
Part of the stack includes a Helm chart, and part of that Helm chart is an Nginx Ingress Controller and Service (type LoadBalancer). When this gets deployed in AWS, it creates a Load Balancer. A Load Balancer has "Listeners" which are rules that define which port and protocol the load balancer accepts traffic on, and what to do with that traffic. For example, forwarding HTTPS traffic to port 443 of a backend server.
The load balancer could not create a listener because it requires a valid certificate. The certificate (that was still PENDING_VALIDATION) was not valid because ACM could not resolve the CNAME record that it created, since the domain had no proper DNS delegation chain.