Tag Archives: retrospective

Throwing GoDaddy Under The Startup Bus

Ever since we registered our startup’s domain with them  years ago, we’ve been anxious to get off the free DNS provided by GoDaddy at the least, and ideally change registrars as well.  With all the issues other companies have had with them + their political positioning … we just want out.   It’s actually embarrassing to admit we were in this situation for so long, but I’m swallowing my pride in hopes that this will help others out – open-sourced embarrassment (O-ASSMENT).  Until recent, we really haven’t had the time/resources to tackle it without affecting product development efforts and higher priorities.  One of our senior guys has been exploring options for weeks, and we thought we were in a good position to make a change.

There are two parts of this puzzle that need to be fit: GoDaddy is (was) the registrar of our domain, and they also are hosting DNS for us.   That’s a typical set-up when you first register your domain these days; most registrars also offer managed DNS.  But it’s not a good practice to leave your DNS hosted with your registrar – it’s better to separate them right when you register the domain, if you can.

The two parts (registrar and managed DNS) are intertwined; I’m trying to avoid DNS details for the non-technical, but essentially/simply/horribly put: DNS is much like a big phone book that yourDomain.com has a page in, that page maps IP addresses to friendly names like http://www.yourDomain.com and api.yourDomain.com.   One particularly critical mapping provides the IPs pointing to our authoritative name servers.  This mapping is also stored in the index of the phone book, by a higher DNS authority…like Elvis.  Servers that need to know where http://www.yourDomain.com is (in other words, its IP address) look in the index if they need to, and then get the IP from our page in the book.  This is where the registrar comes in – you can only change the IP of the authoritative name servers through the registrar of the domain.  Otherwise, with regard to DNS/WHOIS records, the registrar is just a text string, a name without a number.

But this makes registrars ultimately all-powerful; you can make all the DNS changes you want, but if the authoritative name servers are changed and pointed to hosts that don’t have our DNS information, or don’t have the right information  – you’re totally FUBARD.

We shopped around for a different registrar, and at one point were ready to sign an expensive deal with MarkMonitor, who from all accounts is the market leader in terms of locking things down from a security standpoint.  But they couldn’t seem to get their act together fast enough and were too expensive for our growth stage anyway.   We decided to go with NetworkSolutions, the “first” registry operator and registrar for the com, net, and org registries.

GoDaddy offers free DNS when you register your domain with them, but they also offer Premium DNS.  We upgraded to premium weeks ago, to get a better idea for our DNS traffic and to price out competitors.  To be totally clear, at this point in the story we’re paying GoDaddy for their premium DNS hosting option.   GoDaddy offers this to their customers as a stand-alone service; in other words, you can use GoDaddy just as a managed DNS provider (as long as you have a domain or two registered with them, I’d assume ).

So, given that we wanted to move our registrar (because we didn’t want GoDaddy to own the gate to our authoritative name servers), and our DNS, we had a few options:

  1. Try to move both at once.  Not a good-feeling option for probably obvious reasons.
  2. Move to a different managed DNS provider, then once that’s complete, move registrars.  Moving DNS is more complicated and in theory (or logically) more risky than moving registrars.
  3. Move registrars, and once that’s complete move to a different managed DNS provider.  This seemed like the lowest risk option, given all the inputs at the time, and it’s what we tried to do.

Here’s the relative timeline, what happened, and what we expected/should have happened:

  1. Our senior engineer talked on the phone with folks at GoDaddy and our new registrar, NetworkSolutions, both of which confirmed our understanding and expectation that during and after the change, the name server addresses would remain pointed at GoDaddy’s name servers until we took action to change them.  The only thing that was supposed to change was the registrar’s name.  We reiterated with them that downtime wasn’t an option for us, and they reassured.
  2. Our engineer initiated the transfer song-and-dance.  The first thing he noticed was that we couldn’t get to any DNS information in GoDaddy anymore, including the NS records.  OK….kinda makes sense to prevent changes while the transfer is happening I suppose, but we should at least have read-access to the current records, right?
  3. So he called GoDaddy, who pointed us to a page where we could access the current DNS records if we did a ‘view source’ (!), and also pointed us to a ‘pending transfers’ section of their site that would expedite the acceptance process.  No email or other instructions about this bit were previously given; this whole process normally takes place mostly via automated email, and registrar documentation on all of this is sh** across the market.
  4. Then we took a step that I’m quite glad about : we saved all of our zone file information and DNS records to a spreadsheet.   Go do this now for yourself, if you are in the same kind of situation.  Seriously.
  5. As instructed by someone at GoDaddy, we then ‘accepted’ the registrar transfer on their site.
  6. At this point, we’re thinking that our premium DNS is going to sit there untouched, and that it’s going to be five days before the registrar is transferred.  5 days because the registry operator – the root authority for the .com domain –  has that as a grace period before making the change, in case any party cancels the transfer.  Wrong on both counts.
  7. Shortly after accepting the transfer in GoDaddy’s web interface, they deleted our DNS records. We had a short time-to-live setting on the records, so after 30 minutes, hosts aren’t able to look up what IP to use for any ourDomain.com services.  The name server entries weren’t changed of course, because GoDaddy is no longer in a position to do so.  But the information sitting on those name servers that pointed IPs to our services was gone.   That meant that slowly, across the net, customers stopped being able to access services onourDomain.com – including email.
  8. Our engineer called them immediately, described what happened, and asked why our DNS records disappeared.   Answer : “because you moved your domain to another registrar“.
  9. OK, can we get that re-instated?  Answer : “I can give you access to the DNS manager page again for this domain, but you have to put all the information in yourself.”  I’m pretty sure they’re required to keep this information for some period of time, to be in compliance with their registrar agreement with ICAAN.
  10. So our engineer gets out our trusty spreadsheet, and manually copies the information back in.  Shortly thereafter we start to see a gradual recovery, as clients start to be able to resolve hostnames to IP addresses again.
  11. Hours later, the registrar transfer actually happens and is verified via email from NetworkSolutions.

Fortunately our end-user-facing product fails gracefully under these circumstances and customer impact was minimal.

That whole escapade pretty much escalated the priority of us getting off their managed DNS, which we did in the next week.   After looking at various (mostly expensive) options, we moved over to Amazon’s AWS Route53, which went relatively seamlessly.  The nice thing about Route53 is that it’s accessible programmatically and can be managed via scripts just like the rest of our AWS resources.

I totally get that the herky-jerky that comes with WHOIS-on-first; name server and DNS transfer of ownership puts registrars in an odd situation, one that requires competitors to coordinate if they’re going to act in the best interest of their soon-to-be/just-cancelled customers.  But there’s got to be a better way than this ridiculous bullsh** we just went through.  Registrars who offer DNS hosting as a service have an obligation to publish the ‘how do I get out without getting ass-f*****’ instructions at the very least. Better yet, for a grace period, leave DNS the way it is until an NS record gets changed at the root level, messaging their customers about what’s coming in the meanwhile.   I know that some registrars do provide a grace period like this.

I’m obviously not a registrar,  and admit that my proposed solutions may not be tenable.  But there’s got to be a better way.

We’re not the only startup in the bus that’s running over GoDaddy, there’s pretty much wide agreement on this topic.  I’m glad we’re over that speed-bump and the startup bus is barreling forward at high-speed as usual.

I’m tempted to turn this into an ICANN complaint – any input on whether that would hold up, or be worthwhile?  ( to comment you have to be on this post’s page, rather than the blog home page)

Update : in case it’s helpful for anyone, I’ve started gathering some numbers on what some other friend’s startups are using (without major complaint)  for registrar and hosted DNS, and will update here for now.   Please email me directly if you’d like me to add something to this list.

  • Registrars (count of companies using them)
  • Hosted DNS providers
    • EasyDNS.com (2)
    • Zerigo (2)
    • AWS Route53 (2)
Tagged , , , , , , , , ,
%d bloggers like this: