Part 1 of the Real-Life IT Horror Stories Series
Editor’s note: In celebration of Halloween, we’ve asked a few of our instructors to share some of the horror stories from their own consulting careers. This four-part series includes tales of espionage, employee sabotage and website theft. Check back each day for a new post. Read on if you dare.
Almost every IT administrator, whether they work in storage, networking, servers, or clients, has a story about the Domain Name System (DNS) not functioning properly. I collect DNS horror stories like some people collect bottle caps.
The Case of the Disappearing Domain
I was working at a start-up company that had grown from a dozen people to about 80 employees in a couple of years. It was a busy time, Web traffic was picking up every day. And then one Wednesday, we noticed a dramatic drop in incoming email, and the hits on the website dropped like a rock. But not rock bottom. Frantically, I tested from the outside and inside. Everything was fine.
The vice president of engineering called me. Some important quotes and contract negotiations from our customers weren’t coming through. I sent test messages. All of mine went through. The chief financial officer stopped by my office. He had just gotten off the phone with an investor. Our website was down, and we weren’t receiving emails from them. Then the CEO was in my office.
My sweaty fingers were wildly typing frantic magic incantations, desperately struggling to resuscitate our presence and existence in the networked world. We were slipping away. Hour by hour it was getting worse. And then nothing. Our network communications flatlined. All inbound communications had ceased. Yet we could still ping out. No one could get into our business. We were dead.
That was the painful day that I discovered the agony and defeat that having someone buy your domain name out from under you can cause. I knew the network was solid and working at the Internet Protocol level. We could get out to the rest of the world with the exception of PTR record authentications for outbound email that had begun to fail to some destinations. I discovered that another registrar had allowed some company to register the domain name we already “owned.” Our company name was stolen out from under us.
You might not think that DNS is a dangerous business that involves theft, espionage, sabotage, or outright hacker malice with no competitive angle. I know of several other second-hand occurrences of domain theft, including people having to file certification of their right to their business name, and other cases that went to court. But I have never had it happen to me again directly. We got our domain back, though it took several days for the email and Web services to start flowing fully again. I am now sure to lock in the registration so that no one can unregister the domain names of my business or those of my clients. Please be safe and always protect yourself.
Not-So-Active Directory
Domain names are everywhere—on billboards, radio, television, web videos, Skype, Lync, business cards and any other kind of communications. To have a DNS is to be able to communicate in business and personal aspects of life. All internal computer-based business operations, and all Internet-based and personal communications depend on the Domain Name System. Even being able to log in to computers with Active Directory, depends on DNS. Servers also need to log into their computer accounts in Active Directory and that depends on a properly functioning DNS too.
A colleague of mine was setting up a new Active Directory domain. They scripted the configuration and population of the domain for a development environment to be a clone of test system so that a team of people could work with realistic test data during development. The new domain controller worked perfectly. Redundancy is always a good idea. So they were setting up another Active Directory server.
The promotion of the second domain controller kept failing, telling them that the domain didn’t exist. They tried the domain several times and had been troubleshooting, but they were running out of time. The system needed to go live. They asked me to take a look. We had 15 minutes to get the second domain controller up and get 20 client machines joined into the domain and inoculated with group policies.
I was in the middle of another build, so I talked them through the troubleshooting of the problem. “Domain does not exist or cannot be contacted” they muttered. “But it’s right here!” they said, pointing at the console of the first—and only—domain controller.
I asked them to tell me the DNS settings. They brushed off the request, claiming that they had checked it a dozen times and it worked great, while they kept trying to get the second server online. It failed again.
I walked over to their console. “Show me the preferred and alternate DNS servers on the second machine,” I asked. The settings popped up and I noted coolly, “Those are the DNS servers on the client’s network, not the development network. Show me the first server.”
They brought up the DNS settings. Both the preferred and alternate DNS settings for the first domain controller were set to the client’s network. We didn’t have permissions to register any records in their DNS infrastructure. My colleague had simply forgotten to set up an independent DNS zone and host it on our systems, then forward other requests to the client’s DNS servers. They adjusted the configuration and after a couple of hours of frustration were able to get everything running within a few minutes, just in time for the team of developers to arrive.
Always double-check your work when overtired. Always ask for help or another person to cross-check the settings when in doubt, or even when you’re sure of yourself. And always suspect DNS. I always say that “there are two things that can go wrong with Active Directory—time and DNS.”
Do you have any stories of your own? If so, please share below or on Twitter and make sure you use #ITHorrorStories.