For many many years we ran our own DNS servers using PowerDNS with MySQL backends. That's all fine and well and it's a very powerful architecture and relatively reliable and strong to even high amounts of traffic... that was until we started regularly seeing 10Gbps+ DDoS attacks, so we put a DDoS protection service in front of the sites, but there wasn't really anything at the time capable of protecting the DNS servers as well.
So a few months go by, everything has been fine and then another DDoS hits. Looking through the web-servers, not hitting there, all is fine. Study firewall traffic through the network and note that the majority of incoming traffic (that was making it through to the network that is), was headed to the DNS servers. The attackers were sending the DDoS to the DNS servers, requesting some of our root domains and given that it appeared as valid traffic there was not much to be done to filter any of it. The easy solution in this case happened to be blocking all Chinese and Russian IP's for a couple days which mitigated enough of it to solve. After that, we stopped hosting our own DNS.
Moral of the story: DNS services are a great learning tool to use and PowerDNS is probably where you want to start, but at scale you may or may not want to actually run your own.
(Thanks for the links you provided to the front ends in your other comment).
Would like to point out that ddos is a ymmv item obviously for anyone scared off by the potential for this which is the same as the potential for any ddos attack and mitigation. A company affiliated with ours has been managing dns for a customer since 1996 with 10k zones. Back when you had to read Cricket Liu Oreilly book. (movie.edu if anyone remembers). They've never experienced a problem. All run on pretty low powered hardware for that matter.
That said there are things to know about dns while not difficult time could be better spent elsewhere if there is no compelling reason to diy.
DDoS risk is based on the type of business you're running (online casino, ecommerece, etc) and not really much to do with if you are running infrastructure in house or not.
For that matter, I don't know of a reputable DDoS mitigation service that doesn't provide DNS hosting as part of the package. It's pretty core to the whole thing.
Indeed, but what I was mostly getting at is that you have another possible attack vector if your run your own DNS, while services that provide it for you are basically pennies on the dollar compared to potential losses.
"To host your own DNS servers the registrars require you to list two IP addresses "
Registrar here. This may be a requirement of some registrars but not a requirement of all registrars. And in any case you can fake this (see warning) by simply using the same IP address with a different host name or by using a different IP address with a made up IP address.
Warning: Obviously you should have two valid DNS servers no doubt. I am simply pointing out that there are work arounds to this especially if you are trying to get up and running and won't have the 2nd dns operational for a short period of time you can at least get started with DNS that should work (since it only takes one and assuming that one is working you will be serving up records.)
.com .net "require" only 1 nameserver (Verisign). PIR/Afilias require two (hence the need to fake or use a real one).
Please note also that my parent comment was with regards to the statement "the registrars require you to list two IP addresses" not why that may or may not have been required by the registry.
It's possible of course that even though .com .net only require one nameserver, a registrar might require two for some reason. It's also possible (but unlikely) that even though .org requires two the registrar could take care of this by putting in a dummy record if they wanted and only requiring one of the customer.
End users don't deal with the registry. They deal with the registrar.
I also run my own domain name servers. I use PowerDNS with a MySQL backend; setting up replication between the two MySQL servers wasn't too hard, which gives me redundant name servers in two data centers. I have a few more than 32 zones, but the name services stuff has almost zero impact on server resources.
Updating and querying the name records is a piece of cake thanks to the MySQL database, and if I ever get around to finishing it, I can have my own web-based front-end for it.
Best of all, I can have a nice, short ttl on all of my entries, and PowerDNS is configured to always check the database, so I have a two minute refresh period on any changes made to entries on my name servers -- that alone has been well worth running my own. (Client: "Don't I have to wait like a few hours or a day or something for this to spread over the internet or something?" "Nope. 'Bout two minutes in most cases, little bit more for AT&T's customers.")
I definitely didn't do it right the first time, though. DNS has a few gotchas, like making sure you disable axfr requests, making sure you have an SOA record for appropriate zones, and so on. I'm probably still doing something wrong, but I don't know what it is at this point. Also, the default recommended PowerDNS MySQL setup is needlessly complex, as far as I can tell.
I don't understand why you use short TTLs. Do you care that every visitor has to wait an extra 100ms or so to see the page they came for? I personally want all my DNS records at the ISP.
Sure, Google can get away with a 600 second TTL because there is so much traffic that only one person in a million will have to wait.
I already explained why I use short TTLs. Let me try putting it another way: because it solves more problems than it creates. That's the same yardstick I use for all my infrastructure decisions, because sure 'nuff, no matter what I decide, someone is going to think it was the wrong decision and that I should care about their personal opinion.
Do I care that every visitor has to wait a little bit more every few pages or so? Yes, of course I do. But you fail to understand our business or customers at all when you think that a delay caused by a short ttl in dns is a problem that we need to address right now.
A much bigger problem for our customers is when they call or email and ask for help with some kind of hosting change -- moving onto or off of our servers or changing some service or some such thing -- and they get told that there will be a period of minutes or hours or maybe even a day or two where visitors will see either the new version or the old version and no, we can't say which.
When we tell people that, yes, we can change their mail services for them, and by the way, that change should be instantaneous for most people, they are delighted. And delighting our customers is what's most important to us.
Further, you fail to realize that most of our hosting customers are coming from services like BlueHost or GoDaddy, where their sites have probably been living on overprovisioned servers and so they get better performance with us anyway, which also delights them.
You also don't realize that our average customer, so far, is running a poorly optimized WordPress or Joomla installation, and that either of those systems introduces page delays that are far more severe than a DNS lookup. In other words, our customers are not currently people that are looking for the fastest possible website; they merely want it to be "fast" -- which usually means, "renders in about a second or less on my crappy 1.5Mb DSL" -- but mostly, they want it to always be up and they never want to worry about backups or hackers or phishing or viruses or anything else.
So, yeah, once short ttls start to cause problems -- a la an email from a customer saying, "hey, I just profiled my site, and I noticed it's taking an extra 132 ms to load because of a dns lookup, can you check into that" (a message which we haven't even come close to receiving yet) -- then that'll become a bigger issue and I'll address that instead of whatever else I'm working on at the time.
I know I shouldn't get irritated at comments like yours -- I should simply ignore them -- but they bug the hell out of me because you probably don't even realize what the effect is. You're not sharing helpful information; you're not telling me something about short ttls that I didn't already know; you're not telling me about some bug in PowerDNS caused by ttls less than 194 or that ttls less than 233 can cause issues with MySQL if you have more than pi requests/second. The only thing that comments like this do is further demotivate me from sharing information like this in the future. My thought process before commenting in a thread about server configuration is already something like, "Hey, should I bother to mention how we do this? It was a bit fiddly and there's some bad documentation out there. Someone might ask a question that I could answer. Someone might mention something I don't know about. That could be good. But probably someone's just going to tell me I'm wrong. Am I in the mood for that? Nah, not really."
Sorry for slamming you with a wall-o-text response. I had thought about the last time KeepAlive was mentioned in an HN thread before posting my comment about PowerDNS; you just happened to make me feel like an idiot for not listening to my instincts and deciding to not comment in this thread.
Don't be so sensitive. Even if jcampbell1's question is addressed to you, it's on a public comment thread so it's really aimed at a wider audience. What his question (and your response) does is point out that there's a tradeoff involved here - which of course you know, but not everyone reading might. Think of yourself as a participant in a Socratic dialogue.
I apologize for my comment. I'd have been more civil in person and certainly if I had bothered to look at how open you are in your profile.
I come from a different background. I bootstrapped a SaaS business that now has about 10 employees. I spent a bunch of time thinking about the best DNS solution for us. My conclusion was that we should go with the most expensive provider coupled with long TTLs. The most expensive provider was because of SEO reasons, and long TTLs are best for PPC conversion rates.
It's not so much wrong or right, but you should be aware of the trade-off you're making. It's either the instant gratification of getting your DNS updates propagated swiftly, or the longer term benefit of faster page loads and generating less DNS requests. Whatever your preferences are, objectively there are most certainly going to be much more page loads vs. DNS updates (which should be rare; not counting dynamic IPs which are a special case of course). I realize your users might not understand and just like the quick DNS options, but I think people should have a say over this trade-off, at least as a multiple choice question.
Please don't refrain yourself of posting things like this because jerks poninted their snarky guns at you, just ignore them.
By doing that you achieve two great things. First, you help people who genuinely need your caring comments on real world experience. Secondly, you don't feed attention whores.
Great post by the way. You saved me weeks of hair pulling by showing that once you achieve the right mindset this problem, like any other, can be realistically managed.
I use PowerDNS too, but with BIND flat files as a backend. Setting up replication is even easier than MySQL: I just rsync the flat files from a central server to my 2 DNS servers whenever I make changes. PowerDNS automatically reloads the files based on their mtime.
Yeah, I'd considered that, and there are some serious advantages to flat files. To be honest, relying on MySQL for production anything makes me nervous. But: SQL queries occasionally save my butt when needing to make lots of changes all at once, instead of having to figure out sed again or edit lots of files; and, I wanted something that would sync automatically, since I'm an idiot and sometimes forget to do something like an rsync command after a change.
Looking at the featureset of these frontends, my own implementation pales in comparison. It would need a lot of work before matching the features these offer.
I've rolled my own very basic PHP frontend to my own SQLite-backended self-hosted PowerDNS setup. If there is serious interest, I might just throw it up on Gitorious or Github as I've always intended.
As for the SSH attempts just move it to a different port, disable password logins. SSH shouldn't really be accessible on the same IP as any publicly facing service for your domain. It invites this kind of automated scanning, which is actually a huge risk unless you're on top of your patches. Also change your SSH config to misreport the OS/SSH version.
The IP for SSH access should then be heavily filtered to only SSH and the requisite ICMP packets.
Or fail2ban which puts in iptables rules to ban IPs that fail to login. The ssh scrapers try a variety of user accounts, and it is pretty easy to spot them (for example root can't login on my system from anywhere except the physical console, so trying to login as root,bin,games,demo,Etc from ssh is an instant ban.
Of course you need to not run an open DNS resolver which means 'don't do recursive lookups for anything except your trusted addresses'. I saw one setup that worked really well which was to only do recursive lookups from loopback, then client would create a vpn tunnel to the dns server, get their dns service that way. Seemed a bit extreme but it allows off site DNS service.
I removed fail2ban after it prevented me from logging in to my server from a Panera cafe. It took me a long time to figure out what was going on after logging in from a second server: someone had been trying to break in some days before from the IP that I happened to be using. So fail2ban could be a problem if you work from the road.
> So fail2ban could be a problem if you work from the road.
Your config is bad. Mine is set up for a 12 hour ban. From my logs, it's perfectly sufficient to drop all scanners (I rarely see an IP banned more than twice, and even twice is not common).
If I were to login from a place where an attacker is trying to log in, I would actually appreciate not being exposed - they might only be using networks, but they might also be using cameras and other stuff that would compromise my info.
I use fail2ban as well. I honestly don't get why people get so worked up about SSH. As long as you have a long, secure password, how is it a security vulnerability?
So split the conversation, there are little threats and big threats. I don't have 6 locks on a steel door at my house, the big threats are going to break the picture window and walk in that way. So if you're running one of the 100s of millions of non-descript servers out there then fail2ban and putting ssh on a non-standard port will keep the opportunists out of your server (at least via ssh). If they are using a zero day against you, or are waiting to capture traffic to do a plaintext attack they aren't opportunists, they are gunning for you. That is a different threat model.
Looking at it from a defense-in-depth perspective, its another layer of protection. Even if you use fail2ban and mandate key-based logins, it answers questions like "what if there is a 0day for my sshd?"
Its not a bad idea, it just happens that careful sshd practices are good enough that more layers are subject to highly diminished returns.
For a while I had one of the DEC VAXes in my collection attached to the network identifying itself as KREMVAX (old usenet joke) it was fun watching people try to break into it. (they could as guest, which had the password guest).
It was particularly fun to watch young people who were new to the scene sort of freak out at this weird architecture. Old farts were easy to spot, they would start right away with a 'show dev' or 'set host' to try to move around the network. Kind of like a hacker aquarium.
I was thinking giving any hacking attempt some sort of fake shell access would be a great learning experience, how awesome is it that other people had that thought and made it.
Port Knocking packets can have strong crypto payload. So yes, it's not really any different than using VPN. (Yes it is technically yes, but it allows you to open access from one IP address, using strong authentication for service, which still got strong authenticaiton. TOTP, private key and password.) It's just front blocking access to that tcp port without the right opening packet.
Another solution I like is adding fail2ban. It adds a rule to iptables to block the IP after a set number of incorrect attempts and you can set when to unblock it.
Thanks, that's a good tip, I'll work on that today.
I had considered Route53 but it's actually in the same ballpark as this setup, money wise, and it's less flexible for what I want it to do. I haven't evaluated Cloudflare at all yet.
I'm using CloudFlare (without the proxy/CDN feature) just because of their awesome DNS management. Haven't had any issues and it's got a really neat interface.
To further improve reliability, consider using a secondary-as-a-service (as well as your self-hosted secondary name service). This will only work if tinydns supports zone transfers (AXFR/IXFR), which as far as I can tell, it doesn't.
I don't use a self-hosted secondary. I use a secondary-as-a-service as my secondary /and primary/, which has the advantage of my VPS going down (for less than 4 weeks) wouldn't impact my DNS's uptime.
Before I picked up the Ramnode server I was using BuddyNS as my primary with the prgmr node as the stealth primary. It worked fine but it was less straightforward than I wanted.
Though the stealth primary solution involves more moving pieces, and introduces zone transfers, which are more complex than provisioning via puppet, I've had no noticeable issues.
I say "noticeable issues", because although my DNS is monitored, the monitoring isn't granular enough to detect sub-minute outages, for example.
Did you have any issues in particular with the stealth primary solution?
Not any technical issues, it was just more moving pieces than I want. BuddyNS is a fine service but having to force an update with an email seemed pretty hacky.
I'm using prgmr and buddyns as secondaries to my djbdns primary. axfrdns (part of the djbdns package) handles the zone transfer requests, and I can force a transfer with a really simple perl script.
It does. I'm doing this currently, I host primary DNS myself with tinydns, and the makefile sends a notify to the registrar-hosted slave DNS, which gets data via this daemon (with tcp server configured to only allow that IP).
Those "machines" seem huge for what they're doing (DNS services for 32 zones). They add up to more resources than all of my machines put together from just a few years ago.
I guess this is progress. It just seems strange to me.
Yeah they're way too beefy for what I'm using them for at the moment. My ongoing plan is to move various services onto these machines but it's been slow going.
I do this myself off a Linode. It's neat to understand how this stuff actually works, it expands the imagination of what's possible. I like being able to quickly stand up a subdomain for development / testing / staging. I really want to get an email server running, I think the ability to spark up email addresses and servers for automated reporting or file distribution opens up some useful possibilities.
I switched from running my own servers to GMail. It's easily worth $5/mo to not have to worry about security bulletins, kernel updates, etc. Self-hosting is fine for stuff like my personal website, but I live and die by my email.
I don't use google apps for my domains so Gmail MX wouldn't really work, plus I have some aliases that I think are maybe too complicated for Gmail. What do you think of roundcube? I haven't tried it yet, since php puts me off.
Yeah, roundcube is the only reason I have installed PHP. I really would prefer not to use PHP. I don't personally like roundcube too much. For sure it's not great, but it does still work okish. I'm usually using proper mail client(s), so roundcube is just a backup. Btw. roundcube is also the reason why I'm running MySQL too. I'm using Python 3 & MongoDB with all of my own apps.
What aliases could be too complicated for Gmail?
Roundcube is an OK web email client, but let's be honest, it doesn't even match up what Gmail can offer you, in terms of integrated services like Google Docs, free phone dialing, video conferencing, etc.
"To host your own DNS servers the registrars require you to list two IP addresses with the idea that you'll be providing redundant service. The one thing you don't want is downtime with DNS, it screws everything up."
I've got around this by having two hostnames for one machine.
The way I see it, since all my services are on this one machine anyway, I'll have bigger problems than DNS if the machine is down: there will be nothing at the address the names are pointing to, so who cares if they resolve properly?
Fair point. I've got email and DNS distributed onto both machines so I don't lose anything, and prgmr was pretty clear that I couldn't have a second IP just to fake out DNS so I went with a whole second server.
Also, if your email box goes down and is unreachable then the sending server should retry later when it can't get a response. If that box is also your DNS server, and the sending email server is unable to lookup the MX record it will automatically bounce. Having a slave/secondary email server should mitigate this to some extent though.
Many applications do plan for unreachable services, but throwing the fault down the stack screws up how the error is handled. You can still use a primary application server box as a DNS server, but I would be certain that it's a secondary/slave server, not the primary server.
I wasn't trying to sell you another server. conventional wisdom is that you should have your secondary DNS on a different server, on a different network, run by a different provider, not another server with me.
That sounds weird (what prgmr said). I've had dedicated servers running dns at several providers for ten years, and all I've ever done is get a second ip for the same machine and bind it to a different hostname.
Things are different now. IPv4 addresses are almost gone so ARIN requires ISPs to have valid justification for every IP address they hold. "Faking out DNS" is not a valid justification.
I think it's fairer to say you got around to it because they're not checking.
Even with a single machine it's still better to have redundant DNS. You want to be able to move around the machine to a new connection / IP, without having to wait for the registrars to update the IP of the DNS server; similarly when your one DNS provider is DDOSed, your site will stop working, but not if you had 2 different providers. So to answer your question: you as the operator will care, because you want to get things back up quickly ...
I think for small servers it's much preferable to run a hidden primary (not publicly listed in DNS), and let secondary DNS servers update from there. That way you have complete control over your DNS, even when your machine goes down.
I ran my own DNS server from my own home DSL static IP address 10+ years ago. It's fairly innocuous as long as you only respond to DNS requests for your own domain and don't forward requests. But it ends up being more hassle than its worth.
As recently as a few months ago, I hosted my own DNS server on a Linksys SLUG, but after I suffered a power outage and realized that there's no reason for hosting this stuff myself, I just decided to use namecheap.com for their Free DNS service.
Personally I would have gone with a dedicated server from somewhere like OVH, about the same price as those VPS's with similar resources (but they are dedicated to you...)
As for SSH, just move it on to a port other than 22, that will fix 99% of the bots trying to guess your password.
Route53 is a much better alternative to running your own service. I used to have a similar setup for taskup.com and other sites that I'm hosting but for the $0.20 that I pay every month to Amazon, I get weighted DNS responses based on the region that the request comes from and requests get served to the local server for that region, redundancy in 12+ zones, etc. The latency alone for a DNS lookup from India was about 300ms to our server in PA, but now it's under 30ms.
Having said that, it's a good learning opportunity.
My reading of the Route53 prices is that you pay $.50/month/domain, plus query charges after a million. How do you pay $0.20/month? I find it very irritating that Amazon says "Pay only for what you use. There is no minimum fee." when there obviously is a minimum fee of $.50/month, even if zero queries are resolved. There are entire low end VPS servers that cost only marginally more ($14/yr) than what Route53 costs for one lousy domain, i.e. one entry in a database table.
with Route53, it's not just another entry in a DNS zone file, you actually get multiple-zone IP addresses for each domain hosted, by default. Also, you get WRR and anycast, as well as latency based routing. Not to mention that you can create aliases that are not visible to resolvers but point to internal Amazon resources that you have access to.
I switched to Route 53 (from TinyDNS with custom RoR/Postgres frontend) and I'm completely satisfied. The bills are tiny, the service has been reliable, the console lets me make all our desired records without a lot of hoop-jumping, and somebody else maintains everything. Migration was a one-day project, including DR.
There are some added bonuses if you happen to be using other AWS services, but even without those, I think Route53 is one of Amazon's better offerings.
twisted.names (that's the name of the DNS package) is simple enough to use for DNS serving (demonstrated above) and flexible enough to get it to do pretty much whatever you want.
I run my own DNS, but in a rather unique way. The registrar record points to my hosting company's DNS servers, but they are slaved off my DNS server; the only IPs that can hit my DNS server are the hosting company's. I make the changes I want, they're pushed out to the actual Internet facing DNS servers.
So a few months go by, everything has been fine and then another DDoS hits. Looking through the web-servers, not hitting there, all is fine. Study firewall traffic through the network and note that the majority of incoming traffic (that was making it through to the network that is), was headed to the DNS servers. The attackers were sending the DDoS to the DNS servers, requesting some of our root domains and given that it appeared as valid traffic there was not much to be done to filter any of it. The easy solution in this case happened to be blocking all Chinese and Russian IP's for a couple days which mitigated enough of it to solve. After that, we stopped hosting our own DNS.
Moral of the story: DNS services are a great learning tool to use and PowerDNS is probably where you want to start, but at scale you may or may not want to actually run your own.