Here is the deal. When associates try to go into carrier websites or any secure website for that matter, they are met with:

403 : Forbidden

With a little research this is what I found.

Nslookup for 53.com (Fifth Third's website, it's another one that wasn't working) returned an IP address different then when I pinged 53.com

Ok, so I cleared the cache on both DNS Servers and restarted the DNS. This resolved the issue. But... it happened the next day. Same thing, same solution to fix.

Here is the layout of our PC's accessing a website:

PC>DNS cache on local machine>Primary DNS>Secondary DNS>2 Forwarders (Iserv DNS)

I am thinking one of two things:
1. A forwarder DNS issue resolving hostname to bogus IP's. But... further testing makes me think

2. Every website that didn't work I did a nslookup on and it returns the same bogus IP.

Is there such thing as a 403 Virus?

Any thoughts are appreciated!

It's almost like my queries are getting hijacked and being redirected to 64.95.64.197 (this is the bogus IP)

I've did a dns lookup of this IP and everything states that the IP is from Weltham, MA. Looks like it's a house.net domain being hosted by activeaudience/smartname.

NSLOOKUP for 53.com
Returns this bogus IP, even though 53.com's IP isn't 64.95.64.197
Multiple other sites do this as well, and it seems to be a majority of our carrier sites.

It brings the browser to a white page saying 403 Error: Forbidden

I will try and explain the situation a little bit better!

Here's the breakdown:

When I go to google.com my PC reverts to it's local DNS cache first to try and resolve the hostname google.com to an IP address. If it cannot find it on there then it will go to it's primary DNS Server (in my case it's my DC with DNS Role) if it cannot find it there it will go to it's secondary DNS Server (in my case its our backup DC w/ DNS role) if it cannot find it then it goes to our two forwarding DNS Servers (in my case it's the two IServ DNS Servers)

All in all it will find the IP for google.com and my PC will go to google.com.


However, when I try to go to some other sites (ie: 53.com, carrier websites, secure websites) it goes to a 403 Error: Forbidden page. This happens because the DNS query from my PC is resolved by one of the DNS Servers with a bogus IP.

53.com's real IP address is 216.82.178.20. But when I try to go into from a PC inside my network it resolves 53.com domain to 64.95.64.197 thus causing the PC to go to a 403 error: Forbidden page.

It does this for multiple sites. All nslookups for the sites in question are returning the same IP address of 64.95.64.197 as the resolved IP for the domain that's tested. (obvi. This is the wrong IP)

I did do some research as to where this 64.95.64.197 IP is from and its from Weltham, MA it's listed as a house.net domain and looks like its hosted from activeaudience.com (which is a webhosting company in MA). If I do an NSLOOKUP of 64.95.64.197 it resolved the IP to lender.activeaudience.com.


What I am wondering is why my DNS Servers, or the forwarding DNS Servers are resolving all these websites to the same IP?

Figured it out. It was a security update for DNS, all resolved now!

It seems that way for now anyway!

Certainly weirdness.

https://www.askmehelpdesk.com/networ...or-526690.html

THe above link is a thread I started earlier this month about the same issues.

I have figured out that we are the victims of cache poisoning and I was wondering if anyone had any input on how to go about fixing this? There are 130 employees so keep in mind simple 1 PC fixes may or maynot be out of the question.

The above link will explain my problem and what we've been experiencing! As of right now we have to go in and clear the cache for each DNS Server and restart the service and flush the local cache for the PC's to work. Ughhh this is annoying and time consuming!

Thanks!

"We have flushed the local and cleared the cache on all servers.

However you got me thinking. I can pinpoint it by doing one at a time instead of just both.

For example.

A local PC is trying to go to unum.com
They are DNS query is resolved with a rogue IP.
Usually I flush their DNS and the servers and it works, but if I do it seperatly I can narrow down the issue. hmmm am I correct?

By flushing DNS on the local machine it is then forced to pull from the DNS Servers for first time resolution and then it stores that resolved IP in its cache. If I do this and the DNS returns the same rogue IP I have narrowed it down to the DNS Server it pulled the IP from. If the DNS Server returns the correct IP (and I didn't clear the server cache) I narrowed it down to a client issue.

Does this sound right? I am thinking myself into circles"

Upon further thinking I remembered that there is a service called DNS Client that runs that does cache timing. Every x seconds (default I believe is like 87,000 ie: 1 day) the local cache is cleared and entries from the server are inputed.

The above quote (from another post) I was trying to narrow down where the poisoning was coming from, either the server or the local machine. With this service running, it will be impossible to tell because:

Even if I clear the local cache, forcing the PC to query the DNS Server, thus putting its result in its local cache. The DNS Client service will wipe tis cache and input an updated entry from the DNS Server.

SO... even thought the local machine may get a clean IP at first (making it seem like a PC issue, because the DNS resolved the correct IP) when the DNS Client service wipes the local cache and re-enters an updated entry it could then be the rogue IP (thus becoming a DNS Server issue)...

hmmm... disabling this service, clearing the local cache should be my answer to narrowing this down.

Anybody got any ideas, I am literally running in circles thinking my brain away!

I've merged these for you so it's all in one place.

Now here's my thoughts on this.

Do you suspect your own DNS server or could it possibly from 1 or both of the external DNS's.

Try changing your external DNS resolution to something "trusted" like Google on 8.8.8.8 and 8.8.4.4 as your DNS servers.

My own setup has a number of DNS's setup.
Primary and secondary query each other along with my sister site, group office as well as a couple of external sources.
Covers all of the bases.

If I check the Server end when this issue happens it will tell me if it's a DNS Server issue or a local PC issue. I'll get back with you when it happens and I can check this out!

This whole time Ive been checking the PC end of things which can be 1 of 2 things, checking the server end will narrow this down.

Our two DNS Servers point to themselves for queries not each other. If I do a nslookup and the Rogue IP is returned then I narrowed it down to the server it qeuried too.

Server A wants to access google.com
1. Checks local cache for IP
2. Checks its DNS Service for IP
3. Checks Server B for IP
4. Checks forwarder A then B.

If I get the rogue IP when I nslookup google.com and it's resolved by itself then Server A is the issue. If it's resolved by Server B, Server B is the issue.

Beings Server A will query it's own service it will most likely resolve the domain-IP. I believe it is a Server A issue (which just so happens to be our primary DNS, thus every PC points to it first [after local cache of course] for resolution)

Ill let you know!

You can always disable one of your DNS servers to confirm your thoughts.
In fact that might be the best idea until you can either permanently resolve this or nuke it.

I figured it out. If you go into your DNS service and right click on your DNS' and go to properties, there is an option in there to disable/enable cache poisoning. (Microsoft calls it something else, but I've been drinking and I can't remember what they call it)

Either way this was disabled but by default it should be enabled. Either it was changed at some point or something changed it but we enabled it and all is good as of now.

Ill let you know if anything changes!

Interesting stuffs.
What was the option called?
Why would you want to allow this, hang on I can see a use for this now, as a kind of net filter redirecting workers to the corp site rather than Facebook for example.

This is where you go and enable/disable this option. This option is enabled by default but something must have changed it.

1. Open the DNS Management Console by clicking Start, Programs, Adminstrative Tools, and then clicking DNS.

2. Right click on the server name in the left window panel.

3. Choose Properties.

4. Choose the Advanced tab.

5. Confirm that the "Secure cache against pollution" check box is selected.

Rick,

Now that you've found the problem you might want to change who can reach your Caching DNS servers. This may not be an option but if it is then you should consider it. If you place your caching DNS servers inside your firewall (since no one on the outside really needs to reach them anyway) then you can mitigate cache poisoning along with other DNS exploits/attacks.

Enabling "Secure cache against pollution" is good as long as your server is not heavily loaded. You should keep an eye on the server load after you turn this feature on since every single DNS request (from your clients) will need to be verified (by your DNS server) against an authoritative DNS server. If your load becomes too excessive then either get a faster server or move your caching servers inside your stateful firewall. The firewall will in turn indirectly ensure that the query is coming from the source that you asked and not an attacker (most firewalls can do this).

Caching DNS servers will accept an answer from anyone provided the timing is right. All an attacker has to do to poison your DNS is hammer away at your caching DNS server with responses for queries that they hope you'll be making and banking on getting their response in before the real server responds. There was a good use for this at one point with ISPs and hosting centers so that you are directed towards the closest server.

Something else you can do is also check who is hammering at your server with bogus responses to queries you're not making. You'll need to sniff the line, install Wireshark, or setup some kind of a packet capture so that you can see see the attacks. Once you find out where it is coming from you can block that IP which should give you some temporary protection.

Casey

"then you can mitigate cache poisoning along with other DNS exploits/attacks."
Our servers are inside the firewall, with this said, putting servers inside firewalls don't completely protect them.

"or move your caching servers inside your stateful firewall"
They are!

"Something else you can do is also check who is hammering at your server with bogus responses to queries you're not making. You'll need to sniff the line, install Wireshark, or setup some kind of a packet capture so that you can see see the attacks. Once you find out where it is coming from you can block that IP which should give you some temporary protection."
Won't monitoring the firewall show you traffic hitting the DNS Servers? I'm pretty sure it can!


Either way, thanks for the response and it's something I'll take a more in-depth look at tomorrow.

Rick,

You're right, simply putting the DNS servers inside the firewall isn't enough especially if your rules aren't crafted appropriately. I should have been clearer. You need to block inbound requests to port 53 (udp and tcp) to the caching servers while allowing only outbound requests from the caching server. This can be tricky depending on how complex the firewall's capabilities are. A statefull firewall should be able to do the above. Some firewalls allow you to define a particular IP/Server as a DNS server and it can further enhance what is allowed in/out by acting as a DNS application gateway.

The other option which is simpler and won't require a complex firewall capability is to re-IP your caching DNS servers into unregistered space and place behind a generic NAT address (many to one). What this will do is allow outbound access from your caching DNS but block all inbound unsolicited requests from reaching your caching DNS servers. When your caching DNS makes an Internet bound request the firewall will setup a connection state for the NAT table that has the following:

src IP (Your unregistered caching DNS IP)
src Port (Random number chosen by the Caching DNS server)
dst IP (some authoritative DNS server)
dst Port (53)
protocol (udp)

Example:

Src IP: 10.0.0.10
Src Port: 44067*
Dst IP: 216.239.32.10 (one of google's authoritative DNS servers)
Dst Port: 53
Protocol: UDP
NAT: 20.20.20.20 (let's say that's your external NAT)

The NAT translation then happens such that when a DNS response reaches the firewall in the form of:

Src IP: 216.239.32.10
Src Port: 53
Dst IP: 20.20.20.20
Dst Port: 44067*
Protocol: UDP

The firewall will see that connection in its NAT Table and allow the response to reach your DNS server (after it does the reverse NAT translation).

Now let's suppose an attacker is trying to send you a spoofed poison response and the firewall sees this:

Src IP: 200.100.20.10
Src Port: 53
Dst IP: 20.20.20.20
Dst Port: 44067*
Protocol: UDP

The firewall will see the above and since it doesn't have a NAT table entry for that it will drop it. If the DNS server is exposed either directly or UDP traffic is allowed to it through the firewall then the above spoofed connection is passed on to the caching DNS server. At that point you must have the server configured to check for the source to ensure that it is coming from 216.239.232.10 and not 200.100.20.10.

In terms of monitoring traffic, yes you should be able to monitor your firewall. You might have to do some filtering though to narrow down the requests (i.e. look for traffic sourced from your DNS and for traffic destined to your DNS from port UDP/53). If you install wireshark on the caching DNS server then all you need to look for is port UDP/53 in any direction. This is purely a personal choice.

I typically use tcpdump since all of my DNS servers are Linux. Windows doesn't offer native packet capture (hence wireshark).

I starred the port 44067 because that's a random port that your DNS server will use to initiate outbound requests on. All an attacker has to do is figure out that port and then send "replies" to that port. If your DNS server is reachable directly via the outside or a firewall then it will accept the request. If it is behind NAT then the firewall will check for the NAT connection before it will pass it back to the server.

I hope that helps.

Casey

Very good information here. I have forwarded this to the Systems Admin as I started working here long after this was setup and am unsure of the exact setup.

He may or maynot implement this strategy already. Thanks again for the information!