Dec 202014
 

We have received a report theNational Finance Center site www.nfc.usda.gov is currently returning a 500: Server Error (thanks Melissa) and the U.S. Department of Agriculture www.usda.gov is returning an IBM HTTP WebSphere software page. We are currently investigating to get additional information.

Update 1: www.usda.gov is now back up at 02:30 GMT

-----------

Guy Bruneau IPSS Inc. gbruneau at isc dot sans dot edu

(c) SANS Internet Storm Center. https://isc.sans.edu Creative Commons Attribution-Noncommercial 3.0 United States License.
Dec 202014
 

(Also see our earlier diary about thisvulnerability)

While people generally know where their real NTP servers are, all to often they dont know that theyve got a raft of accidental NTP servers - boxes that have NTP enabled without the system maintainers knowing about it. Common servers on the network like routers or switches (often when these are NTP clients, they are also NTP servers), PBXs and VOIP gateways, mail servers, certificate authorities and so on.

In these days of auto-updates, you would think that most NTP servers would be patched against the vulnerabilities found by the Google team and described in story written up by Johannes earlier this evening.

However, it only took until the second host checked to find a very out of date server. Unfortunately, its the main NTP server of a large Canadian ISP (Oops). What I also found along the way was that many servers only report 4 as a version, and that from the -sV switch, not from ntp-info. So depending on your internal servers and how they are configured, it may be time for us to start using authenticated scans using tools like Nessus to get service versions for our NTP servers. Hopefully that">C:\">Nmap scan report for ntp.someisp.ca (x.x.x.x)
Host is up (0.0045s latency).
rDNS record for x.x.x.x: khronos.tor.someisp.ca
PORT STATE SERVICE VERSION
123/udp open ntp NTP v4
| ntp-info:
| receive time stamp: 2014-12-20T02:47:52
|">version: ntpd [email protected] Thu Feb 13 12:17:19 EST 2003 (1)
| processor: i686
| system: Linux2.4.20-8smp
| leap: 0
| stratum: 3
| precision: -17
| rootdelay: 11.079
| rootdispersion: 33.570
| peer: 32471
| refid: x.x.x.x
| reftime: 0xd83f5fad.b46b9c30
| poll: 10
| clock: 0xd83f61d5.3a71ef30
| state: 4
| offset: -0.329
| frequency: 46.365
| jitter: 3.468
|_">Service detection performed. Please report any incorrect results at http://nmap.
org/submit/ .
Nmap done: 1 IP address (1 host up) scanned in 180.08 seconds

This server on the other hand, doesnt report the version in the ntp-info output. -sV reports version 4, but that">C:\ ">Nmap scan report for time.someotherserver.com (y.y.y.y)
Host is up (0.010s latency).
PORT STATE SERVICE VERSION
123/udp open ntp NTP v4
| ntp-info:
|_">Service detection performed. Please report any incorrect results at http://nmap.
org/submit/ .
Nmap done: 1 IP address (1 host up) scanned in 143.24 seconds

But really, after this year of vulnerabilties that weve seen in basic system services, its about time that folks took the SANS Top 20 to heart - the SANS Critical Controls that you really should be looking at if its your goal to secure your network - https://www.sans.org/critical-security-controls . The top 5 in the list sum up your first line of defense against stuff like this. Know whats on your network, know whats running on that, have a formal program of patches and updates, and scan regularly for new hosts, new services and new vulnerabilities. If its your thought that a single scan for this one vulnerability is the most important thing on your plate (or scanning for heartbleed or shellshock was earlier this year), then you have already lost - it">Quick Addendum/Update (Johannes):

CentOS and other Linux distros did release updates. However, the version string may not change. Check the Build Date. For example, on CentOS6:
Before patch:ntpd [email protected] Sat Nov 23 18:21:48 UTC 2013 (1)
After patch:">">" type="cosymantecnisbfw">

(c) SANS Internet Storm Center. https://isc.sans.edu Creative Commons Attribution-Noncommercial 3.0 United States License.
Dec 192014
 

The Google security team discovered several vulnerabilities in current NTP implementations, one of whichcan lead to arbitrary code execution [1][2]. NTP servers prior to version 4.2.8 are affected.

There are some rumors about active exploitation of at least some of the vulnerabilities Google discovered.

Make sure to patch all publicly reachable NTP implementations as fast as possible.

Mitigating Circumstances:

Try to block inbound connections to ntp servers who do not have to be publicly reachable. However, be aware that simple statefull firewalls may not track UDP connections correctly and will allow access to internal NTP servers from any external IP if the NTP server recently established an outbound connection.

ntpd typically does not have to run as root. Most Unix/Linux versions will configure NTP using a lower privileged users.

According to the advisory at ntp.org, you can also:

Disable Autokey Authentication by removing, or commenting out, all configuration directives beginning with thecryptokeyword in yourntp.conf">A few Ubuntu and CentOS systems I tested, as well as OS X systems, do not seem to use autokey.

[1]http://www.kb.cert.org/vuls/id/852879
[2]">In the NTP code, a section of code is missing a return, and the resulting error indicates processing did not stop.

" type="cosymantecnisbfw">

(c) SANS Internet Storm Center. https://isc.sans.edu Creative Commons Attribution-Noncommercial 3.0 United States License.
Dec 192014
 

With two stories on the topic of bridging datacenters, youd think I was a real believer. And, yes, I guess I am, with a couple of important caveats.

The first is encapsulation overhead. As soon as you bridge using encapsulation, the maximum allowed transported packet size will shrink, then shrink again when you encrypt. If your Server OSs arent smart about this, theyll assume that since its all in the same broadcast domain, a full packet is of course OK (1500 bytes in most cases, or up to 9K if you have jumbo frames enabled). Youll need to test for this - both for replication and the failed-over configuration - as part of your design and test phase.

The second issue si that if you bridge datacenters to a DR or second (active) datacenter site, you are well positioned to fail over the entire server farm, as long as you can fail over your WAN connection and Internet uplink with them. If you dont, you end up with what Greg Ferro calls a network traffic trombone. (http://etherealmind.com/vmware-vfabric-data-centre-network-design/)

If you fail one server over, or if you fail over the farm and leave the WAN links behind, you find that the data to and from the server will traverse that inter-site link multiple times for any one customer transaction.

For instance, lets say that youve moved the active instance of your mail server to the DR Site. To check an email, a packet will arrive at the primary site, traverse to the mail server at site B, then go back to site A to find the WAN link to return to the client. Similarly, inbound email will come in on the internet link, but then have to traverse that inter-site link to find the active mail server.

Multiply that by the typical email volume in a mid-sized company, and you can see why this trombone issue can add up quickly. Even with a 100mb link, folks that were used to GB performance will now see their bandwidth cut to 50mb or likely less than that, with a comensurate impact on response times. If you draw this out, you do get a nice representation of a trombone - hence the name.

What this means is that you cant design your DR site for replication and stop there. You really need to design it for use during the emergency cases you are planning for. Consider the bandwidth impacts when you fail over a small portion of your server farm, and also what happens when your main site has been taken out (short or longer term) by a fire or electrical event - will your user community be happy with the results?

Let us know in our comment section how you have designed around this trombone issue, or if (as Ive seen at some sites), management has decided to NOT spend the money to account for this.

===============
Rob VandenBrink
Metafore

(c) SANS Internet Storm Center. https://isc.sans.edu Creative Commons Attribution-Noncommercial 3.0 United States License.
%d bloggers like this: