Mitigating the BlackNurse exploit on Cisco and Fortigate equipment

 

"Blacknurse is a low bandwidth ICMP attack that is capable of doing denial of service to well known firewalls.  Most ICMP attacks that we see are based on ICMP Type 8 Code 0 also called a ping flood attack. 

BlackNurse is based on ICMP with Type 3 Code 3 packets. We know that when a user has allowed ICMP Type 3 Code 3 to outside interfaces, the BlackNurse attack becomes highly effective even at low bandwidth. 

Low bandwidth is in this case around 15-18 Mbit/s. This is to achieve the volume of packets needed which is around 40 to 50K packets per second. It does not matter if you have a 1 Gbit/s Internet connection." – as per http://blacknurse.dk/ 

Securing Fortigate

Andras the Techie created an Fortigate IPS signature to detect and drop this traffic:

config ips custom
    edit ICMP.Blacknurse
      set signature "F-SBID( --name \"ICMP.Blacknurse\"; --protocol icmp; --icmp_type 3; --icmp_code 3; --rate 250,1;)"
        set severity medium
        set location server
        set application Other
        set action block
        set status enable
end

His blog post also contains instructions on how to apply it to you policies.

Securing Cisco

Best is of course to filter this traffic upstream, before it hits your firewall.  This is what you’ll need to do on an Cisco IOS router:

config t
ip icmp rate-limit unreachable 100 1000
do copy run start
end

For Cisco IOS-XR you will need to do this:

icmp ipv4 rate-limit unreachable 2000
commit

Cisco traceroute status codes

Just a quickie.  Cisco routers can give seemingly weird (especially if you’re coming from the Windows world) traceroute outputs.  This is something that I did not see documented in either the CCNA or CCNP training guides.

In Cisco routers, the codes for a traceroute command reply are:
! -- success
* -- time out
N -- network unreachable
H -- host unreachable
P -- protocol unreachable
A -- admin denied
Q -- source quench received (congestion)
? -- unknown (any other ICMP message)

For further reading refer to the IBM AIX documentation – as far as I can make out the AIX codes applies to Linux as well.

Setting up a Cisco port-channel

Setting up port-channels on a Cisco switch can sometimes be a real PITA due to your interface settings having to match the port-channel settings.  I’ve come up with a way to eliminate the config mismatches, which I’ll describe below.

First we clear the configuration on the interfaces that we’re going to add to our port-channel by doing a “default int gi1/1/1”

Then we add our interfaces to the port-channel and only then do we configure our port-channel, like so:

int gi1/1/1
channel-group <x> mode active
!
int gi2/1/1
channel-group <x> mode active
!
int po<x>
switchport mode trunk
switchport trunk allowed vlan 10,50,...
switchport trunk native vlan 999

This forces your interfaces to join the port-channel before any configuration is added to them. It also reduces the mistake of having mismatched configuration on both interfaces, which could cause independent members in your port-channel.

VLAN Up, Line Protocol Down

Creating a VLAN and a SVI

Today, not for the first time, I was asked to troubleshoot a issue with an SVI (Switch Virtual Interface) not coming up.  The customer had created a SVI like so:

switch# conf t
switch (config)# int vlan 10
switch (config-if)# ip address 192.168.10.1
switch (config-if)# no shut

The problem was that the SVI stayed down, and the VLAN did not even show up when you do a “sh vlan”.  Why did this happen?  Because the customer created the SVI without first creating the Layer 2 VLAN for it to live in.  Here is what the customer should have done:

switch# conf t
switch (config)# vlan 10 ***this creates the layer 2 VLAN***
switch (config-vlan)# name VLAN_NAME
switch (config-vlan)# exit
switch (config)# int vlan 10 ***this creates the layer 3 SVI for the VLAN***
switch (config-if)# ip address 192.168.10.1 ***assigns the IP and netmask to the SVI***
switch (config-if)# no shut

Now if we do a “show vlan” it should show as active.  Whilst we’re on this topic, let’s have a look at what exactly is required for a SVI to come up and be ping-able on your network.

Requirements to get a SVI to the Up/Up state

An SVI must meet all of the below requirements to transition to the “Up/Up” state

• Its VLAN must exist and show as active in the VLAN database “sh vlan”
• One switched port in the VLAN (can be access or trunk) must be up
• The port must be in a STP forwarding state

Schedule a Reload on a Cisco Router

Who hasn’t been in the situation where you have to make changes to a Cisco devices that you do not currently have physical access to. Fun and games until you fat finger something and lose access to the device. 

An easy way around this is to schedule a reload of the device before you start making changes.  That way if something does go wrong the device will reload and come up as before.  Of course if the reload is not required you can cancel it.  Below is the steps needed to achieve all that.

Cisco_Switch#copy run start
Destination filename [startup-config]?
Building configuration...
[OK]
Cisco_Switch#reload in 015
Reload scheduled in 15 minutes
Proceed with reload? [confirm]
Cisco_Switch#reload cancel
Cisco_Switch#

***
*** --- SHUTDOWN ABORTED ---
***

Cisco_Switch#

Microsoft Network Load Balancing, and how to make it work with a Cisco Nexus

Microsoft Network Load Balancing has one or two “interesting” issues due to the way it does networking.  More specifically, it utilises a Multicast MAC address with a Unicast IP address.  Someone much smarter than me explains the ins and outs of the issue here. 

This issue manifests itself in that clients outside of the cluster’s LAN cannot connect to the IP.

Now that we understand the issue,  how do we resolve it?  The fixes listed in the article are all valid, however we can get this working by tweaking the config on our Nexus.

interface Vlan10
  no ip redirects
  ip address 192.168.10.1/24
  ip arp 192.168.10.100 03BF.0A03.00CF
  description Server Farm
  no shutdown

Your clients should now be able connect to the cluster IP address without issue.

Cisco Port Configuration Best Practices

I run into inconsistent network configurations wherever I go, from customers who just lets everything live in the native VLAN to ones who horrendously over-complicate things.  For the majority of Cisco deployments I have a simple set of configuration standards I adhere to.  You will always get corner-case requiring special configurations, but I find that the below works very well for most use cases.

Standard Access Port Configuration

This configuration is applied to ports connecting to standard end-user equipment, like PC’s, printers etc.
switchport mode access
switchport access vlan 5
spanning-tree portfast
spanning-tree bpduguard enable

The first two lines are self-explanatory, it’s an access port living in vlan 5.  Portfast is something we configure on an access port, which tells STP not to bother and just put the port in a forwarding state, as opposed to taking time to go through the listening and learning states. 
BPDU’s are basically STP messages exchanged between switches, therefore a BPDU is not something we expect to see on a normal access-port.  BPDUguard tells the switch that if it receives a BPDU (for example because someone connected an unauthorized switch), it should shut the port down.  Finally BPDUFilter tells the switch to not send or receive BPDU’s on ports configured as portfast.

ESXi Host Port Configuration

ESXi uses internal vSwitches and PortGroups which allows for VM’s running on one host to reside in different VLANs
switchport mode trunk
switchport trunk allowed vlan 5-10,200
spanning-tree port type edge trunk
spanning-tree bpduguard enable
Fairly straightforward, we configure our link as a trunk which carries vlan 5 to 10 and 200.  We then tell the switch that even though it’s a trunk we’re no connecting to another switch so no need to worry about STP on the port.  Lastly bpduguard protects us against incorrect cabling – if the port receives a BPDU (which will happen if you inadvertently hook it up to another switch) it will shut down.
 

Standard Trunk Port Configuration

In the Cisco world a trunked link is a link that carries multiple VLANs.  Not to be confused with link aggregation, which in Cisco parlance is called an Port-Channel.
switchport mode trunk
switchport trunk encapsulation dot1q
switchport trunk native vlan 10
switchport trunk allowed vlan 10-15
Both sides of the trunk needs to have the same default VLAN, by default the native VLAN is VLAN 1, but in all but the simplest deployments you will have to change this.  We can also do VLAN pruning on our trunks, that is only carry certain VLANs accross our trunk.

Port-Channel Configuration

It is possible to aggregate multiple links between two switches and treat them as a single link.  This gives us link redundancy and bandwidth increases.  As a rule we do not use LACP to ensure compatibility with, for example, vSphere vSwitches.
interface Port-channel10
switchport trunk encapsulation dot1q
switchport mode trunk
Once again simplicity is the name of the game.  We create the interface, set the encapsulation (not necessary, strictly speaking since Cisco defaults to dot1q) and set it as a trunk.  Of course it does not need to be a trunk link, this is optional.

Setting up vSphere Active / Active iSCSI connections to a NetApp FAS2040

I recently had the opportunity to architect a solution consisting of 3 vSphere 5 boxes connecting to a NetApp FAS2040.  Storage connectivity would be via iSCSI.  The storage network would be running off of 2 Cisco 2960G switches, soon to be replaced by stacked Cisco 3750’s. 

The requirements were stock standard, as high a throughput as possible, with as much redundancy as possible.  This meant going active active on the iSCSI links.  Here is how I did it.

NetApp FAS2040 Configuration

This little SAN has 8 1GB Ethernet ports.  Due to the fact that the Cisco 2960G switches does not support multi-link switch aggregation (this is where the 3750’s will come in) I had to come up with a simpler design – what NetApp terms a Single-Mode design.  My design allows for:

• Two active connections to each controller, thus a total of four active sessions
• Storage path HA
• Load balancing across links
• Uses vSphere storage MPIO as opposed to switch-side configuration

Virtual Interface (VIF) Configuration:

All Vif's are single-mode / active passive
Cont1_Vif01 - e0a/e0b (e0a will be active, connected to switch 1 / e0b passive connected to switch 2) IP – 192.168.1.1
Cont1_Vif02 - e0c/e0d (e0c will be active, connected to switch 2 / e0d passive connected to switch 1) IP – 192.168.2.1
Cont2_Vif01 - e0a/e0b (e0a will be passive, connected to switch 1 / e0b active connected to switch 2) IP – 192.168.1.2
Cont2_Vif02 - e0c/e0d (e0c will be passive, connected to switch 2 / e0d active connected to switch 1) IP – 192.168.2.2

This image, courtesy of NetApp, explains it infinitely better than my wall of text:-)

I also configured partner takeover for all VIF.  In case of controller failure it allows the remaining controller to take over the VIFs.

Ethernet Storage Network Configuration

On the storage network I had to configure 2 critical settings:

• Spanning Tree Portfast
• Jumbo Frames

When connecting ESX and NetApp storage arrays to Ethernet storage networks, NetApp highly recommends configuring the Ethernet ports to which these systems connect as RSTP edge ports.  This is done like so:

Switch2960(config)# interface gigabitethernet2/0/2
Switch2960(config-if)# spanning-tree portfast

Next up, Jumbo Frames:

Switch2960(config)# system mtu jumbo 9000
Switch2960(config)# exit
Switch2960# reload

vSphere Configuration

I am in love with vSphere 5, and one of the biggest reasons for that is the fact that a lot of the configuration parameters that used to be command-line only has been moved into the GUI.  Another reason is Multiple TCP Session Support for iSCSI.  This feature enables round robin load balancing using VMware native multipathing and requires a VMkernel port to be
defined for each physical adapter port assigned to iSCSI traffic.  That said, let’s get configuring:

1. Open your vCenter Serve
2. Select an ESXi host
3. In the right pane, click the Configuration tab
4. In the Hardware box, select Networking
5. In the upper-right corner, click Add Networking to open the Add Network wizard
6. Select the VMkernel radio button and click Next
7. Configure the VMkernel by providing the required network information.  NetApp requires separate subnets for active/active iSCSI connections, therefore we will create two VMkernels, on the 192.168.1.x and 192.168.2.x subnets respectively.
8. Configure each VMkernel to use a single active adapter that is not used by any other iSCSI VMkernel. Also, each VMkernel must not have any standby adapters. If using a single vSwitch, it is necessary to override the switch failover order for each VMkernel port used for iSCSI. There must be only one active vmnic, and all others should be assigned to unused
9. The VMkernels created in the previous steps must be bound to the software iSCSI storage adapter. In the Hardware box for the selected ESXi server, select Storage Adapters.
10. Right-click the iSCSI Software Adapter and select properties. The iSCSI Initiator Properties dialog box appears
11. Click the Network Configuration tab
12. In the top window, the VMkernel ports that are currently bound to the iSCSI software interface are listed
13. To bind a new VMkernel port, click the Add button. A list of eligible VMkernel ports is displayed. If no eligible ports are displayed, make sure that the VMkernel ports have a 1:1 mapping to active vmnics as described earlier
14. Select the desired VMkernel port and click OK.
15. Click Close to close the dialog box
16. At this point, the vSphere Client will recommend rescanning the iSCSI adapters. After doing this, go back into the Network Configuration tab to verify that the new VMkernel ports are shown as active, as per the image below.

Congratulations, you now have active / active, redundant iSCSI sessions into your NetApp SAN!

Repurposing an old Cisco PIX to provide secure public WiFi on a corporate LAN

I am in the planning stages of a fun little project, whereby the clients goal is to provide secure wireless access to guests over a ADSL link dedicated to this purpose.  Simple enough, but this traffic will travel over the same edge to core switches that carry business traffic, so we will have to set up some VLAN’s.

This client recently retired their ageing PIX firewall and replaced it with new Cisco ASA’s.  So instead of chucking the PIX we will press it into service as the secure gateway / firewall for the public ADSL Internet breakout.

The PIX in this case has two physical interfaces named ethernet0 and ethernet1.  ethernet0 will be connected to the ADSL and ethernet1 to the LAN.  ethernet1 will be configured in a VLAN, and the switch ports to which the guest AP’s connect will be configured to do the appropriate VLAN tagging.

Here is how to configure the PIX

1. interface ethernet0 auto
2. interface ethernet1 auto
3. interface ethernet1 vlan1 physical
4. interface ethernet1 vlan10 logical
5. nameif ethernet0 outside security0
6. nameif ethernet1 inside security100
7. nameif vlan10 guest_wifi security10

Most client devices nowadays expect DHCP and since they don’t logically touch the corporate network we’ll have to make do with running DHCP off the PIX:

1. dhcpd address 192.168.202.100-192.168.202.200 guest_wifi
2. dhcpd dns 192.168.0.1 (substitute this with your ISP’s DNS Server)
3. dhcpd lease 3600
4. dhcpd ping_timeout 50
5. dhcpd enable guest_wifi
6. ip address wifi 192.168.202.1 255.255.255.0

And that should work rather brilliantly – I will know for sure in about a weeks time when I implement.

Enabling Jumbo Frames in vSphere

Hi Kids, todays post is brought to you by the letter J. I previously mentioned a vSphere deployment I had to do, connecting to a EMC AX4-5i iSCSI SAN. Once I got the storage hooked up to my ESX hosts I of course wanted to enable jumbo frames.

Jumbo Frames, for the uninitiated heathens out there, is basically a Ethernet frame with a payload of more than 1500 bytes, up to a typical maximum of 9000 bytes per frame. Why would I want to do this? Performance, in a nutshell. The only requirement would be that your switch supports Jumbo Frames, which I believe most, if not all, mid to high-end kit does. So let's get down to business, shall we?

Switch Configuration

I'm fairly proficient with Cisco and HP networking gear, so I'll give a quick rundown of the commands needed to enable jumbo frames on their kit:

• Cisco: Go into conf t mode and enter the following command "system mtu jumbo 9000". Once that is done issue a "reload" command to reboot your switch
• HP Procurve: On these babies Jumbo frames need to be enabled per VLAN. Execute the following command: "vlan # jumbo"

That takes care of the networking side of things, let's move on to the ESX...

Configure Jumbo frames in vSphere

1. Log on to your ESX host using your favorite SSH client
2. Change your chosen vSwitch MTU with the following command "esxcfg-vswitch -m 9000 vSwitch#" Replace vSwitch# with the name of the vSwitch you want to modify
3. Seeing as one cannot change an existing VMKernel port MTU, you will either have to remove and recreate your existing VMKernel port, or create a new one. To delete an existing VMKernel port, enter the following command: "esxcfg-vmknic -d -p VMKernelport"
4. Now let's add a VMKernel port called "iSCSI01" to our vSwitch, like so: "esxcfg-vswitch –A iSCSI01 vSwitch#"
5. Now we enable jumbo frames on our VMKernel port: "esxcfg-vmknic –a –i 192.168.x.x –n 255.255.255.0 –m 9000 iSCSI01". You do know you need to change the IP address and mask to reflect your environment, right?
   
6. Let's make sure we didn't screw up somewhere by running "esxcfg-vmknic –l". Verify that the MTU is set to 9000
7. All done! You can refer to my previous iSCSI Multipathing post to ensure Jumbo frame enabled iSCSI Multipathing goodness!

I have thought of merging this post with the iSCSI Multipathing post, but seeing as not all environments can/will support end to end Jumbo frames, I have decided to keep them seperate for the time being.