Packet loss to UCSE2/0 from external sources

Version 1
    This document was generated from CDN thread

    Created by: Brian Beebe on 24-01-2013 12:21:40 PM
    We currently have UCSE running and is mostly working fine except that every once in while we will get packet loss when hitting the IP address of the CIMC or ESXi. Pinging from the router itself is fine.  Pinging from gig 0/0.10 is fine.  Pinging across the VPN to the gig 0/0.10 is fine.  But pinging across the VPN to the CIMC will sometimes get packet loss.  I can recreate the loss by pinging from a switch connected to gig 0/0.10 to the CIMC which shows no packet loss but causes a ping from across the VPN to start randomly dropping packets.
    Has anyone seen this behavior?  
    We have the imc configures as such:
    interface GigabitEthernet0/1
     ip vrf forwarding INET-1
     ip address dhcp
     ip nat enable
     duplex auto
     speed auto
     media-type rj45
     ip rsvp bandwidth
    interface Tunnel100
     bandwidth 2048
     ip address 172.20.64.2 255.255.255.0
     no ip redirects
     ip mtu 1400
     ip wccp 60 redirect in
     ip wccp 161 redirect in
     ip nhrp authentication mediator
     ip nhrp group 512K
     ip nhrp map multicast dynamic
     ip nhrp map 172.20.64.1 216.49.183.18
     ip nhrp map multicast 216.49.183.18
     ip nhrp network-id 187
     ip nhrp holdtime 360
     ip nhrp nhs 172.20.64.1
     ip nhrp registration timeout 120
     ip nhrp shortcut
     ip nhrp redirect
     ip summary-address eigrp 211 10.224.128.0 255.255.255.0
     ip tcp adjust-mss 1300
     load-interval 30
     delay 1000
     qos pre-classify
     keepalive 10 3
     tunnel source GigabitEthernet0/1
     tunnel mode gre multipoint
     tunnel key 187
     tunnel vrf INET-1
     tunnel protection ipsec profile DMVPN_PRIMARY
     ip rsvp bandwidth
     ip rsvp tunnel overhead-percent 4
     
    interface GigabitEthernet0/0.10
     description BACKBONE
     encapsulation dot1Q 10
     ip address 10.224.128.1 255.255.255.128
     ip wccp 61 redirect in
     ip wccp 162 redirect in
     ip flow ingress
     ip flow egress
     ip nat enable
     ip rsvp bandwidth
    interface ucse2/0
     ip unnumbered GigabitEthernet0/0.10
     ip nat enable
     imc ip address 10.224.128.2 255.255.255.0 default-gateway 10.224.128.1 
     imc access-port shared-lom console
    ip route 10.224.128.2 255.255.255.255 ucse2/0
     

    Subject: RE: Packet loss to UCSE2/0 from external sources
    Replied by: Brett Tiller on 24-01-2013 07:18:18 PM
    Hi Brian,

    We are not aware of this issue on the UCS E-Server.  What IOS release are you using, and what is your router platform?  Apparently this issue only affects VPN packets? In other words if you were not using VPN, this issue does not occur?
     
    Thanks,
    Brett

    Subject: RE: Packet loss to UCSE2/0 from external sources
    Replied by: Brian Beebe on 25-01-2013 08:50:21 AM
    Yes, I have just confirmed that only packers traversing the VPN are affected.  Pings going across that same VPN to the gig 0/0.10 inteface or to anything connected to that VLAN are fine.  Only pings to any IP connected to the UCSE2/0 interface.
    Brian

    Subject: RE: Packet loss to UCSE2/0 from external sources
    Replied by: George Bekmezian on 25-01-2013 10:28:02 AM
    Brett Tiller:
    What IOS release are you using, and what is your router platform?

    15.3(1)T and 3925.
    Brett Tiller:
      Apparently this issue only affects VPN packets? In other words if you were not using VPN, this issue does not occur?

    Correct, we just verified this by removing tunnel protection from the tunnel interfaces of the affected router and the head end router and the packet loss went away.  Nothing else was changed anywhere in the network.  Maybe a very odd interaction between crypto and contention to the backplane?
     

    Subject: RE: Packet loss to UCSE2/0 from external sources
    Replied by: Brett Tiller on 25-01-2013 01:34:45 PM
    Hi George and Brian,
    Thanks for the information.  Assuming that you are experiencing this issue on previous releases of IOS, and given the complexity of this issue, I recommend that you open a case with our TAC team.  Here's the link  http://tools.cisco.com/ServiceRequestTool/create/launch.do .   Our TAC team will investigate the router, IOS and VPN to determine if the problem resides in those areas.  If not, they will escalate it to our team.
    Thanks,
    Brett