lost of access to hypervisor from remote networks

Version 1
    This document was generated from CDN thread

    Created by: ISMAIL PATEL on 20-06-2011 05:50:40 PM
    Following the upgrade to version 2, we have noticed that we lose access to the hypervisor from remote clients.
     
    From our client machine ( 172.18.2.43 ) we are unable to access http://10.24.7.18/ or use the vmware infrastructure client to connect to the server.
     
    The hypervisor is accessible from the local subnet.
     
    virtual machines running on the hypervisor are not affected
     
    A reload of the hypervisor resolves the problem for a short time but is service affecting.
     
    configuration as below:
     
    interface SM1/0
     ip vrf forwarding DISTSCHOOLS
     ip address 10.24.7.17 255.255.255.240
     service-module ip address 10.24.7.18 255.255.255.240
     !Application: VMware ESXi 4.1.0 build-348481 running on SRE-V
     service-module ip default-gateway 10.24.7.17
     hold-queue 60 out
    interface SM1/1
     no ip address
    end
     
    no real timeline when the problem manifests
     
    I have added the module to our NMS system therefore should have statistics going forward  on when the issue happens

    Subject: RE: lost of access to hypervisor from remote networks
    Replied by: ISMAIL PATEL on 21-06-2011 04:28:04 PM
    Next option would be to upgrade to 151-4M Please advise.

    Thanks

    Ismail

    Subject: RE: New Message from Radhika Miriyala in Service Ready Engine Virtualizatio
    Replied by: ISMAIL PATEL on 21-06-2011 05:14:37 PM
    Hi Radhika



    Good to hear from you.



    Configuration is as below: ( Full config is in the post relating to vrf )



    interface SM1/0

    ip vrf forwarding DISTSCHOOLS

    ip address 10.24.7.17 255.255.255.240

    service-module ip address 10.24.7.18 255.255.255.240

    !Application: VMware ESXi 4.1.0 build-348481 running on SRE-V

    service-module ip default-gateway 10.24.7.17

    hold-queue 60 out

    !

    interface SM1/1

    no ip address



    In terms of the questions:



    1)      I create a virtual switch port group under vswitch0 which I moved my virtual machines to

    2)      Yes ¿ output below



    TESTSCHRTR101742#ping vrf DISTSCHOOLS 10.24.7.17



    Type escape sequence to abort.

    Sending 5, 100-byte ICMP Echos to 10.24.7.17, timeout is 2 seconds:

    !!!!!

    Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms

    TESTSCHRTR101742#ping vrf DISTSCHOOLS 10.24.7.18



    Type escape sequence to abort.

    Sending 5, 100-byte ICMP Echos to 10.24.7.18, timeout is 2 seconds:

    !!!!!

    Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/4 ms



    3)      No teaming enabled on nic1 ¿ it is unused



    As per Brett¿s suggestion ¿ I will upgrade the IOS to the recommended version and see if this resolves the issue.



    Are there any commands you want me to run before I shut the module down?



    Kind regards



    Ismail







    From: Cisco Developer Community Forums [mailto:cdicuser@developer.cisco.com]
    Sent: Tuesday, June 21, 2011 9:46 PM
    To: cdicuser@developer.cisco.com
    Subject: New Message from Radhika Miriyala in Service Ready Engine Virtualization - SRE-V Beta Tester Questions: RE: New Message from ISMAIL PATEL in Service Ready Engine Virtualization -



    Radhika Miriyala has created a new message in the forum "SRE-V Beta Tester Questions":
    --------------------------------------------------------------
    Hi ISMAIL,



    Just want to make sure I understand the issue correctly:



    1) It looks like Your Virtual servers are in the same subnet as that of the hypervisor (ESXi) so did you create a new VM network port group under vswitch1?

    2) When you see the issue of ping have you been able to ping the hypervisor ip address from the router?

    3) Do you have any nic teaming enabled on vsiwtch1?





    Thanks,

    Radhika



    From: Cisco Developer Community Forums [mailto:cdicuser@developer.cisco.com]
    Sent: Tuesday, June 21, 2011 1:26 PM
    To: cdicuser@developer.cisco.com
    Subject: New Message from ISMAIL PATEL in Service Ready Engine Virtualization - SRE-V Beta Tester Questions: RE: New Message from ISMAIL PATEL in Service Ready Engine Virtualization -



    ISMAIL PATEL has created a new message in the forum "SRE-V Beta Tester Questions":
    --------------------------------------------------------------
    Hi



    Loss access again today from remote client but virtual servers are ok.



    Client IP 172.18.2.43



    Ping 10.24.7.17 OK Router

    Ping 10.24.7.18 FAIL Hypervisor

    Ping 10.24.7.19 Virtual server running on service module



    C:\>ping 10.24.7.17



    Pinging 10.24.7.17 with 32 bytes of data:



    Reply from 10.24.7.17: bytes=32 time<1ms TTL=253

    Reply from 10.24.7.17: bytes=32 time<1ms TTL=253



    Ping statistics for 10.24.7.17:

    Packets: Sent = 2, Received = 2, Lost = 0 (0% loss),

    Approximate round trip times in milli-seconds:

    Minimum = 0ms, Maximum = 0ms, Average = 0ms

    Control-C

    ^C

    C:\>ping 10.24.7.18



    Pinging 10.24.7.18 with 32 bytes of data:



    Request timed out.



    Ping statistics for 10.24.7.18:

    Packets: Sent = 1, Received = 0, Lost = 1 (100% loss),

    Control-C

    ^C

    C:\>ping 10.24.7.19



    Pinging 10.24.7.19 with 32 bytes of data:



    Reply from 10.24.7.19: bytes=32 time<1ms TTL=125

    Reply from 10.24.7.19: bytes=32 time<1ms TTL=125



    Ping statistics for 10.24.7.19:

    Packets: Sent = 2, Received = 2, Lost = 0 (0% loss),

    Approximate round trip times in milli-seconds:

    Minimum = 0ms, Maximum = 0ms, Average = 0ms

    Control-C

    ^C





    C:\>tracert -d 10.24.7.18



    Tracing route to 10.24.7.18 over a maximum of 30 hops



    1 <1 ms 1 ms 2 ms 172.18.2.33

    2 2 ms 2 ms 3 ms 172.18.40.9

    3 <1 ms <1 ms <1 ms 172.31.14.39

    4 * * ^C



    Show Ip interface brief



    GigabitEthernet0/0.211 172.31.14.39 YES NVRAM up up





    From virtual Server 10.24.7.19



    C:\>ping 10.24.7.18



    Pinging 10.24.7.18 with 32 bytes of data:



    Reply from 10.24.7.18: bytes=32 time<1ms TTL=64

    Reply from 10.24.7.18: bytes=32 time<1ms TTL=64

    Reply from 10.24.7.18: bytes=32 time<1ms TTL=64

    Reply from 10.24.7.18: bytes=32 time<1ms TTL=64



    Ping statistics for 10.24.7.18:

    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),

    Approximate round trip times in milli-seconds:

    Minimum = 0ms, Maximum = 0ms, Average = 0ms



    From DCUI



    # ping 172.18.2.43

    PING 172.18.2.43 (172.18.2.43): 56 data bytes



    --- 172.18.2.43 ping statistics ---

    3 packets transmitted, 0 packets received, 100% packet loss

    ~ # ping 10.24.7.17

    PING 10.24.7.17 (10.24.7.17): 56 data bytes

    64 bytes from 10.24.7.17: icmp_seq=0 ttl=255 time=0.326 ms

    64 bytes from 10.24.7.17: icmp_seq=1 ttl=255 time=0.333 ms

    64 bytes from 10.24.7.17: icmp_seq=2 ttl=255 time=0.270 ms



    --- 10.24.7.17 ping statistics ---

    3 packets transmitted, 3 packets received, 0% packet loss

    round-trip min/avg/max = 0.270/0.310/0.333 ms



    /sbin # esxcfg-vmknic -l

    Interface Port Group/DVPort IP Family IP Address Netmask Broadcast MAC Address MTU TSO MSS Enabled Type

    vmk0 Management Network IPv4 10.24.7.18 255.255.255.240 10.24.7.31 00:23:eb:a1:07:91 1500 65535 true STATIC

    /sbin # esxcfg-route

    VMkernel default gateway is 10.24.7.17

    /sbin # esxcfg-nics -l

    Name PCI Driver Link Speed Duplex MAC Address MTU Description

    vmnic0 0000:01:00.00 e1000e Up 1000Mbps Full 00:23:eb:a1:07:91 1500 82574L Gigabit Network Connection

    vmnic1 0000:02:00.00 bnx2 Up 1000Mbps Full 88:43:e1:bc:51:e1 1500 Broadcom Corporation Broadcom NetXtreme II BCM5709 1000Base-SX

    vmnic2 0000:02:00.01 bnx2 Up 1000Mbps Full 88:43:e1:bc:51:e0 1500 Broadcom Corporation Broadcom NetXtreme II BCM5709 1000Base-SX





    Are there any other commands, that I can use to see what is going on?





    Thanks



    Ismail







    From: Cisco Developer Community Forums [mailto:cdicuser@developer.cisco.com]
    Sent: Monday, June 20, 2011 10:51 PM
    To: cdicuser@developer.cisco.com
    Subject: New Message from ISMAIL PATEL in Service Ready Engine Virtualization - SRE-V Beta Tester Questions: lost of access to hypervisor from remote networks



    ISMAIL PATEL has created a new message in the forum "SRE-V Beta Tester Questions":

    --------------------------------------------------------------
    Following the upgrade to version 2, we have noticed that we lose access to the hypervisor from remote clients.

    From our client machine ( 172.18.2.43 ) we are unable to access http://10.24.7.18/ or use the vmware infrastructure client to connect to the server.

    The hypervisor is accessible from the local subnet.

    virtual machines running on the hypervisor are not affected

    A reload of the hypervisor resolves the problem for a short time but is service affecting.

    configuration as below:

    interface SM1/0
    ip vrf forwarding DISTSCHOOLS
    ip address 10.24.7.17 255.255.255.240
    service-module ip address 10.24.7.18 255.255.255.240
    !Application: VMware ESXi 4.1.0 build-348481 running on SRE-V
    service-module ip default-gateway 10.24.7.17
    hold-queue 60 out
    interface SM1/1
    no ip address
    end

    no real timeline when the problem manifests

    I have added the module to our NMS system therefore should have statistics going forward on when the issue happens
    --
    To respond to this post, please click the following link:

    <http://developer.cisco.com/web/srev/forums/-/message_boards/view_message/4089858>

    or simply reply to this email.



    ---------------------------------------------------------------------
    DISCLAIMER: This email and files transmitted are
    confidential and are intended solely for the use of the
    intended recipient. If you are not the intended
    recipient, or the person responsible for delivering it to
    the intended recipient, you may not copy, disclose,
    distribute or use it in any unauthorised manner. If you
    have received this email in error please notify us by
    email to postmaster@wolverhampton.gov.uk and then delete
    it and any attachments accompanying it. Please note that
    Wolverhampton City Council cannot guarantee that this
    message or any attachments are virus free or have not been
    intercepted and amended.
    Any views or opinions expressed within this email are
    those of the author and may not necessarily reflect those
    of Wolverhampton City Council and no contractual
    arrangement is intended to arise from this communication.
    ---------------------------------------------------------------------
    --
    To respond to this post, please click the following link:
    <http://developer.cisco.com/web/srev/forums/-/message_boards/view_message/4098425>
    or simply reply to this email.
    --
    To respond to this post, please click the following link:
    <http://developer.cisco.com/web/srev/forums/-/message_boards/view_message/4098543>
    or simply reply to this email.



    ---------------------------------------------------------------------
    DISCLAIMER: This email and files transmitted are
    confidential and are intended solely for the use of the
    intended recipient.  If you are not the intended
    recipient, or the person responsible for delivering it to
    the intended recipient, you may not copy, disclose,
    distribute or use it in any unauthorised manner.  If you
    have received this email in error please notify us by
    email to postmaster@wolverhampton.gov.uk and then delete
    it and any attachments accompanying it.  Please note that
    Wolverhampton City Council cannot guarantee that this
    message or any attachments are virus free or have not been
    intercepted and amended.
    Any views or opinions expressed within this email are
    those of the author and may not necessarily reflect those
    of Wolverhampton City Council and no contractual
    arrangement is intended to arise from this communication.
    ---------------------------------------------------------------------

    Subject: RE: lost of access to hypervisor from remote networks
    Replied by: ISMAIL PATEL on 21-06-2011 05:51:14 PM
    upgraded. Will monitor

    Cisco IOS Software, C2900 Software (C2900-UNIVERSALK9-M), Version 15.1(4)M, RELE
    ASE SOFTWARE (fc1)
    Technical Support: http://www.cisco.com/techsupport
    Copyright (c) 1986-2011 by Cisco Systems, Inc.
    Compiled Thu 24-Mar-11 15:31 by prod_rel_team

    ROM: System Bootstrap, Version 15.0(1r)M1, RELEASE SOFTWARE (fc1)

    TESTSCHRTR101742 uptime is 3 minutes
    System returned to ROM by reload at 22:48:10 BST Tue Jun 21 2011
    System image file is "flash0:c2900-universalk9-mz.SPA.151-4.M.bin"
    Last reload type: Normal Reload
    Last reload reason: Reload Command

    Subject: RE: New Message from ISMAIL PATEL in Service Ready Engine Virtualization -
    Replied by: Radhika Miriyala on 21-06-2011 05:53:37 PM
    Please Go ahead and upgrade the IOS image as suggested by Brett. If you still see the issue after the IOS upgrade we can discuss further steps to debug.



    From: Cisco Developer Community Forums [mailto:cdicuser@developer.cisco.com]
    Sent: Tuesday, June 21, 2011 2:15 PM
    To: cdicuser@developer.cisco.com
    Subject: New Message from ISMAIL PATEL in Service Ready Engine Virtualization - SRE-V Beta Tester Questions: RE: New Message from Radhika Miriyala in Service Ready Engine Virtualizatio



    ISMAIL PATEL has created a new message in the forum "SRE-V Beta Tester Questions":
    --------------------------------------------------------------
    Hi Radhika



    Good to hear from you.



    Configuration is as below: ( Full config is in the post relating to vrf )



    interface SM1/0

    ip vrf forwarding DISTSCHOOLS

    ip address 10.24.7.17 255.255.255.240

    service-module ip address 10.24.7.18 255.255.255.240

    !Application: VMware ESXi 4.1.0 build-348481 running on SRE-V

    service-module ip default-gateway 10.24.7.17

    hold-queue 60 out

    !

    interface SM1/1

    no ip address



    In terms of the questions:



    1) I create a virtual switch port group under vswitch0 which I moved my virtual machines to

    2) Yes ¿ output below



    TESTSCHRTR101742#ping vrf DISTSCHOOLS 10.24.7.17



    Type escape sequence to abort.

    Sending 5, 100-byte ICMP Echos to 10.24.7.17, timeout is 2 seconds:

    !!!!!

    Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms

    TESTSCHRTR101742#ping vrf DISTSCHOOLS 10.24.7.18



    Type escape sequence to abort.

    Sending 5, 100-byte ICMP Echos to 10.24.7.18, timeout is 2 seconds:

    !!!!!

    Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/4 ms



    3) No teaming enabled on nic1 ¿ it is unused



    As per Brett¿s suggestion ¿ I will upgrade the IOS to the recommended version and see if this resolves the issue.



    Are there any commands you want me to run before I shut the module down?



    Kind regards



    Ismail







    From: Cisco Developer Community Forums [mailto:cdicuser@developer.cisco.com]
    Sent: Tuesday, June 21, 2011 9:46 PM
    To: cdicuser@developer.cisco.com
    Subject: New Message from Radhika Miriyala in Service Ready Engine Virtualization - SRE-V Beta Tester Questions: RE: New Message from ISMAIL PATEL in Service Ready Engine Virtualization -



    Radhika Miriyala has created a new message in the forum "SRE-V Beta Tester Questions":
    --------------------------------------------------------------
    Hi ISMAIL,



    Just want to make sure I understand the issue correctly:



    1) It looks like Your Virtual servers are in the same subnet as that of the hypervisor (ESXi) so did you create a new VM network port group under vswitch1?

    2) When you see the issue of ping have you been able to ping the hypervisor ip address from the router?

    3) Do you have any nic teaming enabled on vsiwtch1?





    Thanks,

    Radhika



    From: Cisco Developer Community Forums [mailto:cdicuser@developer.cisco.com]
    Sent: Tuesday, June 21, 2011 1:26 PM
    To: cdicuser@developer.cisco.com
    Subject: New Message from ISMAIL PATEL in Service Ready Engine Virtualization - SRE-V Beta Tester Questions: RE: New Message from ISMAIL PATEL in Service Ready Engine Virtualization -



    ISMAIL PATEL has created a new message in the forum "SRE-V Beta Tester Questions":
    --------------------------------------------------------------
    Hi



    Loss access again today from remote client but virtual servers are ok.



    Client IP 172.18.2.43



    Ping 10.24.7.17 OK Router

    Ping 10.24.7.18 FAIL Hypervisor

    Ping 10.24.7.19 Virtual server running on service module



    C:\>ping 10.24.7.17



    Pinging 10.24.7.17 with 32 bytes of data:



    Reply from 10.24.7.17: bytes=32 time<1ms TTL=253

    Reply from 10.24.7.17: bytes=32 time<1ms TTL=253



    Ping statistics for 10.24.7.17:

    Packets: Sent = 2, Received = 2, Lost = 0 (0% loss),

    Approximate round trip times in milli-seconds:

    Minimum = 0ms, Maximum = 0ms, Average = 0ms

    Control-C

    ^C

    C:\>ping 10.24.7.18



    Pinging 10.24.7.18 with 32 bytes of data:



    Request timed out.



    Ping statistics for 10.24.7.18:

    Packets: Sent = 1, Received = 0, Lost = 1 (100% loss),

    Control-C

    ^C

    C:\>ping 10.24.7.19



    Pinging 10.24.7.19 with 32 bytes of data:



    Reply from 10.24.7.19: bytes=32 time<1ms TTL=125

    Reply from 10.24.7.19: bytes=32 time<1ms TTL=125



    Ping statistics for 10.24.7.19:

    Packets: Sent = 2, Received = 2, Lost = 0 (0% loss),

    Approximate round trip times in milli-seconds:

    Minimum = 0ms, Maximum = 0ms, Average = 0ms

    Control-C

    ^C





    C:\>tracert -d 10.24.7.18



    Tracing route to 10.24.7.18 over a maximum of 30 hops



    1 <1 ms 1 ms 2 ms 172.18.2.33

    2 2 ms 2 ms 3 ms 172.18.40.9

    3 <1 ms <1 ms <1 ms 172.31.14.39

    4 * * ^C



    Show Ip interface brief



    GigabitEthernet0/0.211 172.31.14.39 YES NVRAM up up





    From virtual Server 10.24.7.19



    C:\>ping 10.24.7.18



    Pinging 10.24.7.18 with 32 bytes of data:



    Reply from 10.24.7.18: bytes=32 time<1ms TTL=64

    Reply from 10.24.7.18: bytes=32 time<1ms TTL=64

    Reply from 10.24.7.18: bytes=32 time<1ms TTL=64

    Reply from 10.24.7.18: bytes=32 time<1ms TTL=64



    Ping statistics for 10.24.7.18:

    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),

    Approximate round trip times in milli-seconds:

    Minimum = 0ms, Maximum = 0ms, Average = 0ms



    From DCUI



    # ping 172.18.2.43

    PING 172.18.2.43 (172.18.2.43): 56 data bytes



    --- 172.18.2.43 ping statistics ---

    3 packets transmitted, 0 packets received, 100% packet loss

    ~ # ping 10.24.7.17

    PING 10.24.7.17 (10.24.7.17): 56 data bytes

    64 bytes from 10.24.7.17: icmp_seq=0 ttl=255 time=0.326 ms

    64 bytes from 10.24.7.17: icmp_seq=1 ttl=255 time=0.333 ms

    64 bytes from 10.24.7.17: icmp_seq=2 ttl=255 time=0.270 ms



    --- 10.24.7.17 ping statistics ---

    3 packets transmitted, 3 packets received, 0% packet loss

    round-trip min/avg/max = 0.270/0.310/0.333 ms



    /sbin # esxcfg-vmknic -l

    Interface Port Group/DVPort IP Family IP Address Netmask Broadcast MAC Address MTU TSO MSS Enabled Type

    vmk0 Management Network IPv4 10.24.7.18 255.255.255.240 10.24.7.31 00:23:eb:a1:07:91 1500 65535 true STATIC

    /sbin # esxcfg-route

    VMkernel default gateway is 10.24.7.17

    /sbin # esxcfg-nics -l

    Name PCI Driver Link Speed Duplex MAC Address MTU Description

    vmnic0 0000:01:00.00 e1000e Up 1000Mbps Full 00:23:eb:a1:07:91 1500 82574L Gigabit Network Connection

    vmnic1 0000:02:00.00 bnx2 Up 1000Mbps Full 88:43:e1:bc:51:e1 1500 Broadcom Corporation Broadcom NetXtreme II BCM5709 1000Base-SX

    vmnic2 0000:02:00.01 bnx2 Up 1000Mbps Full 88:43:e1:bc:51:e0 1500 Broadcom Corporation Broadcom NetXtreme II BCM5709 1000Base-SX





    Are there any other commands, that I can use to see what is going on?





    Thanks



    Ismail







    From: Cisco Developer Community Forums [mailto:cdicuser@developer.cisco.com]
    Sent: Monday, June 20, 2011 10:51 PM
    To: cdicuser@developer.cisco.com
    Subject: New Message from ISMAIL PATEL in Service Ready Engine Virtualization - SRE-V Beta Tester Questions: lost of access to hypervisor from remote networks



    ISMAIL PATEL has created a new message in the forum "SRE-V Beta Tester Questions":

    --------------------------------------------------------------
    Following the upgrade to version 2, we have noticed that we lose access to the hypervisor from remote clients.

    From our client machine ( 172.18.2.43 ) we are unable to access http://10.24.7.18/ or use the vmware infrastructure client to connect to the server.

    The hypervisor is accessible from the local subnet.

    virtual machines running on the hypervisor are not affected

    A reload of the hypervisor resolves the problem for a short time but is service affecting.

    configuration as below:

    interface SM1/0
    ip vrf forwarding DISTSCHOOLS
    ip address 10.24.7.17 255.255.255.240
    service-module ip address 10.24.7.18 255.255.255.240
    !Application: VMware ESXi 4.1.0 build-348481 running on SRE-V
    service-module ip default-gateway 10.24.7.17
    hold-queue 60 out
    interface SM1/1
    no ip address
    end

    no real timeline when the problem manifests

    I have added the module to our NMS system therefore should have statistics going forward on when the issue happens
    --
    To respond to this post, please click the following link:

    <http://developer.cisco.com/web/srev/forums/-/message_boards/view_message/4089858>

    or simply reply to this email.



    ---------------------------------------------------------------------
    DISCLAIMER: This email and files transmitted are
    confidential and are intended solely for the use of the
    intended recipient. If you are not the intended
    recipient, or the person responsible for delivering it to
    the intended recipient, you may not copy, disclose,
    distribute or use it in any unauthorised manner. If you
    have received this email in error please notify us by
    email to postmaster@wolverhampton.gov.uk and then delete
    it and any attachments accompanying it. Please note that
    Wolverhampton City Council cannot guarantee that this
    message or any attachments are virus free or have not been
    intercepted and amended.
    Any views or opinions expressed within this email are
    those of the author and may not necessarily reflect those
    of Wolverhampton City Council and no contractual
    arrangement is intended to arise from this communication.
    ---------------------------------------------------------------------
    --
    To respond to this post, please click the following link:
    <http://developer.cisco.com/web/srev/forums/-/message_boards/view_message/4098425>
    or simply reply to this email.
    --
    To respond to this post, please click the following link:
    <http://developer.cisco.com/web/srev/forums/-/message_boards/view_message/4098543>
    or simply reply to this email.



    ---------------------------------------------------------------------
    DISCLAIMER: This email and files transmitted are
    confidential and are intended solely for the use of the
    intended recipient. If you are not the intended
    recipient, or the person responsible for delivering it to
    the intended recipient, you may not copy, disclose,
    distribute or use it in any unauthorised manner. If you
    have received this email in error please notify us by
    email to postmaster@wolverhampton.gov.uk and then delete
    it and any attachments accompanying it. Please note that
    Wolverhampton City Council cannot guarantee that this
    message or any attachments are virus free or have not been
    intercepted and amended.
    Any views or opinions expressed within this email are
    those of the author and may not necessarily reflect those
    of Wolverhampton City Council and no contractual
    arrangement is intended to arise from this communication.
    ---------------------------------------------------------------------
    --
    To respond to this post, please click the following link:
    <http://developer.cisco.com/web/srev/forums/-/message_boards/view_message/4096956>
    or simply reply to this email.

    Subject: RE: New Message from ISMAIL PATEL in Service Ready Engine Virtualization -
    Replied by: ISMAIL PATEL on 21-06-2011 04:25:37 PM
    Hi



    Loss access again today from remote client but virtual servers are ok.



    Client IP 172.18.2.43



    Ping 10.24.7.17  OK          Router

    Ping 10.24.7.18 FAIL        Hypervisor

    Ping 10.24.7.19                  Virtual server running on service module



    C:\>ping 10.24.7.17



    Pinging 10.24.7.17 with 32 bytes of data:



    Reply from 10.24.7.17: bytes=32 time<1ms TTL=253

    Reply from 10.24.7.17: bytes=32 time<1ms TTL=253



    Ping statistics for 10.24.7.17:

        Packets: Sent = 2, Received = 2, Lost = 0 (0% loss),

    Approximate round trip times in milli-seconds:

        Minimum = 0ms, Maximum = 0ms, Average = 0ms

    Control-C

    ^C

    C:\>ping 10.24.7.18



    Pinging 10.24.7.18 with 32 bytes of data:



    Request timed out.



    Ping statistics for 10.24.7.18:

        Packets: Sent = 1, Received = 0, Lost = 1 (100% loss),

    Control-C

    ^C

    C:\>ping 10.24.7.19



    Pinging 10.24.7.19 with 32 bytes of data:



    Reply from 10.24.7.19: bytes=32 time<1ms TTL=125

    Reply from 10.24.7.19: bytes=32 time<1ms TTL=125



    Ping statistics for 10.24.7.19:

        Packets: Sent = 2, Received = 2, Lost = 0 (0% loss),

    Approximate round trip times in milli-seconds:

        Minimum = 0ms, Maximum = 0ms, Average = 0ms

    Control-C

    ^C





    C:\>tracert -d 10.24.7.18



    Tracing route to 10.24.7.18 over a maximum of 30 hops



      1    <1 ms     1 ms     2 ms  172.18.2.33

      2     2 ms     2 ms     3 ms  172.18.40.9

      3    <1 ms    <1 ms    <1 ms  172.31.14.39

      4     *        *     ^C



    Show Ip interface brief



    GigabitEthernet0/0.211     172.31.14.39    YES NVRAM  up                    up





    From virtual Server 10.24.7.19



    C:\>ping 10.24.7.18



    Pinging 10.24.7.18 with 32 bytes of data:



    Reply from 10.24.7.18: bytes=32 time<1ms TTL=64

    Reply from 10.24.7.18: bytes=32 time<1ms TTL=64

    Reply from 10.24.7.18: bytes=32 time<1ms TTL=64

    Reply from 10.24.7.18: bytes=32 time<1ms TTL=64



    Ping statistics for 10.24.7.18:

        Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),

    Approximate round trip times in milli-seconds:

        Minimum = 0ms, Maximum = 0ms, Average = 0ms



    From DCUI



    # ping 172.18.2.43

    PING 172.18.2.43 (172.18.2.43): 56 data bytes



    --- 172.18.2.43 ping statistics ---

    3 packets transmitted, 0 packets received, 100% packet loss

    ~ # ping 10.24.7.17

    PING 10.24.7.17 (10.24.7.17): 56 data bytes

    64 bytes from 10.24.7.17: icmp_seq=0 ttl=255 time=0.326 ms

    64 bytes from 10.24.7.17: icmp_seq=1 ttl=255 time=0.333 ms

    64 bytes from 10.24.7.17: icmp_seq=2 ttl=255 time=0.270 ms



    --- 10.24.7.17 ping statistics ---

    3 packets transmitted, 3 packets received, 0% packet loss

    round-trip min/avg/max = 0.270/0.310/0.333 ms



    /sbin # esxcfg-vmknic -l

    Interface  Port Group/DVPort   IP Family IP Address                              Netmask         Broadcast       MAC Address       MTU     TSO MSS   Enabled Type

    vmk0       Management Network  IPv4      10.24.7.18                              255.255.255.240 10.24.7.31      00:23:eb:a1:07:91 1500    65535     true    STATIC

    /sbin # esxcfg-route

    VMkernel default gateway is 10.24.7.17

    /sbin # esxcfg-nics -l

    Name    PCI           Driver      Link Speed     Duplex MAC Address       MTU    Description

    vmnic0  0000:01:00.00 e1000e      Up   1000Mbps  Full   00:23:eb:a1:07:91 1500    82574L Gigabit Network Connection

    vmnic1  0000:02:00.00 bnx2        Up   1000Mbps  Full   88:43:e1:bc:51:e1 1500   Broadcom Corporation Broadcom NetXtreme II BCM5709 1000Base-SX

    vmnic2  0000:02:00.01 bnx2        Up   1000Mbps  Full   88:43:e1:bc:51:e0 1500   Broadcom Corporation Broadcom NetXtreme II BCM5709 1000Base-SX





    Are there any other commands, that I can use to see what is going on?





    Thanks



    Ismail







    From: Cisco Developer Community Forums [mailto:cdicuser@developer.cisco.com]
    Sent: Monday, June 20, 2011 10:51 PM
    To: cdicuser@developer.cisco.com
    Subject: New Message from ISMAIL PATEL in Service Ready Engine Virtualization - SRE-V Beta Tester Questions: lost of access to hypervisor from remote networks



    ISMAIL PATEL has created a new message in the forum "SRE-V Beta Tester Questions":

    --------------------------------------------------------------
    Following the upgrade to version 2, we have noticed that we lose access to the hypervisor from remote clients.

    From our client machine ( 172.18.2.43 ) we are unable to access http://10.24.7.18/ or use the vmware infrastructure client to connect to the server.

    The hypervisor is accessible from the local subnet.

    virtual machines running on the hypervisor are not affected

    A reload of the hypervisor resolves the problem for a short time but is service affecting.

    configuration as below:

    interface SM1/0
    ip vrf forwarding DISTSCHOOLS
    ip address 10.24.7.17 255.255.255.240
    service-module ip address 10.24.7.18 255.255.255.240
    !Application: VMware ESXi 4.1.0 build-348481 running on SRE-V
    service-module ip default-gateway 10.24.7.17
    hold-queue 60 out
    interface SM1/1
    no ip address
    end

    no real timeline when the problem manifests

    I have added the module to our NMS system therefore should have statistics going forward  on when the issue happens
    --
    To respond to this post, please click the following link:

    <http://developer.cisco.com/web/srev/forums/-/message_boards/view_message/4089858>

    or simply reply to this email.



    ---------------------------------------------------------------------
    DISCLAIMER: This email and files transmitted are
    confidential and are intended solely for the use of the
    intended recipient.  If you are not the intended
    recipient, or the person responsible for delivering it to
    the intended recipient, you may not copy, disclose,
    distribute or use it in any unauthorised manner.  If you
    have received this email in error please notify us by
    email to postmaster@wolverhampton.gov.uk and then delete
    it and any attachments accompanying it.  Please note that
    Wolverhampton City Council cannot guarantee that this
    message or any attachments are virus free or have not been
    intercepted and amended.
    Any views or opinions expressed within this email are
    those of the author and may not necessarily reflect those
    of Wolverhampton City Council and no contractual
    arrangement is intended to arise from this communication.
    ---------------------------------------------------------------------

    Subject: RE: New Message from ISMAIL PATEL in Service Ready Engine Virtualization -
    Replied by: Radhika Miriyala on 21-06-2011 04:45:37 PM
    Hi ISMAIL,



    Just want to make sure I understand the issue correctly:

               

    1)      It looks like Your Virtual servers are in the same subnet as that of the hypervisor (ESXi) so did you create a  new VM network port group under vswitch1?

    2)      When you see the issue of ping have you been able to ping the hypervisor ip address from the router?

    3)      Do you have any nic teaming enabled on vsiwtch1?





    Thanks,

    Radhika



    From: Cisco Developer Community Forums [mailto:cdicuser@developer.cisco.com]
    Sent: Tuesday, June 21, 2011 1:26 PM
    To: cdicuser@developer.cisco.com
    Subject: New Message from ISMAIL PATEL in Service Ready Engine Virtualization - SRE-V Beta Tester Questions: RE: New Message from ISMAIL PATEL in Service Ready Engine Virtualization -



    ISMAIL PATEL has created a new message in the forum "SRE-V Beta Tester Questions":
    --------------------------------------------------------------
    Hi



    Loss access again today from remote client but virtual servers are ok.



    Client IP 172.18.2.43



    Ping 10.24.7.17 OK Router

    Ping 10.24.7.18 FAIL Hypervisor

    Ping 10.24.7.19 Virtual server running on service module



    C:\>ping 10.24.7.17



    Pinging 10.24.7.17 with 32 bytes of data:



    Reply from 10.24.7.17: bytes=32 time<1ms TTL=253

    Reply from 10.24.7.17: bytes=32 time<1ms TTL=253



    Ping statistics for 10.24.7.17:

    Packets: Sent = 2, Received = 2, Lost = 0 (0% loss),

    Approximate round trip times in milli-seconds:

    Minimum = 0ms, Maximum = 0ms, Average = 0ms

    Control-C

    ^C

    C:\>ping 10.24.7.18



    Pinging 10.24.7.18 with 32 bytes of data:



    Request timed out.



    Ping statistics for 10.24.7.18:

    Packets: Sent = 1, Received = 0, Lost = 1 (100% loss),

    Control-C

    ^C

    C:\>ping 10.24.7.19



    Pinging 10.24.7.19 with 32 bytes of data:



    Reply from 10.24.7.19: bytes=32 time<1ms TTL=125

    Reply from 10.24.7.19: bytes=32 time<1ms TTL=125



    Ping statistics for 10.24.7.19:

    Packets: Sent = 2, Received = 2, Lost = 0 (0% loss),

    Approximate round trip times in milli-seconds:

    Minimum = 0ms, Maximum = 0ms, Average = 0ms

    Control-C

    ^C





    C:\>tracert -d 10.24.7.18



    Tracing route to 10.24.7.18 over a maximum of 30 hops



    1 <1 ms 1 ms 2 ms 172.18.2.33

    2 2 ms 2 ms 3 ms 172.18.40.9

    3 <1 ms <1 ms <1 ms 172.31.14.39

    4 * * ^C



    Show Ip interface brief



    GigabitEthernet0/0.211 172.31.14.39 YES NVRAM up up





    From virtual Server 10.24.7.19



    C:\>ping 10.24.7.18



    Pinging 10.24.7.18 with 32 bytes of data:



    Reply from 10.24.7.18: bytes=32 time<1ms TTL=64

    Reply from 10.24.7.18: bytes=32 time<1ms TTL=64

    Reply from 10.24.7.18: bytes=32 time<1ms TTL=64

    Reply from 10.24.7.18: bytes=32 time<1ms TTL=64



    Ping statistics for 10.24.7.18:

    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),

    Approximate round trip times in milli-seconds:

    Minimum = 0ms, Maximum = 0ms, Average = 0ms



    From DCUI



    # ping 172.18.2.43

    PING 172.18.2.43 (172.18.2.43): 56 data bytes



    --- 172.18.2.43 ping statistics ---

    3 packets transmitted, 0 packets received, 100% packet loss

    ~ # ping 10.24.7.17

    PING 10.24.7.17 (10.24.7.17): 56 data bytes

    64 bytes from 10.24.7.17: icmp_seq=0 ttl=255 time=0.326 ms

    64 bytes from 10.24.7.17: icmp_seq=1 ttl=255 time=0.333 ms

    64 bytes from 10.24.7.17: icmp_seq=2 ttl=255 time=0.270 ms



    --- 10.24.7.17 ping statistics ---

    3 packets transmitted, 3 packets received, 0% packet loss

    round-trip min/avg/max = 0.270/0.310/0.333 ms



    /sbin # esxcfg-vmknic -l

    Interface Port Group/DVPort IP Family IP Address Netmask Broadcast MAC Address MTU TSO MSS Enabled Type

    vmk0 Management Network IPv4 10.24.7.18 255.255.255.240 10.24.7.31 00:23:eb:a1:07:91 1500 65535 true STATIC

    /sbin # esxcfg-route

    VMkernel default gateway is 10.24.7.17

    /sbin # esxcfg-nics -l

    Name PCI Driver Link Speed Duplex MAC Address MTU Description

    vmnic0 0000:01:00.00 e1000e Up 1000Mbps Full 00:23:eb:a1:07:91 1500 82574L Gigabit Network Connection

    vmnic1 0000:02:00.00 bnx2 Up 1000Mbps Full 88:43:e1:bc:51:e1 1500 Broadcom Corporation Broadcom NetXtreme II BCM5709 1000Base-SX

    vmnic2 0000:02:00.01 bnx2 Up 1000Mbps Full 88:43:e1:bc:51:e0 1500 Broadcom Corporation Broadcom NetXtreme II BCM5709 1000Base-SX





    Are there any other commands, that I can use to see what is going on?





    Thanks



    Ismail







    From: Cisco Developer Community Forums [mailto:cdicuser@developer.cisco.com]
    Sent: Monday, June 20, 2011 10:51 PM
    To: cdicuser@developer.cisco.com
    Subject: New Message from ISMAIL PATEL in Service Ready Engine Virtualization - SRE-V Beta Tester Questions: lost of access to hypervisor from remote networks



    ISMAIL PATEL has created a new message in the forum "SRE-V Beta Tester Questions":

    --------------------------------------------------------------
    Following the upgrade to version 2, we have noticed that we lose access to the hypervisor from remote clients.

    From our client machine ( 172.18.2.43 ) we are unable to access http://10.24.7.18/ or use the vmware infrastructure client to connect to the server.

    The hypervisor is accessible from the local subnet.

    virtual machines running on the hypervisor are not affected

    A reload of the hypervisor resolves the problem for a short time but is service affecting.

    configuration as below:

    interface SM1/0
    ip vrf forwarding DISTSCHOOLS
    ip address 10.24.7.17 255.255.255.240
    service-module ip address 10.24.7.18 255.255.255.240
    !Application: VMware ESXi 4.1.0 build-348481 running on SRE-V
    service-module ip default-gateway 10.24.7.17
    hold-queue 60 out
    interface SM1/1
    no ip address
    end

    no real timeline when the problem manifests

    I have added the module to our NMS system therefore should have statistics going forward on when the issue happens
    --
    To respond to this post, please click the following link:

    <http://developer.cisco.com/web/srev/forums/-/message_boards/view_message/4089858>

    or simply reply to this email.



    ---------------------------------------------------------------------
    DISCLAIMER: This email and files transmitted are
    confidential and are intended solely for the use of the
    intended recipient. If you are not the intended
    recipient, or the person responsible for delivering it to
    the intended recipient, you may not copy, disclose,
    distribute or use it in any unauthorised manner. If you
    have received this email in error please notify us by
    email to postmaster@wolverhampton.gov.uk and then delete
    it and any attachments accompanying it. Please note that
    Wolverhampton City Council cannot guarantee that this
    message or any attachments are virus free or have not been
    intercepted and amended.
    Any views or opinions expressed within this email are
    those of the author and may not necessarily reflect those
    of Wolverhampton City Council and no contractual
    arrangement is intended to arise from this communication.
    ---------------------------------------------------------------------
    --
    To respond to this post, please click the following link:
    <http://developer.cisco.com/web/srev/forums/-/message_boards/view_message/4098425>
    or simply reply to this email.

    Subject: RE: New Message from ISMAIL PATEL in Service Ready Engine Virtualization -
    Replied by: Brett Tiller on 21-06-2011 04:59:37 PM
    Hi Ismail,



    You definitely should be using the 151-4M IOS release.  You can download this release from our tech center here:  http://developer.cisco.com/web/srev/docs



    Thanks,



    Brett Tiller

    Custom Application Engineer

    http://developer.cisco.com/web/axp <http://developer.cisco.com/web/axp>

    http://developer.cisco.com/web/srev <http://developer.cisco.com/web/srev>





    From: Cisco Developer Community Forums [mailto:cdicuser@developer.cisco.com]
    Sent: Tuesday, June 21, 2011 1:28 PM
    To: cdicuser@developer.cisco.com
    Subject: New Message from ISMAIL PATEL in Service Ready Engine Virtualization - SRE-V Beta Tester Questions: RE: lost of access to hypervisor from remote networks



    ISMAIL PATEL has created a new message in the forum "SRE-V Beta Tester Questions":

    --------------------------------------------------------------
    Next option would be to upgrade to 151-4M Please advise.

    Thanks

    Ismail
    --
    To respond to this post, please click the following link:

    <http://developer.cisco.com/web/srev/forums/-/message_boards/view_message/4096839>

    or simply reply to this email.

    Subject: RE: lost of access to hypervisor from remote networks
    Replied by: Brett Tiller on 22-06-2011 06:26:45 PM
    Hi Ismail,

    Please send us the data below so that we can begin to do an analysis of what's happening on your system.

    1.  From the vSphere Client, please export the 'System Logs' and send over.

    2.  On the router before and after the failure occurs please type 'show ip route' and send us the results.  We want to see if the routes have changed compared to when working and when broken.

    3.  Please make sure the router is printing logs to the terminal, and send us the captured logs before/after the failure occurs.

    Thanks,

    Brett

    Subject: RE: lost of access to hypervisor from remote networks
    Replied by: ISMAIL PATEL on 22-06-2011 05:59:21 PM
    Same issue with new version of software

    Router upgraded at 22:44 GMT. Connectivity loss 02:55

    Regards

    Ismail

    Subject: FW: New Message from Brett Tiller in Service Ready Engine Virtualization -
    Replied by: ISMAIL PATEL on 23-06-2011 09:09:37 AM
    Hi Brett



    The email system blocked this due to the size of the logs. Is there anywhere I can FTP them to you



    Thanks



    From: Ismail Patel
    Sent: Thursday, June 23, 2011 12:01 AM
    To: cdicuser@developer.cisco.com
    Cc: Brett Tiller (brtiller) (brtiller@cisco.com)
    Subject: RE: New Message from Brett Tiller in Service Ready Engine Virtualization - SRE-V Beta Tester Questions: RE: lost of access to hypervisor from remote networks



    Hi Brett,



    Information you requested attached and below. I am not using any dynamic routing protocol. Upsteam router can ping virtual machine on the same subnet as management.



    I cannot give you a before show Ip route, as this would require me to reload module.



    Hope this helps.



    Ismail



    TESTSCHRTR101742#sh ip route

    Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP

           D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area

           N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2

           E1 - OSPF external type 1, E2 - OSPF external type 2

           i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2

           ia - IS-IS inter area, * - candidate default, U - per-user static route

           o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP

           + - replicated route, % - next hop override



    Gateway of last resort is 172.31.12.1 to network 0.0.0.0



    S*    0.0.0.0/0 [1/0] via 172.31.12.1

          1.0.0.0/32 is subnetted, 1 subnets

    C        1.1.1.1 is directly connected, Loopback1

          10.0.0.0/8 is variably subnetted, 2 subnets, 2 masks

    C        10.151.15.32/28 is directly connected, GigabitEthernet0/2

    L        10.151.15.33/32 is directly connected, GigabitEthernet0/2

          172.31.0.0/16 is variably subnetted, 2 subnets, 2 masks

    C        172.31.12.0/24 is directly connected, GigabitEthernet0/0.209

    L        172.31.12.39/32 is directly connected, GigabitEthernet0/0.209

    TESTSCHRTR101742#sh ip route vrf DISTSCHOOLS



    Routing Table: DISTSCHOOLS

    Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP

           D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area

           N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2

           E1 - OSPF external type 1, E2 - OSPF external type 2

           i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2

           ia - IS-IS inter area, * - candidate default, U - per-user static route

           o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP

           + - replicated route, % - next hop override



    Gateway of last resort is 172.31.14.1 to network 0.0.0.0



    S*    0.0.0.0/0 [1/0] via 172.31.14.1

          10.0.0.0/8 is variably subnetted, 5 subnets, 3 masks

    C        10.24.0.0/22 is directly connected, Vlan2

    L        10.24.0.1/32 is directly connected, Vlan2

    C        10.24.7.1/32 is directly connected, Loopback0

    C        10.24.7.16/28 is directly connected, SM1/0

    L        10.24.7.17/32 is directly connected, SM1/0

          172.31.0.0/16 is variably subnetted, 2 subnets, 2 masks

    C        172.31.14.0/24 is directly connected, GigabitEthernet0/0.211

    L        172.31.14.39/32 is directly connected, GigabitEthernet0/0.211



    TESTSCHRTR101742#sh ip arp vrf DISTSCHOOLS

    Protocol  Address          Age (min)  Hardware Addr   Type   Interface

    Internet  10.24.0.1               -   0025.4523.8943  ARPA   Vlan2

    Internet  10.24.0.10            235   0004.76aa.9f92  ARPA   Vlan2

    Internet  10.24.0.11              4   0021.7074.b732  ARPA   Vlan2

    Internet  10.24.4.1               -   0025.4523.8943  ARPA   Vlan3

    Internet  10.24.5.1               -   0025.4523.8943  ARPA   Vlan4

    Internet  10.24.6.1               -   0025.4523.8943  ARPA   Vlan6

    Internet  10.24.7.17              -   0025.4523.8950  ARPA   SM1/0

    Internet  10.24.7.18             24   0023.eba1.0791  ARPA   SM1/0

    Internet  10.24.7.19              2   000c.29c3.cfa5  ARPA   SM1/0

    Internet  10.24.7.20              1   000c.29b0.ac6c  ARPA   SM1/0

    Internet  10.24.7.21            256   000c.2906.d5c8  ARPA   SM1/0

    Internet  10.24.7.30             53   000c.29f0.ff07  ARPA   SM1/0





    Upstream router ( cisco 3750G )



    CR-F2-20920#sh ip route vrf DISTSCHOOLS 10.24.7.18

    Routing entry for 10.24.0.0/21

      Known via "static", distance 1, metric 0

      Redistributing via ospf 3

      Advertised by ospf 3 subnets

      Routing Descriptor Blocks:

      * 172.31.14.39

          Route metric is 0, traffic share count is 1



    CR-F2-20920#ping vrf DISTSCHOOLS 10.24.7.17



    Type escape sequence to abort.

    Sending 5, 100-byte ICMP Echos to 10.24.7.17, timeout is 2 seconds:

    !!!!!

    Success rate is 100 percent (5/5), round-trip min/avg/max = 1/3/8 ms

    CR-F2-20920#ping vrf DISTSCHOOLS 10.24.7.18



    Type escape sequence to abort.

    Sending 5, 100-byte ICMP Echos to 10.24.7.18, timeout is 2 seconds:

    .....

    Success rate is 0 percent (0/5)

    CR-F2-20920#ping vrf DISTSCHOOLS 10.24.7.19



    Type escape sequence to abort.

    Sending 5, 100-byte ICMP Echos to 10.24.7.19, timeout is 2 seconds:

    !!!!!

    Success rate is 100 percent (5/5), round-trip min/avg/max = 1/4/9 ms





    From: Cisco Developer Community Forums [mailto:cdicuser@developer.cisco.com]
    Sent: Wednesday, June 22, 2011 11:27 PM
    To: cdicuser@developer.cisco.com
    Subject: New Message from Brett Tiller in Service Ready Engine Virtualization - SRE-V Beta Tester Questions: RE: lost of access to hypervisor from remote networks



    Brett Tiller has created a new message in the forum "SRE-V Beta Tester Questions":

    --------------------------------------------------------------
    Hi Ismail,

    Please send us the data below so that we can begin to do an analysis of what's happening on your system.

    1. From the vSphere Client, please export the 'System Logs' and send over.

    2. On the router before and after the failure occurs please type 'show ip route' and send us the results. We want to see if the routes have changed compared to when working and when broken.

    3. Please make sure the router is printing logs to the terminal, and send us the captured logs before/after the failure occurs.

    Thanks,

    Brett
    --
    To respond to this post, please click the following link:

    <http://developer.cisco.com/web/srev/forums/-/message_boards/view_message/4101981>

    or simply reply to this email.



    ---------------------------------------------------------------------
    DISCLAIMER: This email and files transmitted are
    confidential and are intended solely for the use of the
    intended recipient.  If you are not the intended
    recipient, or the person responsible for delivering it to
    the intended recipient, you may not copy, disclose,
    distribute or use it in any unauthorised manner.  If you
    have received this email in error please notify us by
    email to postmaster@wolverhampton.gov.uk and then delete
    it and any attachments accompanying it.  Please note that
    Wolverhampton City Council cannot guarantee that this
    message or any attachments are virus free or have not been
    intercepted and amended.
    Any views or opinions expressed within this email are
    those of the author and may not necessarily reflect those
    of Wolverhampton City Council and no contractual
    arrangement is intended to arise from this communication.
    ---------------------------------------------------------------------

    Subject: FW: New Message from Brett Tiller in Service Ready Engine Virtualization -
    Replied by: ISMAIL PATEL on 23-06-2011 09:09:37 AM
    Another one that did not get through



    From: Ismail Patel
    Sent: Thursday, June 23, 2011 12:44 AM
    To: cdicuser@developer.cisco.com
    Cc: Brett Tiller (brtiller) (brtiller@cisco.com)
    Subject: RE: New Message from Brett Tiller in Service Ready Engine Virtualization - SRE-V Beta Tester Questions: RE: lost of access to hypervisor from remote networks



    Unexpected service module restart ¿ I restarted the management interface to see if this restored connectivity which it did not. Left it to do some work and came back to find my virtual servers had rebooted.



    From virtual windows server log



    The previous system shutdown at 00:15:38 on 23/06/2011 was unexpected.



    For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.







    Nothing in router log:



    *Jun 22 23:27:16.525: %LINK-3-UPDOWN: Interface GigabitEthernet0/2, changed stat

    e to down

    *Jun 22 23:27:19.525: %LINK-3-UPDOWN: Interface GigabitEthernet0/2, changed stat

    e to up

    *Jun 22 23:27:20.525: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEth

    ernet0/2, changed state to up

    *Jun 22 23:27:55.805: %SRE_SM-6-STATE_CHANGE: SM1/0 changing state from SERVICE_

    MODULE_STATE_ERRQ to SERVICE_MODULE_STATE_STDY



    Service module on-line from remote client



    TESTSCHRTR101742#show ip route vrf DISTSCHOOLS



    Routing Table: DISTSCHOOLS

    Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP

           D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area

           N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2

           E1 - OSPF external type 1, E2 - OSPF external type 2

           i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2

           ia - IS-IS inter area, * - candidate default, U - per-user static route

           o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP

           + - replicated route, % - next hop override



    Gateway of last resort is 172.31.14.1 to network 0.0.0.0



    S*    0.0.0.0/0 [1/0] via 172.31.14.1

          10.0.0.0/8 is variably subnetted, 5 subnets, 3 masks

    C        10.24.0.0/22 is directly connected, Vlan2

    L        10.24.0.1/32 is directly connected, Vlan2

    C        10.24.7.1/32 is directly connected, Loopback0

    C        10.24.7.16/28 is directly connected, SM1/0

    L        10.24.7.17/32 is directly connected, SM1/0

          172.31.0.0/16 is variably subnetted, 2 subnets, 2 masks

    C        172.31.14.0/24 is directly connected, GigabitEthernet0/0.211

    L        172.31.14.39/32 is directly connected, GigabitEthernet0/0.211



    Attached are the latest logs from server



    Many thanks



    Ismail









    ---------------------------------------------------------------------
    DISCLAIMER: This email and files transmitted are
    confidential and are intended solely for the use of the
    intended recipient.  If you are not the intended
    recipient, or the person responsible for delivering it to
    the intended recipient, you may not copy, disclose,
    distribute or use it in any unauthorised manner.  If you
    have received this email in error please notify us by
    email to postmaster@wolverhampton.gov.uk and then delete
    it and any attachments accompanying it.  Please note that
    Wolverhampton City Council cannot guarantee that this
    message or any attachments are virus free or have not been
    intercepted and amended.
    Any views or opinions expressed within this email are
    those of the author and may not necessarily reflect those
    of Wolverhampton City Council and no contractual
    arrangement is intended to arise from this communication.
    ---------------------------------------------------------------------

    Subject: RE: New Message from ISMAIL PATEL in Service Ready Engine Virtualization -
    Replied by: Radhika Miriyala on 23-06-2011 01:19:37 PM
    Ismail,
    When you say you restarted the management interface are you talking about the network interface or the management agents?

    I see the follwoing "*Jun 22 23:27:55.805: %SRE_SM-6-STATE_CHANGE: SM1/0 changing state from SERVICE_

    MODULE_STATE_ERRQ to SERVICE_MODULE_STATE_STDY"
    Do you see any messages before that where the router is trying to reset the Service module? There is a service in the SM that does heatbeat with the router, If for some reason the heartbeat from the module does not ahppen for 20 minutes or so the router resets the service-module thinking there is some issue with the module. This is what could have happened.

    the console logs on thr router will help us understand what heppened.

    Thanks,
    Radhika





    ________________________________

    From: Cisco Developer Community Forums [mailto:cdicuser@developer.cisco.com]
    Sent: Thu 6/23/2011 6:09 AM
    To: cdicuser@developer.cisco.com
    Subject: New Message from ISMAIL PATEL in Service Ready Engine Virtualization - SRE-V Beta Tester Questions: FW: New Message from Brett Tiller in Service Ready Engine Virtualization -


    ISMAIL PATEL has created a new message in the forum "SRE-V Beta Tester Questions":
    --------------------------------------------------------------
    Another one that did not get through



    From: Ismail Patel
    Sent: Thursday, June 23, 2011 12:44 AM
    To: cdicuser@developer.cisco.com
    Cc: Brett Tiller (brtiller) (brtiller@cisco.com)
    Subject: RE: New Message from Brett Tiller in Service Ready Engine Virtualization - SRE-V Beta Tester Questions: RE: lost of access to hypervisor from remote networks



    Unexpected service module restart - I restarted the management interface to see if this restored connectivity which it did not. Left it to do some work and came back to find my virtual servers had rebooted.



    From virtual windows server log



    The previous system shutdown at 00:15:38 on 23/06/2011 was unexpected.



    For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.







    Nothing in router log:



    *Jun 22 23:27:16.525: %LINK-3-UPDOWN: Interface GigabitEthernet0/2, changed stat

    e to down

    *Jun 22 23:27:19.525: %LINK-3-UPDOWN: Interface GigabitEthernet0/2, changed stat

    e to up

    *Jun 22 23:27:20.525: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEth

    ernet0/2, changed state to up

    *Jun 22 23:27:55.805: %SRE_SM-6-STATE_CHANGE: SM1/0 changing state from SERVICE_

    MODULE_STATE_ERRQ to SERVICE_MODULE_STATE_STDY



    Service module on-line from remote client



    TESTSCHRTR101742#show ip route vrf DISTSCHOOLS



    Routing Table: DISTSCHOOLS

    Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP

    D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area

    N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2

    E1 - OSPF external type 1, E2 - OSPF external type 2

    i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2

    ia - IS-IS inter area, * - candidate default, U - per-user static route

    o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP

    + - replicated route, % - next hop override



    Gateway of last resort is 172.31.14.1 to network 0.0.0.0



    S* 0.0.0.0/0 [1/0] via 172.31.14.1

    10.0.0.0/8 is variably subnetted, 5 subnets, 3 masks

    C 10.24.0.0/22 is directly connected, Vlan2

    L 10.24.0.1/32 is directly connected, Vlan2

    C 10.24.7.1/32 is directly connected, Loopback0

    C 10.24.7.16/28 is directly connected, SM1/0

    L 10.24.7.17/32 is directly connected, SM1/0

    172.31.0.0/16 is variably subnetted, 2 subnets, 2 masks

    C 172.31.14.0/24 is directly connected, GigabitEthernet0/0.211

    L 172.31.14.39/32 is directly connected, GigabitEthernet0/0.211



    Attached are the latest logs from server



    Many thanks



    Ismail









    ---------------------------------------------------------------------
    DISCLAIMER: This email and files transmitted are
    confidential and are intended solely for the use of the
    intended recipient. If you are not the intended
    recipient, or the person responsible for delivering it to
    the intended recipient, you may not copy, disclose,
    distribute or use it in any unauthorised manner. If you
    have received this email in error please notify us by
    email to postmaster@wolverhampton.gov.uk and then delete
    it and any attachments accompanying it. Please note that
    Wolverhampton City Council cannot guarantee that this
    message or any attachments are virus free or have not been
    intercepted and amended.
    Any views or opinions expressed within this email are
    those of the author and may not necessarily reflect those
    of Wolverhampton City Council and no contractual
    arrangement is intended to arise from this communication.
    ---------------------------------------------------------------------
    --
    To respond to this post, please click the following link:
    <http://developer.cisco.com/web/srev/forums/-/message_boards/view_message/4108767>
    or simply reply to this email.

    Subject: RE: lost of access to hypervisor from remote networks
    Replied by: Brett Tiller on 23-06-2011 06:41:05 PM
    Hi Ismail,

    Based upon the router logs you've provided above, it appears that the gig0/2 interface is flapping.  Are you seeing this up/down status on the interface constantly?  If so, were you seeing this same status when you were using IOS 15.1-3T?

    Thanks,

    Brett

    Subject: RE: lost of access to hypervisor from remote networks
    Replied by: Brett Tiller on 23-06-2011 01:22:43 PM
    HI Ismail,

    Please upload the files via the steps below and let us know the filenames as well.

    1. Connect to ftp.cisco.com via anonymous ftp.
    2. Files must be uploaded into a folder called 'incoming', which is not visible to anonymous users
    3. If you are using a text-based ftp client, type the command "cd incoming"
    4. If you are using a GUI ftp client such as FileZilla, enter 'incoming' manually as the destination for the remote site
        Upload the file as normal. You will not be able to see it on the server, as commands to list the files in that folder have been disabled. You may see error messages related to that.

    NOTE: You cannot upload a file with the same name more than once as this creates the potential for other users to overwrite the files you have uploaded.  If you need to re-upload the same file, either use a new filename or ask the Cisco employee you are working with to remove the file from the server.

    Thanks,

    Brett

    Subject: RE: lost of access to hypervisor from remote networks
    Replied by: Brett Tiller on 23-06-2011 01:33:17 PM
    Hi Ismail,

    Regarding the 'show ip route' command on the router.  Please run that and provide the output when your system is running properly and again when the communication fails.

    Thanks,

    Brett

    Subject: RE: lost of access to hypervisor from remote networks
    Replied by: ISMAIL PATEL on 23-06-2011 04:15:30 PM
    Hi Guys,

    1) Files uploaded

    IEP-10.24.7.18-vmsupport-2011-06-23@00-34-47.tgz
    IEP-WCC-14CA5C74E84-viclient-support-2011-06-23@00-35-11.zip

    2) Reset of management network was done from the console from the menu option

    3) Show ip route command output from working module is in the above post when after the module rebooted.

    The previous post had the output from when it was not accessible from remote network.

    4) show log below:

    *Jun 21 21:49:12.459: %CRYPTO-6-ISAKMP_ON_OFF: ISAKMP is OFF
    *Jun 21 21:49:12.459: %CRYPTO-6-GDOI_ON_OFF: GDOI is OFF
    *Jun 21 21:49:12.459: %CRYPTO-6-ISAKMP_ON_OFF: ISAKMP is OFF
    *Jun 21 21:49:12.459: %CRYPTO-6-GDOI_ON_OFF: GDOI is OFF
    *Jun 21 21:49:12.512: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 10.130.2.2 p
    ort 514 started - CLI initiated
    *Jun 21 21:49:13.180: %LINK-3-UPDOWN: Interface FastEthernet0/0/3, changed state
    to up
    *Jun 21 21:49:41.668: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan2, cha
    nged state to up
    *Jun 21 21:49:56.480: %SRE_SM-6-STATE_CHANGE: SM1/0 changing state from SERVICE_
    MODULE_STATE_WREG to SERVICE_MODULE_STATE_STDY
    *Jun 22 23:11:37.713: %SRE_SM-6-STATE_CHANGE: SM1/0 changing state from SERVICE_
    MODULE_STATE_STDY to SERVICE_MODULE_STATE_WREG
    *Jun 22 23:24:39.055: %SRE_SM-6-STATE_CHANGE: SM1/0 changing state from SERVICE_
    MODULE_STATE_WREG to SERVICE_MODULE_STATE_ERRQ
    *Jun 22 23:24:40.523: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEth
    ernet0/2, changed state to down
    *Jun 22 23:24:41.523: %LINK-3-UPDOWN: Interface GigabitEthernet0/2, changed stat
    e to down
    *Jun 22 23:24:43.531: %LINK-3-UPDOWN: Interface GigabitEthernet0/2, changed stat
    e to up
    *Jun 22 23:24:44.531: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEth
    ernet0/2, changed state to up
    *Jun 22 23:24:53.524: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEth
    ernet0/2, changed state to down
    *Jun 22 23:24:54.524: %LINK-3-UPDOWN: Interface GigabitEthernet0/2, changed stat
    e to down
    *Jun 22 23:24:57.532: %LINK-3-UPDOWN: Interface GigabitEthernet0/2, changed stat
    e to up
    *Jun 22 23:24:58.532: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEth
    ernet0/2, changed state to up
    *Jun 22 23:25:54.524: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEth
    ernet0/2, changed state to down
    *Jun 22 23:25:55.524: %LINK-3-UPDOWN: Interface GigabitEthernet0/2, changed stat
    e to down
    *Jun 22 23:25:58.524: %LINK-3-UPDOWN: Interface GigabitEthernet0/2, changed stat
    e to up
    *Jun 22 23:25:59.524: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEth
    ernet0/2, changed state to up
    *Jun 22 23:26:24.576: %SM_INSTALL-6-INST_RBIP: SM1/0 received msg: RBIP Registra
    tion Request
    *Jun 22 23:26:25.524: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEth
    ernet0/2, changed state to down
    *Jun 22 23:26:26.524: %LINK-3-UPDOWN: Interface GigabitEthernet0/2, changed stat
    e to down
    *Jun 22 23:26:33.524: %LINK-3-UPDOWN: Interface GigabitEthernet0/2, changed stat
    e to up
    *Jun 22 23:26:34.524: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEth
    ernet0/2, changed state to up
    *Jun 22 23:27:15.525: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEth
    ernet0/2, changed state to down
    *Jun 22 23:27:16.525: %LINK-3-UPDOWN: Interface GigabitEthernet0/2, changed stat
    e to down
    *Jun 22 23:27:19.525: %LINK-3-UPDOWN: Interface GigabitEthernet0/2, changed stat
    e to up
    *Jun 22 23:27:20.525: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEth
    ernet0/2, changed state to up
    *Jun 22 23:27:55.805: %SRE_SM-6-STATE_CHANGE: SM1/0 changing state from SERVICE_
    MODULE_STATE_ERRQ to SERVICE_MODULE_STATE_STDY

    Ismail

    Subject: RE: lost of access to hypervisor from remote networks
    Replied by: ISMAIL PATEL on 23-06-2011 04:17:07 PM
    The two log files are from after the module restarted. I have log files from before, if required.

    Subject: RE: New Message from Brett Tiller in Service Ready Engine Virtualization -
    Replied by: ISMAIL PATEL on 23-06-2011 07:45:37 PM
    Hi Brett



    Did not notice the interface flapping prior to upgrade and looking at the logs it started straight after the 15.1.4 upgrade. The flapping has stopped as there is nothing reported yesterday ( 23rd ). The interface can be disabled if you think it is related





    Thanks



    Ismail



    From: Cisco Developer Community Forums [mailto:cdicuser@developer.cisco.com]
    Sent: Thursday, June 23, 2011 11:41 PM
    To: cdicuser@developer.cisco.com
    Subject: New Message from Brett Tiller in Service Ready Engine Virtualization - SRE-V Beta Tester Questions: RE: lost of access to hypervisor from remote networks



    Brett Tiller has created a new message in the forum "SRE-V Beta Tester Questions":

    --------------------------------------------------------------
    Hi Ismail,

    Based upon the router logs you've provided above, it appears that the gig0/2 interface is flapping. Are you seeing this up/down status on the interface constantly? If so, were you seeing this same status when you were using IOS 15.1-3T?

    Thanks,

    Brett
    --
    To respond to this post, please click the following link:

    <http://developer.cisco.com/web/srev/forums/-/message_boards/view_message/4110144>

    or simply reply to this email.



    ---------------------------------------------------------------------
    DISCLAIMER: This email and files transmitted are
    confidential and are intended solely for the use of the
    intended recipient.  If you are not the intended
    recipient, or the person responsible for delivering it to
    the intended recipient, you may not copy, disclose,
    distribute or use it in any unauthorised manner.  If you
    have received this email in error please notify us by
    email to postmaster@wolverhampton.gov.uk and then delete
    it and any attachments accompanying it.  Please note that
    Wolverhampton City Council cannot guarantee that this
    message or any attachments are virus free or have not been
    intercepted and amended.
    Any views or opinions expressed within this email are
    those of the author and may not necessarily reflect those
    of Wolverhampton City Council and no contractual
    arrangement is intended to arise from this communication.
    ---------------------------------------------------------------------

    Subject: RE: lost of access to hypervisor from remote networks
    Replied by: ISMAIL PATEL on 27-06-2011 05:44:32 PM
    Hi

    I believe the issue is related to CEF and VRF-lite

    CEF table when service module is accessible from remote networks ( service module is listed )..
    When the module is not accessible then entry for the service module is missing. Working table below

    TESTSCHRTR101742#sh ip cef vrf DISTSCHOOLS
    Prefix               Next Hop             Interface
    0.0.0.0/0            172.31.14.1          GigabitEthernet0/0.211
    0.0.0.0/8            drop
    0.0.0.0/32           receive
    10.24.0.0/22         attached             Vlan2
    10.24.0.0/32         receive              Vlan2
    10.24.0.1/32         receive              Vlan2
    10.24.0.10/32        attached             Vlan2
    10.24.0.11/32        attached             Vlan2
    10.24.3.255/32       receive              Vlan2
    10.24.7.1/32         receive              Loopback0
    10.24.7.16/28        attached             SM1/0
    10.24.7.16/32        receive              SM1/0
    10.24.7.17/32        receive              SM1/0
    10.24.7.18/32        attached             SM1/0
    10.24.7.19/32        attached             SM1/0
    10.24.7.20/32        attached             SM1/0
    10.24.7.31/32        receive              SM1/0

    regards

    Ismail

    Subject: RE: lost of access to hypervisor from remote networks
    Replied by: Brett Tiller on 28-06-2011 06:35:12 PM
    Hi Ismail,

    We'd like to take a two pronged approach to debugging this issue by checking it from the router and VMware.  I've made a few suggestions below.  Please follow the steps and provide us the collected data when the issue recurs.

    1.  Regarding your suggestion of the cef table being the issue, please turn on debugging for this on the router by typing: 'debug ip cef drops' and 'debug ip cef events'.  Also please turn on debugging for the vrf tables by typing: 'debug vrf ipv4' and 'debug vrf mpls'. Make sure logging to the console is turned on and send us the data.

    2.  On the ESXi side please ssh into the DCUI and type: 'esxcfg-advcfg --user-var CiscoRBCPDebugLevel --set-user-var 1' .  When the issue recurs, please send us the vmware tech-support logs.

    Thanks,

    Brett

    Subject: RE: New Message from ISMAIL PATEL in Service Ready Engine Virtualization -
    Replied by: ISMAIL PATEL on 04-07-2011 05:23:13 AM
    Logs uploaded to ftp.cisco.com incoming directory



    IEP-10.24.7.18-vmsupport-2011-07-04@09-09-50.tgz

    IEP-WCC-14CA5C74E84-viclient-support-2011-07-04@09-10-01.zip





    For info, the following restores the remote network access:



    TESTSCHRTR101742#conf t

    Enter configuration commands, one per line.  End with CNTL/Z.

    TESTSCHRTR101742(config)#ip route 10.24.7.18 255.255.255.255 sm 1/0

    TESTSCHRTR101742(config)#no ip route 10.24.7.18 255.255.255.255 sm 1/0

    TESTSCHRTR101742(config)#end



    Many thanks



    Ismail



    ---------------------------------------------------------------------
    DISCLAIMER: This email and files transmitted are
    confidential and are intended solely for the use of the
    intended recipient.  If you are not the intended
    recipient, or the person responsible for delivering it to
    the intended recipient, you may not copy, disclose,
    distribute or use it in any unauthorised manner.  If you
    have received this email in error please notify us by
    email to postmaster@wolverhampton.gov.uk and then delete
    it and any attachments accompanying it.  Please note that
    Wolverhampton City Council cannot guarantee that this
    message or any attachments are virus free or have not been
    intercepted and amended.
    Any views or opinions expressed within this email are
    those of the author and may not necessarily reflect those
    of Wolverhampton City Council and no contractual
    arrangement is intended to arise from this communication.
    ---------------------------------------------------------------------

    Subject: RE: lost of access to hypervisor from remote networks
    Replied by: ISMAIL PATEL on 04-07-2011 04:50:51 AM
    Apologies for the delay in reponse.

    Following a 'typo' on the router i.e. typing in

    "ip route 10.24.7.18 255.255.255.255 sm 1/0 " , immediately followed by a no on the same command, the service module remianed accessible from remote networks.

    The correct format of the command should have been "ip route vrf DISTSCHOOLS 10.24.7.18 255.255.255.255 10.24.7.17" ( vrf-lite lite does not allow you to enter interface as the next hop ).

    Anyway...I reloaded the router and the problem came back. Syslog below:

    TESTSCHRTR101742#sh logging
    Syslog logging: enabled (0 messages dropped, 2 messages rate-limited, 0 flushes,
    0 overruns, xml disabled, filtering disabled)

    No Active Message Discriminator.



    No Inactive Message Discriminator.


        Console logging: level debugging, 24184 messages logged, xml disabled,
                         filtering disabled
        Monitor logging: level debugging, 0 messages logged, xml disabled,
                         filtering disabled
        Buffer logging:  level debugging, 24184 messages logged, xml disabled,
                        filtering disabled
        Exception Logging: size (4096 bytes)
        Count and timestamp logging messages: disabled
        Persistent logging: disabled
        Trap logging: level informational, 55 message lines logged
            Logging to 10.130.2.2  (udp port 514, audit disabled,
                  link up),
                  55 message lines logged,
                  0 message lines rate-limited,
                  0 message lines dropped-by-MD,
                  xml disabled, sequence number disabled
                  filtering disabled

    Log Buffer (8192 bytes):
    10.130.2.11 (Gi0/0.211) to 10.24.7.18, Neighbor resolution req
    *Jul  4 08:09:12.598: CEF-Drop: Packet from 10.24.0.10 (Vl2) to 10.24.0.255, Nei
    ghbor resolution req
    *Jul  4 08:09:14.386: CEF-Drop: Packet from 10.130.2.11 (Gi0/0.211) to 10.24.7.1
    8, Neighbor resolution req
    *Jul  4 08:09:16.682: CEF-Drop: Packet from 10.130.2.11 (Gi0/0.211) to 10.24.7.1
    8, Neighbor resolution req
    *Jul  4 08:09:20.682: CEF-Drop: Packet from 10.130.2.11 (Gi0/0.211) to 10.24.7.1
    8, Neighbor resolution req
    *Jul  4 08:09:40.371: CEF-Drop: Packet from 10.24.0.10 (Vl2) to 10.24.0.255, No
    adjacency
    *Jul  4 08:09:41.123: CEF-Drop: Packet from 10.24.0.10 (Vl2) to 10.24.0.255, No
    adjacency
    *Jul  4 08:09:43.623: CEF-Drop: Packet from 10.24.0.10 (Vl2) to 10.24.0.255, Nei
    ghbor resolution req
    *Jul  4 08:09:49.303: CEF-Drop: Packet from 10.24.0.10 (Vl2) to 10.24.0.255, No
    adjacency
    *Jul  4 08:09:50.055: CEF-Drop: Packet from 10.24.0.10 (Vl2) to 10.24.0.255, No
    adjacency
    *Jul  4 08:09:52.555: CEF-Drop: Packet from 10.24.0.10 (Vl2) to 10.24.0.255, Nei
    ghbor resolution req
    *Jul  4 08:09:53.727: CEF-Drop: Packet from 10.130.2.2 (Gi0/0.211) to 10.24.7.18
    , No adjacency
    *Jul  4 08:09:54.635: CEF-Drop: Packet from 10.130.2.11 (Gi0/0.211) to 10.24.7.1
    8, Neighbor resolution req
    *Jul  4 08:09:56.439: CEF-Drop: Packet from 10.130.2.2 (Gi0/0.211) to 10.24.7.18
    , No adjacency
    *Jul  4 08:09:56.823: CEF-Drop: Packet from 10.130.2.11 (Gi0/0.211) to 10.24.7.1
    8, Neighbor resolution req
    *Jul  4 08:10:00.823: CEF-Drop: Packet from 10.130.2.11 (Gi0/0.211) to 10.24.7.1
    8, Neighbor resolution req
    *Jul  4 08:10:34.995: CEF-Drop: Packet from 10.130.2.11 (Gi0/0.211) to 10.24.7.1
    8, Neighbor resolution req
    *Jul  4 08:10:37.291: CEF-Drop: Packet from 10.130.2.11 (Gi0/0.211) to 10.24.7.1
    8, Neighbor resolution req
    *Jul  4 08:10:41.291: CEF-Drop: Packet from 10.130.2.11 (Gi0/0.211) to 10.24.7.1
    8, Neighbor resolution req
    *Jul  4 08:11:15.463: CEF-Drop: Packet from 10.130.2.11 (Gi0/0.211) to 10.24.7.1
    8, Neighbor resolution req
    *Jul  4 08:11:17.651: CEF-Drop: Packet from 10.130.2.11 (Gi0/0.211) to 10.24.7.1
    8, Neighbor resolution req
    *Jul  4 08:11:21.651: CEF-Drop: Packet from 10.130.2.11 (Gi0/0.211) to 10.24.7.1
    8, Neighbor resolution req
    *Jul  4 08:11:54.636: CEF-Drop: Packet from 10.130.2.2 (Gi0/0.211) to 10.24.7.18
    , No adjacency
    *Jul  4 08:11:55.932: CEF-Drop: Packet from 10.130.2.11 (Gi0/0.211) to 10.24.7.1
    8, Neighbor resolution req
    *Jul  4 08:11:57.440: CEF-Drop: Packet from 10.130.2.2 (Gi0/0.211) to 10.24.7.18
    , No adjacency
    *Jul  4 08:11:58.120: CEF-Drop: Packet from 10.130.2.11 (Gi0/0.211) to 10.24.7.1
    8, Neighbor resolution req
    *Jul  4 08:12:02.120: CEF-Drop: Packet from 10.130.2.11 (Gi0/0.211) to 10.24.7.1
    8, Neighbor resolution req
    *Jul  4 08:12:36.836: CEF-Drop: Packet from 10.130.2.11 (Gi0/0.211) to 10.24.7.1
    8, Neighbor resolution req
    *Jul  4 08:12:39.024: CEF-Drop: Packet from 10.130.2.11 (Gi0/0.211) to 10.24.7.1
    8, Neighbor resolution req
    *Jul  4 08:12:43.024: CEF-Drop: Packet from 10.130.2.11 (Gi0/0.211) to 10.24.7.1
    8, Neighbor resolution req
    *Jul  4 08:13:16.980: CEF-Drop: Packet from 10.130.2.11 (Gi0/0.211) to 10.24.7.1
    8, Neighbor resolution req
    *Jul  4 08:13:19.164: CEF-Drop: Packet from 10.130.2.11 (Gi0/0.211) to 10.24.7.1
    8, Neighbor resolution req
    *Jul  4 08:13:23.164: CEF-Drop: Packet from 10.130.2.11 (Gi0/0.211) to 10.24.7.1
    8, Neighbor resolution req
    *Jul  4 08:13:57.229: CEF-Drop: Packet from 10.130.2.11 (Gi0/0.211) to 10.24.7.1
    8, Neighbor resolution req
    *Jul  4 08:13:59.277: CEF-Drop: Packet from 10.130.2.11 (Gi0/0.211) to 10.24.7.1
    8, Neighbor resolution req
    *Jul  4 08:13:59.525: CEF-Drop: Packet from 10.130.2.11 (Gi0/0.211) to 10.24.7.1
    8, No adjacency
    *Jul  4 08:14:01.945: CEF-Drop: Packet from 10.130.2.2 (Gi0/0.211) to 10.24.7.18
    , Neighbor resolution req
    *Jul  4 08:14:05.945: CEF-Drop: Packet from 10.130.2.2 (Gi0/0.211) to 10.24.7.18
    , Neighbor resolution req
    *Jul  4 08:14:18.257: CEF-Drop: Packet from 10.24.0.10 (Vl2) to 10.24.0.255, Nei
    ghbor resolution req
    *Jul  4 08:14:37.369: CEF-Drop: Packet from 10.130.2.11 (Gi0/0.211) to 10.24.7.1
    8, Neighbor resolution req
    *Jul  4 08:14:39.557: CEF-Drop: Packet from 10.130.2.11 (Gi0/0.211) to 10.24.7.1
    8, Neighbor resolution req
    *Jul  4 08:14:43.557: CEF-Drop: Packet from 10.130.2.11 (Gi0/0.211) to 10.24.7.1
    8, Neighbor resolution req
    *Jul  4 08:15:17.509: CEF-Drop: Packet from 10.130.2.11 (Gi0/0.211) to 10.24.7.1
    8, Neighbor resolution req
    *Jul  4 08:15:19.697: CEF-Drop: Packet from 10.130.2.11 (Gi0/0.211) to 10.24.7.1
    8, Neighbor resolution req
    *Jul  4 08:15:23.697: CEF-Drop: Packet from 10.130.2.11 (Gi0/0.211) to 10.24.7.1
    8, Neighbor resolution req
    *Jul  4 08:15:57.430: CEF-Drop: Packet from 10.130.2.11 (Gi0/0.211) to 10.24.7.1
    8, Neighbor resolution req
    *Jul  4 08:15:59.618: CEF-Drop: Packet from 10.130.2.11 (Gi0/0.211) to 10.24.7.1
    8, Neighbor resolution req
    *Jul  4 08:16:02.734: CEF-Drop: Packet from 10.130.2.11 (Gi0/0.211) to 10.24.7.1
    8, Neighbor resolution req
    *Jul  4 08:16:05.446: CEF-Drop: Packet from 10.130.2.2 (Gi0/0.211) to 10.24.7.18
    , Neighbor resolution req
    *Jul  4 08:16:09.446: CEF-Drop: Packet from 10.130.2.2 (Gi0/0.211) to 10.24.7.18
    , Neighbor resolution req
    *Jul  4 08:16:37.246: CEF-Drop: Packet from 10.130.2.11 (Gi0/0.211) to 10.24.7.1
    8, Neighbor resolution req
    *Jul  4 08:16:39.430: CEF-Drop: Packet from 10.130.2.11 (Gi0/0.211) to 10.24.7.1
    8, Neighbor resolution req
    *Jul  4 08:16:43.430: CEF-Drop: Packet from 10.130.2.11 (Gi0/0.211) to 10.24.7.1
    8, Neighbor resolution req
    *Jul  4 08:17:08.650: CEF-Drop: Packet from 10.24.0.10 (Vl2) to 10.24.0.255, No
    adjacency
    *Jul  4 08:17:09.402: CEF-Drop: Packet from 10.24.0.10 (Vl2) to 10.24.0.255, No
    adjacency
    *Jul  4 08:17:10.150: CEF-Drop: Packet from 10.24.0.10 (Vl2) to 10.24.0.255, No
    adjacency
    *Jul  4 08:17:12.650: CEF-Drop: Packet from 10.24.0.10 (Vl2) to 10.24.0.255, Nei
    ghbor resolution req
    *Jul  4 08:17:12.902: CEF-Drop: Packet from 10.24.0.10 (Vl2) to 10.24.0.255, No
    adjacency
    *Jul  4 08:17:13.650: CEF-Drop: Packet from 10.24.0.10 (Vl2) to 10.24.0.255, No
    adjacency
    *Jul  4 08:17:14.402: CEF-Drop: Packet from 10.24.0.10 (Vl2) to 10.24.0.255, Nei
    ghbor resolution req
    *Jul  4 08:17:17.150: CEF-Drop: Packet from 10.24.0.10 (Vl2) to 10.24.0.255, No
    adjacency
    *Jul  4 08:17:17.150: CEF-Drop: Packet from 10.24.0.10 (Vl2) to 10.24.0.255, Nei
    ghbor resolution req
    *Jul  4 08:17:17.162: CEF-Drop: Packet from 10.130.2.11 (Gi0/0.211) to 10.24.7.1
    8, Neighbor resolution req
    *Jul  4 08:17:17.902: CEF-Drop: Packet from 10.24.0.10 (Vl2) to 10.24.0.255, No
    adjacency
    *Jul  4 08:17:18.650: CEF-Drop: Packet from 10.24.0.10 (Vl2) to 10.24.0.255, No
    adjacency
    *Jul  4 08:17:19.350: CEF-Drop: Packet from 10.130.2.11 (Gi0/0.211) to 10.24.7.1
    8, Neighbor resolution req
    *Jul  4 08:17:21.150: CEF-Drop: Packet from 10.24.0.10 (Vl2) to 10.24.0.255, Nei
    ghbor resolution req
    *Jul  4 08:17:23.350: CEF-Drop: Packet from 10.130.2.11 (Gi0/0.211) to 10.24.7.1
    8, Neighbor resolution req
    *Jul  4 08:17:25.402: CEF-Drop: Packet from 10.24.0.10 (Vl2) to 10.24.0.255, Nei
    ghbor resolution req
    *Jul  4 08:17:57.087: CEF-Drop: Packet from 10.130.2.11 (Gi0/0.211) to 10.24.7.1
    8, Neighbor resolution req
    *Jul  4 08:17:59.275: CEF-Drop: Packet from 10.130.2.11 (Gi0/0.211) to 10.24.7.1
    8, Neighbor resolution req
    *Jul  4 08:18:02.751: CEF-Drop: Packet from 10.130.2.11 (Gi0/0.211) to 10.24.7.1
    8, Neighbor resolution req
    *Jul  4 08:18:05.451: CEF-Drop: Packet from 10.130.2.2 (Gi0/0.211) to 10.24.7.18
    , Neighbor resolution req
    *Jul  4 08:18:09.451: CEF-Drop: Packet from 10.130.2.2 (Gi0/0.211) to 10.24.7.18
    , Neighbor resolution req
    *Jul  4 08:18:37.227: CEF-Drop: Packet from 10.130.2.11 (Gi0/0.211) to 10.24.7.1
    8, Neighbor resolution req
    *Jul  4 08:18:39.523: CEF-Drop: Packet from 10.130.2.11 (Gi0/0.211) to 10.24.7.1
    8, Neighbor resolution req
    *Jul  4 08:18:43.523: CEF-Drop: Packet from 10.130.2.11 (Gi0/0.211) to 10.24.7.1
    8, Neighbor resolution req
    *Jul  4 08:19:17.583: CEF-Drop: Packet from 10.130.2.11 (Gi0/0.211) to 10.24.7.1
    8, Neighbor resolution req
    *Jul  4 08:19:19.771: CEF-Drop: Packet from 10.130.2.11 (Gi0/0.211) to 10.24.7.1
    8, Neighbor resolution req
    *Jul  4 08:19:23.771: CEF-Drop: Packet from 10.130.2.11 (Gi0/0.211) to 10.24.7.1
    8, Neighbor resolution req


    Will upload the files onto the Cisco website later this afternoon.

    Subject: RE: lost of access to hypervisor from remote networks
    Replied by: Brett Tiller on 07-07-2011 08:21:05 PM
    Hi Ismail,

    Thanks for the logs.  I've reviewed them along with your notes in the previous postings.  I have a few more questions and a data request.

    1.  This intermittent ping failure using vrf is occurring with IOS release 15.1.4M and I believe also with 15.1.3T correct?  In what IOS release did you not experience this problem?


    2.  You mentioned that adding in the ip route on the router and removing it restores the access to the hypervisor.  This step would add/remove the ip route from the router table.  What change, if any, do you notice in the cef table after this step?  Are there any other changes you notice in the router after this step?


    3.  I'd like to set up my lab and try to reproduce this issue, but unfortunately I'm unable to locate your complete router configuration which you provided earlier.  Please resend it.


    Thanks,

    Brett

    Subject: RE: New Message from Brett Tiller in Service Ready Engine Virtualization -
    Replied by: ISMAIL PATEL on 08-07-2011 04:02:13 AM
    Hi Brett



    Always good to hear from you.



    Answers below, hopefully:



    1)      The IOS levels mentioned are the only two tested. However, our second module  running version 1.2 of the software does not have this issue even though the router is running the same version of IOS. The only difference is the configuration that the SM1/0 is in the global routing tables, as oppose in this case vrf DISTSCHOOLS.



    Also, the problem is solid and not intermittent



    2)      Adding the route to the global routing table puts the following entry into the CEF table for vrf DISTCHOOLS which was not previously present



    10.24.7.18/32        attached             SM1/0

                   

                    Full output below:



    TESTSCHRTR101742#sh ip cef vrf DISTSCHOOLS

    Prefix               Next Hop             Interface

    0.0.0.0/0            172.31.14.1          GigabitEthernet0/0.211

    0.0.0.0/8            drop

    0.0.0.0/32           receive

    10.24.0.0/22         attached             Vlan2

    10.24.0.0/32         receive              Vlan2

    10.24.0.1/32         receive              Vlan2

    10.24.0.10/32        attached             Vlan2

    10.24.0.11/32        attached             Vlan2

    10.24.3.255/32       receive              Vlan2

    10.24.7.1/32         receive              Loopback0

    10.24.7.16/28        attached             SM1/0

    10.24.7.16/32        receive              SM1/0

    10.24.7.17/32        receive              SM1/0

    10.24.7.18/32        attached             SM1/0

    10.24.7.19/32        attached             SM1/0

    10.24.7.20/32        attached             SM1/0

    10.24.7.21/32        attached             SM1/0

    10.24.7.31/32        receive              SM1/0

    127.0.0.0/8          drop

    172.31.14.0/24       attached             GigabitEthernet0/0.211

    172.31.14.0/32       receive              GigabitEthernet0/0.211

    172.31.14.1/32       attached             GigabitEthernet0/0.211

    172.31.14.3/32       attached             GigabitEthernet0/0.211

    172.31.14.5/32       attached             GigabitEthernet0/0.211

    172.31.14.6/32       attached             GigabitEthernet0/0.211

    172.31.14.7/32       attached             GigabitEthernet0/0.211

    172.31.14.10/32      attached             GigabitEthernet0/0.211

    172.31.14.11/32      attached             GigabitEthernet0/0.211

    172.31.14.19/32      attached             GigabitEthernet0/0.211

    172.31.14.23/32      attached             GigabitEthernet0/0.211

    172.31.14.28/32      attached             GigabitEthernet0/0.211

    172.31.14.39/32      receive              GigabitEthernet0/0.211

    172.31.14.49/32      attached             GigabitEthernet0/0.211

    172.31.14.52/32      attached             GigabitEthernet0/0.211

    172.31.14.57/32      attached             GigabitEthernet0/0.211

    172.31.14.58/32      attached             GigabitEthernet0/0.211

    172.31.14.60/32      attached             GigabitEthernet0/0.211

    172.31.14.61/32      attached             GigabitEthernet0/0.211

    172.31.14.62/32      attached             GigabitEthernet0/0.211

    172.31.14.63/32      attached             GigabitEthernet0/0.211

    172.31.14.65/32      attached             GigabitEthernet0/0.211

    172.31.14.255/32     receive              GigabitEthernet0/0.211

    224.0.0.0/4          drop

    224.0.0.0/24         receive

    240.0.0.0/4          drop

    255.255.255.255/32   receive



    3)      Config below:



    Building configuration...



    Current configuration : 8017 bytes

    !

    ! Last configuration change at 10:23:30 BST Mon Jul 4 2011 by nst

    version 15.1

    service timestamps debug datetime msec

    service timestamps log datetime msec

    service password-encryption

    !

    hostname TESTSCHRTR101742

    !

    boot-start-marker

    boot system flash0:c2900-universalk9-mz.SPA.151-4.M.bin

    boot system flash0:c2900-universalk9-mz.SPA.151-3.T.bin

    boot-end-marker

    !

    !

    enable secret 5 **********************************

    !

    aaa new-model

    !

    !

    aaa authentication login default local group tacacs+ enable

    !

    !

    !

    !

    !

    aaa session-id common

    !

    clock timezone GMT 0 0

    clock summer-time BST recurring last Sun Mar 1:00 last Sun Oct 1:00

    !

    no ipv6 cef

    ip source-route

    ip cef

    !

    !

    !

    ip vrf DISTSCHOOLS

    rd 211:211

    route-target export 211:211

    route-target import 211:211

    !

    ip dhcp excluded-address 10.24.0.1 10.24.0.99

    ip dhcp excluded-address 10.151.15.17 10.151.15.20

    !

    ip dhcp pool 10.24.0.100

    network 10.24.0.0 255.255.255.0

    default-router 10.24.0.1

    !

    ip dhcp pool VMWARE_SM1

    network 10.151.15.16 255.255.255.240

    default-router 10.151.15.17

    !

    !

    no ip domain lookup

    ip host sm 2067 1.1.1.1

    !

    multilink bundle-name authenticated

    !

    !

    crypto pki token default removal timeout 0

    !

    crypto pki trustpoint TP-self-signed-89228508

    enrollment selfsigned

    subject-name cn=IOS-Self-Signed-Certificate-89228508

    revocation-check none

    rsakeypair TP-self-signed-89228508

    !

    !

    crypto pki certificate chain TP-self-signed-89228508

    certificate self-signed 01

      30820244 308201AD A0030201 02020101 300D0609 2A864886 F70D0101 04050030

      2F312D30 2B060355 04031324 494F532D 53656C66 2D536967 6E65642D 43657274

      69666963 6174652D 38393232 38353038 301E170D 31303037 31323131 33363434

      5A170D32 30303130 31303030 3030305A 302F312D 302B0603 55040313 24494F53

      2D53656C 662D5369 676E6564 2D436572 74696669 63617465 2D383932 32383530

      3830819F 300D0609 2A864886 F70D0101 01050003 818D0030 81890281 8100DED3

      22ED61B6 AE9103E5 ECF61185 2247B39E 7FF0DAF8 9193C71F 5F162EE7 A9AFEDEE

      6A50AC86 226108D1 3D3ED526 012737B6 E83BD91B 62668089 C7A64F9A 941D2F4A

      9E7B473D 9928A9A6 689AE73E A4C4AC34 D5F5CA1D 54BAC222 F1547E3B CEE25B3A

      770CBECE 34D2BF18 1EE4A00F FBEAAE2E 6ADD3FA3 71EF48B4 06B3A22B C3150203

      010001A3 70306E30 0F060355 1D130101 FF040530 030101FF 301B0603 551D1104

      14301282 10544553 54534348 52545231 30313734 32301F06 03551D23 04183016

      8014A342 775D6D03 84F38104 DAD24AAD EE193DB1 2BCA301D 0603551D 0E041604

      14A34277 5D6D0384 F38104DA D24AADEE 193DB12B CA300D06 092A8648 86F70D01

      01040500 03818100 6EA68FC4 D3313E41 FBAD37C2 7A6B930C F03E9987 5AA7E925

      61391182 8DF45BC4 0854215C A20911D7 EDB0582B 06DA34CD B61D4B83 34F43254

      7CFAB3D8 740C7F5A A961E6AE 89857868 257BEE2F 6EC19F6E 8EFB30A5 16E07ED3

      C4EFE704 3F88FDBC 5CFAB27A E48A5BEF 531031AE 0D7FAF8A 067C27E0 752B53C4

      690AFF2C 55537A94

            quit

    license udi pid CISCO2921/K9 sn FCZ1348713U

    license agent notify http://10.130.2.21:80/clm/servlet/HttpListenServlet dummy dummy 1.0

    hw-module sm 1

    !

    !

    !

    username nst password 7 ***************************

    username cisco privilege 15 password 7 **********

    !

    redundancy

    !

    !

    !

    !

    !

    !

    !

    !

    !

    !

    !

    interface Loopback0

    ip vrf forwarding DISTSCHOOLS

    ip address 10.24.7.1 255.255.255.255

    !

    interface Loopback1

    ip address 1.1.1.1 255.255.255.255

    !

    interface Embedded-Service-Engine0/0

    no ip address

    shutdown

    !

    interface GigabitEthernet0/0

    no ip address

    ip accounting output-packets

    duplex full

    speed 10

    rj45-auto-detect-polarity disable

    !

    interface GigabitEthernet0/0.209

    encapsulation dot1Q 209

    ip address 172.31.12.39 255.255.255.0

    !

    interface GigabitEthernet0/0.211

    encapsulation dot1Q 211

    ip vrf forwarding DISTSCHOOLS

    ip address 172.31.14.39 255.255.255.0

    !

    interface GigabitEthernet0/1

    no ip address

    duplex auto

    speed auto

    !

    interface GigabitEthernet0/1.1

    encapsulation dot1Q 1 native

    no ip redirects

    !

    interface GigabitEthernet0/2

    ip address 10.151.15.33 255.255.255.240

    shutdown

    duplex auto

    speed auto

    !

    interface FastEthernet0/0/0

    switchport access vlan 2

    no ip address

    !

    interface FastEthernet0/0/1

    no ip address

    shutdown

    !

    interface FastEthernet0/0/2

    description laptop_f1

    switchport access vlan 2

    no ip address

    duplex full

    speed 100

    spanning-tree portfast

    !

    interface FastEthernet0/0/3

    switchport access vlan 2

    no ip address

    !

    interface SM1/0

    ip vrf forwarding DISTSCHOOLS

    ip address 10.24.7.17 255.255.255.240

    service-module ip address 10.24.7.18 255.255.255.240

    !Application: VMware ESXi 4.1.0 build-348481 running on SRE-V

    service-module ip default-gateway 10.24.7.17

    hold-queue 60 out

    !

    interface SM1/1

    no ip address

    !

    interface Vlan1

    no ip address

    !

    interface Vlan2

    ip vrf forwarding DISTSCHOOLS

    ip address 10.24.0.1 255.255.252.0

    !

    interface Vlan3

    ip vrf forwarding DISTSCHOOLS

    ip address 10.24.4.1 255.255.255.0

    !

    interface Vlan4

    ip vrf forwarding DISTSCHOOLS

    ip address 10.24.5.1 255.255.255.0

    !

    interface Vlan6

    ip vrf forwarding DISTSCHOOLS

    ip address 10.24.6.1 255.255.255.0

    !

    interface Vlan801

    ip address 10.151.8.1 255.255.255.0

    !

    ip default-gateway 172.23.0.2

    ip forward-protocol nd

    !

    ip http server

    no ip http secure-server

    ip flow-export source Loopback0

    ip flow-export version 5

    ip flow-export destination 172.22.0.51 9996

    !

    ip route 0.0.0.0 0.0.0.0 172.31.12.1

    ip route vrf DISTSCHOOLS 0.0.0.0 0.0.0.0 172.31.14.1

    ip tacacs source-interface Loopback0

    !

    ip access-list extended admin

    permit ip 10.0.5.0 0.127.248.255 172.18.0.0 0.0.255.255

    permit ip 10.0.5.0 0.127.248.255 172.24.1.0 0.0.0.255

    permit ip 10.0.5.0 0.127.248.255 62.232.224.0 0.0.7.255

    permit ip 10.0.5.0 0.127.248.255 10.24.24.0 0.0.3.255

    permit ip 10.0.5.0 0.127.248.255 host 193.164.125.38

    permit ip 10.0.5.0 0.127.248.255 10.34.22.0 0.0.0.255

    permit ip 10.0.5.0 0.127.248.255 172.20.32.0 0.0.15.255 log

    ip access-list extended curriculum

    permit ip 10.0.0.0 0.127.251.255 172.18.0.0 0.0.255.255

    permit ip 10.0.0.0 0.127.251.255 62.232.224.0 0.0.7.255

    permit ip 10.0.0.0 0.127.251.255 host 193.164.125.38

    permit ip 10.0.0.0 0.127.251.255 172.20.32.0 0.0.15.255 log

    ip access-list extended management

    permit ip 10.0.7.0 0.127.248.255 172.22.0.0 0.0.255.255

    permit ip 10.0.7.0 0.127.248.255 10.130.2.0 0.0.0.255

    permit ip 10.0.7.0 0.127.248.255 10.132.232.0 0.0.0.31

    permit ip 10.0.7.0 0.127.248.255 10.136.2.0 0.0.0.255

    permit ip 10.0.7.0 0.127.248.255 host 10.24.0.10

    ip access-list extended media

    permit ip 10.0.4.0 0.127.248.255 172.24.0.0 0.0.0.255

    permit ip 10.0.4.0 0.127.248.255 10.0.4.0 0.127.248.255

    ip access-list extended mediatraffic

    permit udp any 172.18.2.0 0.0.0.255 eq 554

    permit udp any 172.18.2.0 0.0.0.255 eq 1755

    ip access-list extended telnet

    deny   ip 10.0.0.0 0.255.255.255 any

    permit ip any any

    ip access-list extended voice

    permit ip 10.0.6.0 0.127.248.255 10.127.0.0 0.0.31.255

    permit ip 10.0.6.0 0.127.248.255 10.0.6.0 0.127.248.255

    permit ip 10.0.6.0 0.127.248.255 10.129.0.0 0.0.15.255

    permit ip 10.0.6.0 0.127.248.255 10.148.4.0 0.0.251.255

    deny   ip any any

    ip access-list extended webtraffic

    permit tcp any 172.18.2.0 0.0.0.255 eq www

    permit tcp any 172.18.2.0 0.0.0.255 eq 8335

    !

    logging 10.130.2.2

    access-list 99 permit 10.130.2.0 0.0.0.255

    access-list 99 permit 10.136.2.0 0.0.0.255

    access-list 99 permit 10.132.232.0 0.0.0.31

    !

    !

    !

    !

    !

    snmp-server community green RO 99

    snmp-server community purple RW 99

    snmp-server trap-source Loopback0

    tacacs-server host 10.130.2.11

    tacacs-server host 10.136.2.11

    tacacs-server key 7 **************

    !

    !

    !

    control-plane

    !

    !

    !

    line con 0

    password 7 ******************

    line aux 0

    line 2

    no activation-character

    no exec

    transport preferred none

    transport input all

    transport output pad telnet rlogin lapb-ta mop udptn v120 ssh

    stopbits 1

    line 67

    no activation-character

    no exec

    transport preferred none

    transport input all

    transport output pad telnet rlogin lapb-ta mop udptn v120 ssh

    stopbits 1

    line vty 0 4

    password 7 ***************************

    transport input all

    !

    exception data-corruption buffer truncate

    scheduler allocate 20000 1000

    ntp server 10.24.0.10

    end



    many thanks



    Ismail



    From: Cisco Developer Community Forums [mailto:cdicuser@developer.cisco.com]
    Sent: Friday, July 08, 2011 1:21 AM
    To: cdicuser@developer.cisco.com
    Subject: New Message from Brett Tiller in Service Ready Engine Virtualization - SRE-V Beta Tester Questions: RE: lost of access to hypervisor from remote networks



    Brett Tiller has created a new message in the forum "SRE-V Beta Tester Questions":

    --------------------------------------------------------------
    Hi Ismail,

    Thanks for the logs. I've reviewed them along with your notes in the previous postings. I have a few more questions and a data request.

    1. This intermittent ping failure using vrf is occurring with IOS release 15.1.4M and I believe also with 15.1.3T correct? In what IOS release did you not experience this problem?


    2. You mentioned that adding in the ip route on the router and removing it restores the access to the hypervisor. This step would add/remove the ip route from the router table. What change, if any, do you notice in the cef table after this step? Are there any other changes you notice in the router after this step?


    3. I'd like to set up my lab and try to reproduce this issue, but unfortunately I'm unable to locate your complete router configuration which you provided earlier. Please resend it.


    Thanks,

    Brett
    --
    To respond to this post, please click the following link:

    <http://developer.cisco.com/web/srev/forums/-/message_boards/view_message/4185871>

    or simply reply to this email.



    ---------------------------------------------------------------------
    DISCLAIMER: This email and files transmitted are
    confidential and are intended solely for the use of the
    intended recipient.  If you are not the intended
    recipient, or the person responsible for delivering it to
    the intended recipient, you may not copy, disclose,
    distribute or use it in any unauthorised manner.  If you
    have received this email in error please notify us by
    email to postmaster@wolverhampton.gov.uk and then delete
    it and any attachments accompanying it.  Please note that
    Wolverhampton City Council cannot guarantee that this
    message or any attachments are virus free or have not been
    intercepted and amended.
    Any views or opinions expressed within this email are
    those of the author and may not necessarily reflect those
    of Wolverhampton City Council and no contractual
    arrangement is intended to arise from this communication.
    ---------------------------------------------------------------------

    Subject: New Message from Brett Tiller in Service Ready Engine Virtualization - SRE-
    Replied by: SCOTT PITTS on 08-07-2011 09:26:13 AM
    Not sure if this helps but I have seen this intermittent issue getting to the ESXi console on a 2911 running 15.1.3T if a BVI1 interface is created. I could not even ping the hypervisor from the router. It seem to interfere with VLAN1. We did not need the BVI1 interface so we deleted it and everything was then stable.


    >>> Cisco Developer Community Forums <cdicuser@developer.cisco.com> 7/7/2011 7:21 PM >>>
    Brett Tiller has created a new message in the forum "SRE-V Beta Tester Questions":

    --------------------------------------------------------------
    Hi Ismail,

    Thanks for the logs. I've reviewed them along with your notes in the previous postings. I have a few more questions and a data request.

    1. This intermittent ping failure using vrf is occurring with IOS release 15.1.4M and I believe also with 15.1.3T correct? In what IOS release did you not experience this problem?


    2. You mentioned that adding in the ip route on the router and removing it restores the access to the hypervisor. This step would add/remove the ip route from the router table. What change, if any, do you notice in the cef table after this step? Are there any other changes you notice in the router after this step?


    3. I'd like to set up my lab and try to reproduce this issue, but unfortunately I'm unable to locate your complete router configuration which you provided earlier. Please resend it.


    Thanks,

    Brett
    --
    To respond to this post, please click the following link:

    <http://developer.cisco.com/web/srev/forums/-/message_boards/view_message/4185871>

    or simply reply to this email.

    Subject: RE: New Message from Brett Tiller in Service Ready Engine Virtualization -
    Replied by: Brett Tiller on 11-07-2011 09:29:38 PM
    Hi Ismail,

    Thanks for your router configuration.  Using your router configuration as a template I've been having some trouble setting up my lab and need a bit more information from you to better understand your configuration from the switch side as well.  I have a couple questions below.


    1.  Since your workstation is pinging the router by what physical interface on the router are the packets from the workstation flowing through?


    2.  Is your switch configured for vlan tagging?  For example in the switch interface to the router you would typically setup dot1q encapsulation and specify this port is doing trunking.  On the switch interface to the workstation you'd setup the vlan access specifying the vlan number.  Perhaps you could provide a snippet of your switch configuration and let me know which parts are relevant.


    Thanks,

    Brett

    Subject: RE: New Message from Brett Tiller in Service Ready Engine Virtualization -
    Replied by: ISMAIL PATEL on 12-07-2011 04:25:10 AM
    Hi Brett



    The router only has one external connection to a Cisco 3750 switch:



    TESTSCHRTR101742#sh cdp neighbors

    Capability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge

                      S - Switch, H - Host, I - IGMP, r - Repeater, P - Phone,

                      D - Remote, C - CVTA, M - Two-port Mac Relay



    Device ID        Local Intrfce     Holdtme    Capability  Platform  Port ID

    CR-F2-20920      Gig 0/0            158             R S   WS-C3750G Gig 2/0/5





    In terms of configuration, the following may help:



    Cisco 2911 with SRE module connected to Cisco 3750G L3 switch on port GI 2/0/5

    Work station with IP address 172.18.2.43 connected to the same L3 switch on port GI 2/0/4 ( default gateway 172.18.2.33 )

    System image file is "flash:c3750-ipservicesk9-mz.122-55.SE3.bin"



    Traffic comes in on interface vlan 211 onto the router from the workstation



    Configuration snippet from 3750 as below:



    ip vrf DISTSCHOOLS

    rd 103:103

    route-target export 103:103

    route-target import 103:103



    interface GigabitEthernet2/0/4

    no switchport

    ip vrf forwarding DISTSCHOOLS

    ip address 172.18.2.33 255.255.255.240





    interface GigabitEthernet2/0/5

    description LINK TO TEST SCHOOL F2

    switchport trunk encapsulation dot1q

    switchport trunk allowed vlan 1,209,211

    switchport mode trunk

    speed 10

    duplex full



    interface Vlan209

    description CORPDIST

    ip address 172.31.12.1 255.255.255.0

    ip ospf priority 200



    interface Vlan211

    description DISTSCHOOLS

    ip vrf forwarding DISTSCHOOLS

    ip address 172.31.14.1 255.255.255.0     



    kind regards



    Ismail













    ---------------------------------------------------------------------
    DISCLAIMER: This email and files transmitted are
    confidential and are intended solely for the use of the
    intended recipient.  If you are not the intended
    recipient, or the person responsible for delivering it to
    the intended recipient, you may not copy, disclose,
    distribute or use it in any unauthorised manner.  If you
    have received this email in error please notify us by
    email to postmaster@wolverhampton.gov.uk and then delete
    it and any attachments accompanying it.  Please note that
    Wolverhampton City Council cannot guarantee that this
    message or any attachments are virus free or have not been
    intercepted and amended.
    Any views or opinions expressed within this email are
    those of the author and may not necessarily reflect those
    of Wolverhampton City Council and no contractual
    arrangement is intended to arise from this communication.
    ---------------------------------------------------------------------

    Subject: RE: lost of access to hypervisor from remote networks
    Replied by: Raghavendra Gutty Veeranagappa on 12-07-2011 06:15:19 AM
    Hi Ismail,

    please try to configure "switchport mode trunk" for sm1/1 in your router

    Router(config)# interface sm 1/1
    Router(config-if)# switchport mode trunk
    Router(config-if)# exit

    Thanks,
    Raghavendra

    Subject: RE: New Message from Raghavendra Gutty Veeranagappa in Service Ready Engine
    Replied by: ISMAIL PATEL on 12-07-2011 06:31:10 AM
    Hi



    Not supported because of the four port Ethernet module we have installed:



    TESTSCHRTR101742(config)#int sm 1/1

    TESTSCHRTR101742(config-if)#switchport mode trunk

                                 ^

    % Invalid input detected at '^' marker.



    TESTSCHRTR101742(config-if)#





    Thanks



    Ismail

    From: Cisco Developer Community Forums [mailto:cdicuser@developer.cisco.com]
    Sent: Tuesday, July 12, 2011 11:15 AM
    To: cdicuser@developer.cisco.com
    Subject: New Message from Raghavendra Gutty Veeranagappa in Service Ready Engine Virtualization - SRE-V Beta Tester Questions: RE: lost of access to hypervisor from remote networks



    Raghavendra Gutty Veeranagappa has created a new message in the forum "SRE-V Beta Tester Questions":

    --------------------------------------------------------------
    Hi Ismail,

    please try to configure "switchport mode trunk" for sm1/1 in your router

    Router(config)# interface sm 1/1
    Router(config-if)# switchport mode trunk
    Router(config-if)# exit

    Thanks,
    Raghavendra
    --
    To respond to this post, please click the following link:

    <http://developer.cisco.com/web/srev/forums/-/message_boards/view_message/4213240>

    or simply reply to this email.



    ---------------------------------------------------------------------
    DISCLAIMER: This email and files transmitted are
    confidential and are intended solely for the use of the
    intended recipient.  If you are not the intended
    recipient, or the person responsible for delivering it to
    the intended recipient, you may not copy, disclose,
    distribute or use it in any unauthorised manner.  If you
    have received this email in error please notify us by
    email to postmaster@wolverhampton.gov.uk and then delete
    it and any attachments accompanying it.  Please note that
    Wolverhampton City Council cannot guarantee that this
    message or any attachments are virus free or have not been
    intercepted and amended.
    Any views or opinions expressed within this email are
    those of the author and may not necessarily reflect those
    of Wolverhampton City Council and no contractual
    arrangement is intended to arise from this communication.
    ---------------------------------------------------------------------

    Subject: RE: lost of access to hypervisor from remote networks
    Replied by: Brett Tiller on 14-07-2011 02:22:00 PM
    Hi Ismail,

    Sorry I've been out a couple of days.  Thanks for your switch configuration.  While I'm setting up my lab I'd like to revisit what we should have looked at from the start and that Raghavendra pointed out regarding interface SM 1/1.  You pointed out that the 4 port ethernet module you are using prevents use of setting this interface to trunk access.   On the router please type 'show diag | inc FRU' and send us the data. This way we can see what type of ether-switch you're using along with the other hardware present.

    Thanks,

    Brett

    Subject: RE: New Message from Brett Tiller in Service Ready Engine Virtualization -
    Replied by: ISMAIL PATEL on 16-07-2011 07:15:49 PM
    Apologies for the delay



    TESTSCHRTR101742#show diag | inc FRU

            Product (FRU) Number     : CISCO2921/K9

            Product (FRU) Number     : PWR-2921-51-AC

            Product (FRU) Number     : HWIC-4ESW

            Product (FRU) Number     : SM-SRE-900-K9





    ---------------------------------------------------------------------
    DISCLAIMER: This email and files transmitted are
    confidential and are intended solely for the use of the
    intended recipient.  If you are not the intended
    recipient, or the person responsible for delivering it to
    the intended recipient, you may not copy, disclose,
    distribute or use it in any unauthorised manner.  If you
    have received this email in error please notify us by
    email to postmaster@wolverhampton.gov.uk and then delete
    it and any attachments accompanying it.  Please note that
    Wolverhampton City Council cannot guarantee that this
    message or any attachments are virus free or have not been
    intercepted and amended.
    Any views or opinions expressed within this email are
    those of the author and may not necessarily reflect those
    of Wolverhampton City Council and no contractual
    arrangement is intended to arise from this communication.
    ---------------------------------------------------------------------

    Subject: RE: New Message from Brett Tiller in Service Ready Engine Virtualization -
    Replied by: Brett Tiller on 18-07-2011 08:38:07 PM
    Hi Ismail,

    Thanks for the data.  I've got some comments below.

    1. I've been able to reproduce the issue you are experiencing in my lab.  It is quite challenging to reproduce this issue on a consistent basis as the arp table does not always contain the route to the service module while the cef vrf table seems to require it.  I will continue working on this matter in order to determine a possible work around and a root cause.  I believe you mentioned that you did not experience this issue with the SRE-V 1.1.x release correct?  If so, which IOS release were you using as well?

    2.  Regarding the EtherSwitch hardware you are using please be aware that as documented in our configuration guide that the supported Etherswitches are EHWIC type which is why the SM 1/1 interface is not providing the desired CLI.  This information is located in section "Hardware Requirements".  I've provided that compatibility list below.

    Cisco EtherSwitch EHWIC
    EHWIC-D-8ESG-P=, EHWIC-D-8ESG-P, EHWIC-D-8ESG=, EHWIC-D-8ESG, EHWIC-4ESG-P=, EHWIC-4ESG-P, EHWIC-4ESG=, and EHWIC-4ESG

    Cisco EtherSwitch Service Module
    SM-D-ES3G-48-P, SM-D-ES3-48-P, SM-D-ES2-48, SM-ES3G-24-P, SM-ES3-24-P, SM-ES2-24-P, SM-ES2-24, and SM-ES3G-16-P

    Thanks,

    Brett

    Subject: RE: New Message from Brett Tiller in Service Ready Engine Virtualization -
    Replied by: ISMAIL PATEL on 19-07-2011 09:35:49 AM
    Good news about being able to recreate issue.



    With regards to software versions, I only saw the issue with version 1.5 of the SRE software, as the configuration was different i.e. below ( taken for another SRE module )



    IOS version         flash0:c2900-universalk9-mz.SPA.151-4.M.bin



    interface SM1/0

    ip address 10.160.55.129 255.255.255.224

    service-module ip address 10.160.55.130 255.255.255.224

    !Application: SRE-V Running on SMV

    service-module ip default-gateway 10.160.55.129

    !

    interface SM1/1

    ip vrf forwarding DISTSCHOOLS

    ip address 10.44.7.65 255.255.255.192

    service-module ip address 10.44.7.66 255.255.255.192

    service-module ip default-gateway 10.44.7.65



    prior to version 1.5, the management of the SRE module was on the global routing table.



    2) point taken ¿ We are not intending to use the module with the SRE-V module



    Many thanks



    Ismail





    ---------------------------------------------------------------------
    DISCLAIMER: This email and files transmitted are
    confidential and are intended solely for the use of the
    intended recipient.  If you are not the intended
    recipient, or the person responsible for delivering it to
    the intended recipient, you may not copy, disclose,
    distribute or use it in any unauthorised manner.  If you
    have received this email in error please notify us by
    email to postmaster@wolverhampton.gov.uk and then delete
    it and any attachments accompanying it.  Please note that
    Wolverhampton City Council cannot guarantee that this
    message or any attachments are virus free or have not been
    intercepted and amended.
    Any views or opinions expressed within this email are
    those of the author and may not necessarily reflect those
    of Wolverhampton City Council and no contractual
    arrangement is intended to arise from this communication.
    ---------------------------------------------------------------------

    Subject: RE: lost of access to hypervisor from remote networks
    Replied by: Brett Tiller on 19-07-2011 08:50:46 PM
    Hi Ismail,

    I think I may have a workaround that you can try.   As you pointed out in an earlier correspondence their appears to be a connection between the global arp and cef vrf DISTSCHOOLS tables.  The SM 1/0 service-module IP must be present in the cef vrf table to be pingable.  Also as you pointed out we cannot create a vrf route with the SM 1/0 interface as the destination as IOS complains of the next hop address being incorrect. At this time I've found two cases which I'd like to describe below.

    1.  SM 1/0 service module ip appears in the cef vrf table, but not in the global arp table.  When this situation occurs the system appears to function normally with no apparent issues in pinging it.  However, on reload this scenario can change and the arp entry can easily be added via access of the service module.  As a result, this state is not a reliable one.

    2.  SM 1/0 service module ip appears in both the cef vrf table and in the global arp table.  When this ip is removed either manually or via timeout from the arp table, it is automatically removed from the cef table, and the pinging of the service module fails.  I've tried the following
       a.  Remove the cef table via 'no ip cef'.  Removing this table does not fix the issue.
       b.  Restore the arp entry by requesting the SM 1/0 service module status -  'service-module sm1/0 status' via the router.  This adds back the arp entry, temporarily fixing the issue, but the ip is removed again after timeout.
       c.  Set a no timeout in the arp entry in the SM 1/0 interface.  This is a possible workaround since the arp entry will now never be removed which maintains the entry in the cef table preserving the service module connection.

    As a result, I'd like you to try out the workaround I've listed in item 2c, and let me know if this preserves your connection. I've provided detailed steps below.

    1.  If the SM 1/0 service module ip is not present in the global arp table (show arp), you can add it via item 2b above.
    2.  Get into configuration mode, access the SM1/0 interface and enter 'arp timeout 0'.  This step sets a no timeout limit.
    3. Exit the configuration and save it.
    4.  Check the cef vrf table - 'show ip cef vrf DISTSCHOOLS' making sure the SM 1/0 service module ip is present.
    5.  Ping the SM 1/0 service module.

    Here's a snippet of my router configuration with relevant data.  I've highlited the suggested change.

    boot-start-marker                                                              
    boot system flash0:c3900-universalk9-mz.SPA.151-4.M.bin                        
    boot-end-marker

    ip vrf DISTSCHOOLS                                                             
    rd 100:100                                                                    
    route-target export 100:100                                                   
    route-target import 100:100

    interface GigabitEthernet0/2.100                                               
    encapsulation dot1Q 100                                                       
    ip vrf forwarding DISTSCHOOLS                                                 
    ip address 192.168.8.1 255.255.255.0                                          
    !                                                                              
    interface SM1/0                                                                
    ip vrf forwarding DISTSCHOOLS                                                 
    ip address 10.24.7.17 255.255.255.0                                           
    service-module ip address 10.24.7.18 255.255.255.0                            
    !Application: VMware ESXi 4.1.0 build-348481 running on SRE                   
    service-module ip default-gateway 10.24.7.17                                  
    arp timeout 0                                                                 
    hold-queue 60 out

    interface SM1/1                                                                
    description Internal switch interface connected to Service Module             
    switchport mode trunk                                                                                                          
    no ip address  

    ** Here are my arp and cef tables:
    3945Router#sh arp                                                              
    Protocol  Address          Age (min)  Hardware Addr   Type   Interface         
    Internet  10.24.7.18             32   c84c.752f.254d  ARPA   SM1/0             
    Internet  192.168.1.20           32   001b.548a.07f0  ARPA   GigabitEthernet0/0
    Internet  192.168.1.70            -   e8b7.4899.6e80  ARPA   GigabitEthernet0/0
    Internet  192.168.1.72           45   e8b7.4899.6e86  ARPA   Embedded-Service-E0

    3945Router#sh ip cef vrf DISTSCHOOLS                                           
    Prefix               Next Hop             Interface                            
    0.0.0.0/0            no route                                                  
    0.0.0.0/8            drop                                                      
    0.0.0.0/32           receive                                                   
    10.24.7.0/24         attached             SM1/0                                
    10.24.7.0/32         receive              SM1/0                                
    10.24.7.17/32        receive              SM1/0                                
    10.24.7.18/32        attached             SM1/0                                
    10.24.7.255/32       receive              SM1/0                                
    127.0.0.0/8          drop                                                      
    192.168.8.0/24       attached             GigabitEthernet0/2.100               
    192.168.8.0/32       receive              GigabitEthernet0/2.100               
    192.168.8.1/32       receive              GigabitEthernet0/2.100               
    192.168.8.100/32     attached             GigabitEthernet0/2.100               
    192.168.8.255/32     receive              GigabitEthernet0/2.100               
    224.0.0.0/4          drop                                                      
    224.0.0.0/24         receive                                                   
    240.0.0.0/4          drop                                                      
    255.255.255.255/32   receive

    ** Here's the ping of the service module from my workstation:
    [brett@atglab2-lnx ~]$ ping 10.24.7.18
    PING 10.24.7.18 (10.24.7.18) 56(84) bytes of data.
    64 bytes from 10.24.7.18: icmp_seq=0 ttl=63 time=2.23 ms
    64 bytes from 10.24.7.18: icmp_seq=1 ttl=63 time=0.230 ms
    64 bytes from 10.24.7.18: icmp_seq=2 ttl=63 time=0.230 ms

    Thanks,
    Brett

    Subject: RE: New Message from Brett Tiller in Service Ready Engine Virtualization -
    Replied by: ISMAIL PATEL on 26-07-2011 09:38:42 AM
    Hi Brett



    More apologies ¿ Holiday season!



    Reconfigured the router as suggested and will monitor.



    Not sure why the SM 1/0 interface needs to be in the global arp table. Surely, vrf-lite is used to segregate user traffic.  Other standard router interfaces do not suffer this issue i.e. none of the arp entries for the following show up in the global arp table.



    TESTSCHRTR101742#show ip vrf

      Name                             Default RD          Interfaces

      DISTSCHOOLS                      211:211             Lo0

                                                           Gi0/0.211

                                                           SM1/0

                                                           Vl2

                                                           Vl3

                                                           Vl4

                                                           Vl6



    Regards



    Ismail









    ---------------------------------------------------------------------
    DISCLAIMER: This email and files transmitted are
    confidential and are intended solely for the use of the
    intended recipient.  If you are not the intended
    recipient, or the person responsible for delivering it to
    the intended recipient, you may not copy, disclose,
    distribute or use it in any unauthorised manner.  If you
    have received this email in error please notify us by
    email to postmaster@wolverhampton.gov.uk and then delete
    it and any attachments accompanying it.  Please note that
    Wolverhampton City Council cannot guarantee that this
    message or any attachments are virus free or have not been
    intercepted and amended.
    Any views or opinions expressed within this email are
    those of the author and may not necessarily reflect those
    of Wolverhampton City Council and no contractual
    arrangement is intended to arise from this communication.
    ---------------------------------------------------------------------

    Subject: RE: lost of access to hypervisor from remote networks
    Replied by: Brett Tiller on 26-07-2011 01:46:03 PM
    Hi Ismail,

    I agree with you and have filed a bug for our IOS team to review.  The workaround I've provided should be stable as I tested it for a few days and saw no issues with the SM IP vanishing from the cef table.  Hopefully your test results will be the same.

    Thanks,

    Brett

    Subject: RE: New Message from Brett Tiller in Service Ready Engine Virtualization -
    Replied by: ISMAIL PATEL on 02-08-2011 11:50:12 AM
    For info ¿ No further loss of service since the arp timeout command was added to the config



    Many thanks



    Ismail



    ---------------------------------------------------------------------
    DISCLAIMER: This email and files transmitted are
    confidential and are intended solely for the use of the
    intended recipient.  If you are not the intended
    recipient, or the person responsible for delivering it to
    the intended recipient, you may not copy, disclose,
    distribute or use it in any unauthorised manner.  If you
    have received this email in error please notify us by
    email to postmaster@wolverhampton.gov.uk and then delete
    it and any attachments accompanying it.  Please note that
    Wolverhampton City Council cannot guarantee that this
    message or any attachments are virus free or have not been
    intercepted and amended.
    Any views or opinions expressed within this email are
    those of the author and may not necessarily reflect those
    of Wolverhampton City Council and no contractual
    arrangement is intended to arise from this communication.
    ---------------------------------------------------------------------