Lost access to the ESXi

Version 1
    This document was generated from CDN thread

    Created by: SCOTT PITTS on 20-11-2010 01:33:55 AM
    The router had a power failure and now ESXi is not working.
     
    This shows that it should be running:
     
    Router#service-module sm 1/0 statu
    Service Module is Cisco SM1/0
    Service Module supports session via TTY line 67
    Service Module is in Steady state
    Service Module heartbeat-reset is enabled
    Getting status from the Service Module, please wait..
    Cisco SRE-V Software 1.0.0.19
    SRE-V Running on SMV
    No install/uninstall in progress
    No local partition
    Router#
     
    This looks bad:
    se-192-168-75-51# show hypervisor ip
    Hypervisor information cannot be displayed. Error: (HostSystem not found)
     
    I have reset the SRE. I have power-cycled the router. I am not sure where to look now.
     
     

    Subject: RE: New Message from SCOTT PITTS in Service Ready Engine Virtualization - S
    Replied by: Michael Kinzlmaier on 20-11-2010 01:33:55 AM
    Hi Scott,

    Can you check that the mgf ip address and default gateway configuration in the IOS config ?

    There is an open issue where the mgf default gateway and the mgf IP address disappear from the IOS config.

    Mike

    Subject: RE: Lost access to the ESXi
    Replied by: Zhenwei Hu on 20-11-2010 01:33:55 AM
    Hi Scott,
     
    From the module, can you execute the following CLI commands and let us know the output?
    1) show hypervisor vswitch
    2) show hypervisor vmknic
     
    In addition, it would be good if you can retrieve some logs from the console manager by executing the following CLI command on the module:
    copy log messages.log url ftp://<FTP_URL>
     
    Thanks,
    Allan.
     


    The router had a power failure and now ESXi is not working.
     
    This shows that it should be running:
     
    Router#service-module sm 1/0 statu
    Service Module is Cisco SM1/0
    Service Module supports session via TTY line 67
    Service Module is in Steady state
    Service Module heartbeat-reset is enabled
    Getting status from the Service Module, please wait..
    Cisco SRE-V Software 1.0.0.19
    SRE-V Running on SMV
    No install/uninstall in progress
    No local partition
    Router#
     
    This looks bad:
    se-192-168-75-51# show hypervisor ip
    Hypervisor information cannot be displayed. Error: (HostSystem not found)
     
    I have reset the SRE. I have power-cycled the router. I am not sure where to look now.
     
     
     
     

    Subject: RE: Lost access to the ESXi
    Replied by: SCOTT PITTS on 20-11-2010 01:33:55 AM
    Here is the config which is a little different due to the use of a HWIC-D-9ESW-PoE
     
    interface SM1/0
     ip address 192.168.75.50 255.255.255.192
     ip nat inside
     ip virtual-reassembly in
     service-module ip address 192.168.75.51 255.255.255.192
     !Application: SRE-V Running on SMV
     service-module ip default-gateway 192.168.75.50
    !
    interface SM1/1
     ip address 192.168.75.150 255.255.255.192
     service-module ip address 192.168.75.151 255.255.255.192
     
    Here is the show hypervisor vswitch
     
    se-192-168-75-51# show hypervisor vswitch
    Switch Name      Num Ports   Used Ports  Configured Ports  MTU     Uplinks  
    vSwitch0         128         3           128               1500    vmnic2   
      PortGroup Name        VLAN ID  Used Ports  Uplinks  
      VM Network            0        0           vmnic2   
      Management Network    0        1           vmnic2   
    Switch Name      Num Ports   Used Ports  Configured Ports  MTU     Uplinks  
    hgSwitch         8           3           8                 1500             
      PortGroup Name        VLAN ID  Used Ports  Uplinks  
      hgNet                 0        2                    
    Switch Name      Num Ports   Used Ports  Configured Ports  MTU     Uplinks  
    ciscoSwitch      8           3           8                 1500    vmnic1   
      PortGroup Name        VLAN ID  Used Ports  Uplinks  
      CiscoReserved         0        1           vmnic1 
     
    Here is the show hypervisor vmknic
     se-192-168-75-51# show hypervisor vmknic
    Intf. Portgroup/DVPort    IP Address      Netmask         MAC      
    ---------------------------------------------------------------------------
    vmk0  Management Network  192.168.75.151  255.255.255.192 c8:4c:75:23:ea:1c
    vmk1  hgNet               169.254.1.1     255.255.255.0   00:50:56:72:47:c4
    2 total VMkernel nic(s)
    se-192-168-75-51#
     
    I am trying to get the logs off now.

    Subject: RE: Lost access to the ESXi
    Replied by: SCOTT PITTS on 20-11-2010 01:33:55 AM
    I am have problems getting the logs off the server. It would be nice if you included SZ and RZ (Send/Recieve Zmodem) as part of the Linux shell. This would make is super easy to send and recieve files with very little overhead.
     
     
    Here is the config which is a little different due to the use of a HWIC-D-9ESW-PoE
     
    interface SM1/0
     ip address 192.168.75.50 255.255.255.192
     ip nat inside
     ip virtual-reassembly in
     service-module ip address 192.168.75.51 255.255.255.192
     !Application: SRE-V Running on SMV
     service-module ip default-gateway 192.168.75.50
    !
    interface SM1/1
     ip address 192.168.75.150 255.255.255.192
     service-module ip address 192.168.75.151 255.255.255.192
     
    Here is the show hypervisor vswitch
     
    se-192-168-75-51# show hypervisor vswitch
    Switch Name      Num Ports   Used Ports  Configured Ports  MTU     Uplinks  
    vSwitch0         128         3           128               1500    vmnic2   
      PortGroup Name        VLAN ID  Used Ports  Uplinks  
      VM Network            0        0           vmnic2   
      Management Network    0        1           vmnic2   
    Switch Name      Num Ports   Used Ports  Configured Ports  MTU     Uplinks  
    hgSwitch         8           3           8                 1500             
      PortGroup Name        VLAN ID  Used Ports  Uplinks  
      hgNet                 0        2                    
    Switch Name      Num Ports   Used Ports  Configured Ports  MTU     Uplinks  
    ciscoSwitch      8           3           8                 1500    vmnic1   
      PortGroup Name        VLAN ID  Used Ports  Uplinks  
      CiscoReserved         0        1           vmnic1 
     
    Here is the show hypervisor vmknic
     se-192-168-75-51# show hypervisor vmknic
    Intf. Portgroup/DVPort    IP Address      Netmask         MAC      
    ---------------------------------------------------------------------------
    vmk0  Management Network  192.168.75.151  255.255.255.192 c8:4c:75:23:ea:1c
    vmk1  hgNet               169.254.1.1     255.255.255.0   00:50:56:72:47:c4
    2 total VMkernel nic(s)
    se-192-168-75-51#
     
    I am trying to get the logs off now.


    Subject: RE: Lost access to the ESXi
    Replied by: Zhenwei Hu on 20-11-2010 01:33:55 AM
    Thanks for the update, Scott.
     
    What's the problem you encountered when copying out the logs? Is it due to the access issue that the SRE-V module has so that it cannot reach the ftp server?
     
    What's the current state of the SRE-V? Can you verify the followings as well?
    1) Are you able to connect to the ESXi via vSphere Client? Basically are able to ping/access 192.168.75.151 (the mgf ip address) from the router and outside of the router?
    2) Are you able to ping/access 192.168.75.51 (the console manager ip address) from the router and outside of the router?
    3) What's the ip route table in IOS? Can you do "show ip route" from IOS?
    4) Do you see any error when executing "show virtual-machine" CLI command on console manager?
    5) Before the power failure, did you do any changes to any of the default vSwitches?
     
    Thanks,
    Allan.


    I am have problems getting the logs off the server. It would be nice if you included SZ and RZ (Send/Recieve Zmodem) as part of the Linux shell. This would make is super easy to send and recieve files with very little overhead.


     

    Subject: RE: Lost access to the ESXi
    Replied by: SCOTT PITTS on 20-11-2010 01:33:56 AM
    1) I can not connect with vSphere, or ping the the MGF ip address from the network. I can not ping the 192.168.75.151 from the SRE or the router. The hypervisor is apparently not running.
     
    2) I can ping the 192.168.75.51 from inside the router and outside the router.
     
    3) Router#show ip route
    Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
           D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
           N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
           E1 - OSPF external type 1, E2 - OSPF external type 2
           i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
           ia - IS-IS inter area, * - candidate default, U - per-user static route
           o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
           + - replicated route, % - next hop override
    Gateway of last resort is 10.11.16.1 to network 0.0.0.0
    S*    0.0.0.0/0 [1/0] via 10.11.16.1
          10.0.0.0/8 is variably subnetted, 2 subnets, 2 masks
    C        10.11.16.0/20 is directly connected, GigabitEthernet0/0
    L        10.11.16.99/32 is directly connected, GigabitEthernet0/0
          192.168.75.0/24 is variably subnetted, 2 subnets, 2 masks
    C        192.168.75.0/26 is directly connected, SM1/0
    L        192.168.75.50/32 is directly connected, SM1/0
     
    4) se-192-168-75-51# show virtual-machine
    Name                             Status
    ------------------------------------------
    0 total virtual machine(s)
    se-192-168-75-51#
    5) No changes were made to the vswitch. It was just the base load ESXi and I loaded an appliance from VMWare. I then started working on other thing. Since is was on my desk I did not have it on UPS and the power was lost. I did load one of our ESXi licenses into VMWARE but it worked fine afterwards. I am relatively sure I rebooted the SRE after adding our ESXi license.
     
    Thanks for the update, Scott.
     
    What's the problem you encountered when copying out the logs? Is it due to the access issue that the SRE-V module has so that it cannot reach the ftp server?
     
    What's the current state of the SRE-V? Can you verify the followings as well?
    1) Are you able to connect to the ESXi via vSphere Client? Basically are able to ping/access 192.168.75.151 (the mgf ip address) from the router and outside of the router?
    2) Are you able to ping/access 192.168.75.51 (the console manager ip address) from the router and outside of the router?
    3) What's the ip route table in IOS? Can you do "show ip route" from IOS?
    4) Do you see any error when executing "show virtual-machine" CLI command on console manager?
    5) Before the power failure, did you do any changes to any of the default vSwitches?
     
    Thanks,
    Allan.


    I am have problems getting the logs off the server. It would be nice if you included SZ and RZ (Send/Recieve Zmodem) as part of the Linux shell. This would make is super easy to send and recieve files with very little overhead.


     


    Subject: RE: New Message from SCOTT PITTS in Service Ready Engine Virtualization - S
    Replied by: Radhika Miriyala on 20-11-2010 01:33:56 AM
    Scott,

    Thanks for getting back with all the information.

    1)      Can   you also give the output of ¿show hypervisor nics¿ cli from the console manager.

    2)      Also after you get the output of the cli above can you Power off the router and then reseat the SRE module in the router?

    Thanks for your patience and help as we debug this issue.

    Radhika

    Subject: RE: New Message from Radhika Miriyala in Service Ready Engine Virtualizatio
    Replied by: Radhika Miriyala on 20-11-2010 01:33:56 AM
    Scott,

    Can you do one more step before you try to reseat the module.

    1)      On the router from the config mode please do the following

    config t

    Enter configuration commands, one per line.  End with CNTL/Z.

    router (config)#interface sm 1/1

    router (config-if)#shut

    router (config-if)#exit

    router#config t

    router (config)#interface sm 1/1

    router (config-if)# no shut

    router (config-if)#exit



    Then from the console manager try the ¿show hypervisor nics¿ output.



    Thanks,

    Radhika

    Subject: RE: New Message from Radhika Miriyala in Service Ready Engine Virtualizatio
    Replied by: SCOTT PITTS on 20-11-2010 01:33:56 AM
    Here is the output

    se-192-168-75-51# show hypervisor nics
    Name    PCI           Driver      Link Speed     Duplex MAC Address       MTU
    vmnic0  0000:01:00.00 e1000e      Down 0Mbps     Half   c8:4c:75:23:ea:1c 1500
    vmnic1  0000:02:00.00 bnx2        Up   1000Mbps  Full   88:43:e1:bc:a4:01 1500
    vmnic2  0000:02:00.01 bnx2        Up   1000Mbps  Full   88:43:e1:bc:a4:00 1500
    3 total nics
    se-192-168-75-51#
     
    But still have this
     
    se-192-168-75-51# show hypervisor ip
    Hypervisor information cannot be displayed. Error: (HostSystem not found)
    se-192-168-75-51#

    Subject: RE: New Message from SCOTT PITTS in Service Ready Engine Virtualization - S
    Replied by: Radhika Miriyala on 20-11-2010 01:33:56 AM
    Scott,

    Is the output of ¿show hypervisor nics¿ before you did a shut and no shut on the IOS or after you did a shut. If it is after you did a shut by any chance do you have the output of the ¿show hypervisor nics¿ before you did the shut no shut?

    Thanks,

    Radhika

    Subject: RE: New Message from Radhika Miriyala in Service Ready Engine Virtualizatio
    Replied by: Michael Kinzlmaier on 20-11-2010 01:33:56 AM
    Hi Scott,

    Just out of curiosity could you execute the 'show license in-use' command from the SRE module.

    We have an issue where evaluation licenses are expiring prematurely and I just want to verify you are not running into this problem.

    Mike

    Subject: RE: New Message from Radhika Miriyala in Service Ready Engine Virtualizatio
    Replied by: SCOTT PITTS on 20-11-2010 01:33:57 AM
    se-192-168-75-51# show license in-use
    StoreIndex:  8  Feature: SRE-V-HOST-LIC                    Version: 1.0
            License Type: Evaluation
            License State: Active, In Use
                Evaluation total period:  8 weeks  4 days
                Evaluation period left:  6 weeks  5 days
                Expiry date: Wed Feb 20 12:49:20 2008
            License Count: Non-Counted
            License Priority: Low
    se-192-168-75-51#

    Subject: RE: New Message from Radhika Miriyala in Service Ready Engine Virtualizatio
    Replied by: SCOTT PITTS on 20-11-2010 01:33:57 AM
    Is it time un unload and reload the ESXi application to the SRE? I would be gald to setup a webex if someone wanted to troubleshoot the issue. We are on central time 8am-5pm so tomorrow I can be available.

    Subject: RE: New Message from SCOTT PITTS in Service Ready Engine Virtualization - S
    Replied by: Radhika Miriyala on 20-11-2010 01:33:57 AM
    Scott,

    I think having a WebEx session would help us to debug the issue. Would 10 am to 11:00 am PST ( 12:00 to 1:00 Pm CST) work for you?

    If that does not give us a few time slots that would work for you and we can go from there.

    Also can you please include your email address so we can send the meeting invite to you.

    Thanks,

    Radhika

    Subject: RE: New Message from Radhika Miriyala in Service Ready Engine Virtualizatio
    Replied by: Radhika Miriyala on 20-11-2010 01:33:58 AM
    Scott,

    Can you let us know if 10 Am pST or 12:00 CST works for you or not?

    Thanks,

    Radhika

    Subject: RE: New Message from Radhika Miriyala in Service Ready Engine Virtualizatio
    Replied by: SCOTT PITTS on 20-11-2010 01:33:58 AM
    Sorry just looked at the beta forums today (1:17pm CST). Now is fine with me. You can contact me at 901-252-5462, Scott.


    Scott,

    Can you let us know if 10 Am pST or 12:00 CST works for you or not?

    Thanks,

    Radhika

    Subject: RE: New Message from SCOTT PITTS in Service Ready Engine Virtualization - S
    Replied by: Radhika Miriyala on 20-11-2010 01:33:58 AM
    Scott,

    We are in a meeting right now. Can I give you a call around 2:00 CST?

    Thanks,

    Radhika

    Subject: RE: New Message from SCOTT PITTS in Service Ready Engine Virtualization - S
    Replied by: SCOTT PITTS on 20-11-2010 01:33:59 AM
    Thanks for the call, I have removed and reloaded 1.0.0.19 beta and got everything back running and then upgraded to the 1.0.0.35 beta to make sure the upgrade process works. It seems to have upgraded just fine without issue. I will not continue testing and post any results.