connectivity

Version 1
    This document was generated from CDN thread

    Created by: Raymond Armstrong on 15-05-2012 08:49:39 AM
    1. Have been trying to set up server via PCIe but cannot get the internal link to come up following procedure
    interface ucse2/0
     ip address 69.184.252.1 255.255.255.240
     imc ip address 192.168.1.91 255.255.255.0 default-gateway 192.168.1.60
     imc access-port dedicated
    RedHat rel 6 on server
    Server eth0 set as follows:
    DEVICE="eth0"
    HWADDR="F0:F7:55:12:01:BE"
    ONBOOT="yes"
    BOOTPROTO="static"
    IPADDR="69.184.252.3"
    #ETHTOOL_OPTS="speed 1000 duplex full space autoneg on" <tried with this line in and out of the config>
    NETMASK="255.255.255.240"
    We are seeing no link between the router and the server on the PCIe connection - is there anything we are missing?

    2. Also tried using MGF Vlan following procedure
    interface ucse2/1
     description Internal switch interface connected to Service Module
     switchport trunk allowed vlan 1,2,100,1002-1005
     switchport mode trunk
     no ip address
    !
    interface Vlan1
     no ip address
    !
    interface Vlan100
     ip address 69.184.252.17 255.255.255.240
    !
    DEVICE="eth0"
    HWADDR="F0:F7:55:12:01:BE"
    ONBOOT="yes"
    BOOTPROTO="static"
    IPADDR="69.184.252.19"
    #ETHTOOL_OPTS="speed 1000 duplex full space autoneg on" <tried with this line in and out of the config>
    NETMASK="255.255.255.240"
    r107476#show ip int br
    Interface                  IP-Address      OK? Method Status                Protocol
    Embedded-Service-Engine0/0 unassigned      YES NVRAM  up                    up     
    GigabitEthernet0/0         172.16.1.2      YES NVRAM  down                  down   
    GigabitEthernet0/1         69.184.252.33   YES NVRAM  up                    up     
    GigabitEthernet0/2         135.0.13.34     YES NVRAM  up                    up     
    ucse2/0                    69.184.252.1    YES NVRAM  administratively down down   
    ucse2/1                    unassigned      YES unset  up                    up     
    Loopback0                  135.2.5.181     YES NVRAM  up                    up     
    NVI0                       135.2.5.181     YES unset  up                    up     
    Vlan1                      unassigned      YES unset  up                    up     
    Vlan100                    69.184.252.17   YES manual down                  down 

    Unsure how the vlan can come up without an interface placed in vlan 100? again we are seeing no link

    3. We tried configuring the server with a direct cable between GE2 and the router GE0/1 and this would ping intermittently and neither the router or the server could see the arp of the other device.
    r107476#ping 69.184.252.35
    Type escape sequence to abort.
    Sending 5, 100-byte ICMP Echos to 69.184.252.35, timeout is 2 seconds:
    *May 15 11:35:34.812: IP ARP: sent req src 69.184.252.33 70ca.9ba3.0c81,
                     dst 69.184.252.35 0000.0000.0000 GigabitEthernet0/1.
    *May 15 11:35:36.812: IP ARP: sent req src 69.184.252.33 70ca.9ba3.0c81,
                     dst 69.184.252.35 0000.0000.0000 GigabitEthernet0/1.
    *May 15 11:35:38.812: IP ARP: sent req src 69.184.252.33 70ca.9ba3.0c81,
                     dst 69.184.252.35 0000.0000.0000 GigabitEthernet0/1.
    *May 15 11:35:40.812: IP ARP: sent req src 69.184.252.33 70ca.9ba3.0c81,
                     dst 69.184.252.35 0000.0000.0000 GigabitEthernet0/1.
    *May 15 11:35:42.812: IP ARP: sent req src 69.184.252.33 70ca.9ba3.0c81,
                     dst 69.184.252.35 0000.0000.0000 GigabitEthernet0/1.
    Success rate is 0 percent (0/5)
    r107476#
    We then swopped to GE3 > GE0/1 on the router and we could ping and ssh before everything locked up.
    r107476#ping 69.184.252.35
    Type escape sequence to abort.
    Sending 5, 100-byte ICMP Echos to 69.184.252.35, timeout is 2 seconds:
    !!!!!
    Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms
    r107476#ping 69.184.252.35
    Type escape sequence to abort.
    Sending 5, 100-byte ICMP Echos to 69.184.252.35, timeout is 2 seconds:
    !!!!!
    Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms
    r107476#ssh -l root 69.184.252.35
    Password:
    Last login: Thu Mar 24 12:23:46 2005 from 69.184.252.33
    [root@cisco3925 ~]# ls
    anaconda-ks.cfg  Documents  install.log         Music     Public     Videos
    Desktop          Downloads  install.log.syslog  Pictures  Templates
    [root@cisco3925 ~]# ls /
    bin   cgroup  etc   lib    lost+found  misc  net  proc  sbin     srv  tmp  var
    boot  dev     home  lib64  media       mnt   opt  root  selinux  sys  usr
    [root@cisco3925 ~]#
    ***************locked up*************
    We also noted the following:
    r107476#
    *May 15 12:07:53.303: old: SERVICE_MODULE_STATE_ERRQ new: SERVICE_MODULE_STATE_SHDN
    r107476#

    r107476#
    *May 15 12:13:58.875: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/1, changed state to down
    *May 15 12:13:59.875: %LINK-3-UPDOWN: Interface GigabitEthernet0/1, changed state to down
    r107476#
    *May 15 12:14:01.615: IP ARP rep filtered src 69.184.252.35 f0f7.5512.01c1, dst 69.184.252.35 f0f7.5512.01c1 wrong cable, interface GigabitEthernet0/1
    *May 15 12:14:02.875: %LINK-3-UPDOWN: Interface GigabitEthernet0/1, changed state to up
    r107476#
    *May 15 12:14:02.875: IP ARP: sent rep src 69.184.252.33 70ca.9ba3.0c81,
                     dst 69.184.252.33 ffff.ffff.ffff GigabitEthernet0/1
    *May 15 12:14:03.631: IP ARP req filtered src 69.184.252.35 f0f7.5512.01c1, dst 69.184.252.35 ffff.ffff.ffff wrong cable, interface GigabitEthernet0/1
    *May 15 12:14:03.875: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/1, changed state to up
    r107476#
    r107476#ping 69.184.252.35
    Type escape sequence to abort.
    Sending 5, 100-byte ICMP Echos to 69.184.252.35, timeout is 2 seconds:
    .!!!!
    Success rate is 80 percent (4/5), round-trip min/avg/max = 1/1/1 ms
    r107476#
    r107476#ssh -l root 69.184.252.35
    Password:
    Last login: Thu Mar 24 12:33:40 2005 from 69.184.252.33
    [root@cisco3925 ~]# more /etc/sysc
    r107476(config-if)#switchport
    *May 15 12:17:53.303: old: SERVICE_MODULE_STATE_SHDN new: SERVICE_MODULE_STATE_FAIL
     
    we are seeing NO BUFFER SPACE AVAILABLE on the server

    we are currently looking at the drivers and configurations for point 3.

    Subject: RE: connectivity
    Replied by: Raymond Armstrong on 15-05-2012 11:44:41 AM
    We belive the issue with 3. is due to the broadcom driver on the external gig ports - we have tried updating the kernal and the drivers but still cannot get this working.

    Subject: RE: connectivity
    Replied by: Brett Tiller on 15-05-2012 02:18:22 PM
    Hi Raymond,

    Regarding your configuration in the UCSE 2/0 interface , "access-port dedicated" means that you have an ethernet cable connection to the 'M' port on the ucse module which would then connect to your switch.  This connection separates packets going to the host bypassing the router.   If you desire to use the PCIE connection then you would remove that Ethernet connection and connect to the GE2 port on the front panel and connect the other end of that cable to the switch.  The access-port command should then be changed to "access-port host GE2".  After you've changed this configuration, please power down the router completely and reload it.   This reboot may clear up the status changes that you are experiencing.

    After you've completed the steps above, if you are unable to connect to the RedHat OS, please add an ip route in the router specifying the ip of this server linking it to the module interface.  I've provided an example below.

    ip route 69.184.252.3  255.255.255.255 ucse 2/0

    If you continue to see problems connecting to Red Hat please check that LInux SE is disabled, and that there is no Red Hat firewall blocking the connection.  Failing that, please attach your router configuration and let us know the Ethernet connections as well.

    Thanks,

    Brett

    Subject: RE: connectivity
    Replied by: Raymond Armstrong on 16-05-2012 04:54:56 AM
    Hi Brett - current status

    Test 1 PCIe
    what I am actually trying to do for is use the PCIe layer3 connection have removed

    imc ip address 192.168.1.91 255.255.255.0 default-gateway 192.168.1.60
    imc access-port dedicated

    So if we strip the above out are we saying that the PCIe connection for GE0 should act like a normal "GE" connection?


    Test 2 MGF vlan
    further investigation required

    Test 3 using external cabling GE2/3
    We may have got this working and can now connect using the latest kernel though we need to understand whether the kernel is using the same broadcom drivers as the version on rel 6.0 or whether it is now using the bnx2 broadcom drivers - further testing and analysis required.

    Subject: RE: connectivity
    Replied by: Raymond Armstrong on 16-05-2012 05:56:38 AM
    Test 1 PCIe
    with the following removed
    imc ip address 192.168.1.91 255.255.255.0 default-gateway 192.168.1.60
    imc access-port dedicated
    it now appears to work

    Test 2 MGF vlan
    changed
    switchport mode to access / access vlan 100 and this now works

    Test 3 using external cabling GE2/3
    further testing and analysis required.

    Subject: RE: connectivity
    Replied by: Manuka Jayakody on 16-05-2012 01:34:29 PM
    Hi Raymond,

    Redhat's eth0 (PCIe) connectivity should work regardless of the imc ip address and imc access-port commands. you may need to add a route to the eth0 ip address as Brett pointed out.

    I've been using Redhat's eth1 (MGF) connectivity with vlan1. for eg:

    interface Vlan1
    ip address 100.1.1.1 255.255.255.0
    ---
    ip route 100.1.1.2 255.255.255.255 Vlan1

    --
    100.1.1.2 is the eth1 address and using eth1 default-gateway as 100.1.1.1


    Redhat has just posted the latest signed driver that works with ucse, per our request. If you still see driver issues you can download it from here and install it

    https://rhn.redhat.com/network/software/packages/name_overview.pxt?package_name=kmod-tg3&archIdList=520&archLabelList=x86_64&search_subscribed_channels=