Difficulties injecting broadcast packets

Document created by cdnadmin on Jan 25, 2014
Version 1Show Document
  • View in full screen mode
This document was generated from CDN thread

Created by: Viktor S. Wold Eide on 22-02-2013 10:44:42 AM
Hi,
I'm having some difficulties injecting broadcast packets from a onePK application. I can see that each packet is correctly delivered from the application via dpss_mp to the router inside a GRE packet. However, the packets seem to "disappear" inside the router somewhere. Unicast packets are delivered just fine.

I thought it could have something to do with privileges on the router, as for example IP broadcast ping is disallowed from exec user level. I configured privilege 15 for the onePK user on the router. Then I can login as this user and from the command line interface perform broadcast ping. However, it did not make any differences to the onePK application, as the broadcast packets still disappear.

debug onep is enabled on the router and it provides some debugging information. However, I can can not see any information regarding the disappearing broadcast packets.

I would like to narrow the problem down somewhat. Currently, I'm not sure where the packets disappear. Could it be that onep inside the router silently discards/drops packets? Are there any relevant checks in this respect? If the packet is delivered from the onep part inside the router to the router itself, what would be the easiest way to determine where and why the packet is not sent out on the interface?

Any suggestions are welcome.

Subject: RE: Difficulties injecting broadcast packets
Replied by: Zach Seils on 22-02-2013 12:46:44 PM
Hi Viktor,
Can you please check the output of the command show cef dpss?  In particular, I'm interested if any of the error counters are increasing.  If they are, try enabling debug cef dpss error on the router and see what messages appear.  If not, please enable the following debugs on the router:
debug cef dpss
debug cef dpss error
debug cef dpss internal
debug cef dpss packet
Also collect the output of the command show ip cef switching statistics before and after packet injection.
Thanks, Zach  

Subject: RE: Difficulties injecting broadcast packets
Replied by: Joseph Clarke on 23-02-2013 11:17:38 AM
It would be helpful to see your code as well if possible (or at least a sample frame).  Note that the DPSS only supports generation of IPv4 and IPv6 frames.  I'm not sure what broadcast frames you're sending, but non-IP frames will throw an error when debug cef dpss is enabled.

Subject: Re: New Message from Zach Seils in onePK - Troubleshooting: RE: Difficultie
Replied by: Viktor S. Wold Eide on 24-02-2013 09:24:24 AM
Thanks a lot for the suggestions. I gave it a quick look here today. Below
you find some debug info when injecting packets. Note that two packets are
injected almost at the same time - an ARP requst and a IP broadcast message
to 10.12.1.255. I can not see that any of these packets are sent from the
interface. This might be a (mis)configuration on my side of the router as
dpss at the end seems to report that all 6 packets have been injected? I am
using the router setup in the
"sample apps/CustomEncryption"

The following debug settings were used:
debug cef dpss
debug cef dpss error
debug cef dpss internal

*** Initially, before injecting packets. ***

NE101#show cef dpss
VM Tport State Prt/Eth
Errors SeqNum
1 GRE listening
Local-addr: 10.1.1.101 0x8921
0 0
Remote-addr: UNKNOWN 0x0000
0 4294967295
Sticky errors: no, Tunnel: ?, GRE handle: 0x0
OCE: <NULL>
Punt/Copy: 0/0, Reinject/Inject/Drop: 0/0/0
Refcount: 0, Memory: 0xF1AFABF8

Errors: 0
CFT FOs free/in use: 264/0, errors: 0
Reinject handles free/in use: 500/0, errors: 0
Encode/decode buffer size: 65665 bytes, memory: 0xF2AB0888
NE101#

*** When connecting, injecting first two packets, and running show cef dpss
afterwards ***

NE101#show cef dpss
*Feb 24 15:01:06.754: CEF DPSS: cef_dpss_uri_parse_open - uri: gre://
10.1.1.5:8921/Ethernet0/0
*Feb 24 15:01:06.754: CEF DPSS: cef_dpss_find_vm looking for 3, 35105, 2
*Feb 24 15:01:06.754: CEF DPSS: cef_dpss_find_vm comparing with 3, 35105, 2
(0xF1AFABF8)
*Feb 24 15:01:06.754: CEF DPSS: cef_dpss_find_vm returning 0xF1AFABF8
NE101#show cef dpss
*Feb 24 15:01:06.755: %TUN-4-MTUDEFAULTEXCEEDSL2MTU_
IPV4: Tunnel0 transport
MTU 9192 exceeds Ethernet0/0 configured IPv4 MTU 1500, fragmentation may
occur
*Feb 24 15:01:07.760: %LINEPROTO-5-UPDOWN: Line protocol on Interface
Tunnel0, changed state to up
NE101#show cef dpss
*Feb 24 15:01:15.078: CEF DPSS GRE: CEF rx pak: 0xEF78CA60, size: 146
*Feb 24 15:01:15.078: CEF DPSS: Parsing inject message
*Feb 24 15:01:15.078: CEF DPSS: Injecting packet from INPUT(Ethernet0/1)
*Feb 24 15:01:15.078: CEF DPSS FA: Injected packet. Bypassing DPSS
*Feb 24 15:01:15.078: CEF DPSS: IPV4 punting injected packet to
INPUT(Ethernet0/1)
*Feb 24 15:01:15.078: CEF DPSS GRE: CEF rx pak: 0xEF78CA60, size: 86
*Feb 24 15:01:15.078: CEF DPSS: Parsing inject message
*Feb 24 15:01:15.078: CEF DPSS: Injected IPV4 packet punted to PS
*Feb 24 15:01:15.078: CEF DPSS FA: Injected packet. Bypassing DPSS
NE101#show cef dpss
VM Tport State Prt/Eth
Errors SeqNum
1 GRE up
Local-addr: 10.1.1.101 0x8921
1 0
Remote-addr: 10.1.1.5 0x8921
0 2
Sticky errors: yes, Tunnel: Tunnel0, GRE handle: 0xF2C296A4
OCE: RAW midchain out of Tunnel0, addr 10.1.1.5 F2C56F58
Punt/Copy: 0/0, Reinject/Inject/Drop: 0/2/0
Refcount: 1, Memory: 0xF1AFABF8

Errors: 0
CFT FOs free/in use: 264/0, errors: 0
Reinject handles free/in use: 500/0, errors: 0
Encode/decode buffer size: 65665 bytes, memory: 0xF2AB0888
NE101#


*** After injecting 4 more packets, six in total ***
NE101#show cef dpss
VM Tport State Prt/Eth
Errors SeqNum
1 GRE up
Local-addr: 10.1.1.101 0x8921
1 0
Remote-addr: 10.1.1.5 0x8921
0 6
Sticky errors: yes, Tunnel: Tunnel0, GRE handle: 0xF2C296A4
OCE: RAW midchain out of Tunnel0, addr 10.1.1.5 F2C56F58
Punt/Copy: 0/0, Reinject/Inject/Drop: 0/6/0
Refcount: 1, Memory: 0xF1AFABF8

Errors: 0
CFT FOs free/in use: 264/0, errors: 0
Reinject handles free/in use: 500/0, errors: 0
Encode/decode buffer size: 65665 bytes, memory: 0xF2AB0888
NE101#

Best regards
Viktor

Subject: RE: Re: New Message from Zach Seils in onePK - Troubleshooting: RE: Difficu
Replied by: Zach Seils on 25-02-2013 07:29:49 PM
Were you able to collect the output of the command show ip cef switching statistics before and after packet injection?

Subject: RE: Re: New Message from Zach Seils in onePK - Troubleshooting: RE: Difficu
Replied by: Joseph Clarke on 25-02-2013 11:18:59 PM
The ARP generation will not work as the ethertype is not yet supported.  ARP is being considered, though.
As for general broadcast IP, I have confirmed this works.  I used my packet generator code to generate a broadcast ICMP frame with these parameters:
Dst MAC : ff:ff:ff:ff:ff:ff
Src MAC : (NE intf MAC)
Src IP : 10.1.1.4
Dst IP : 10.1.1.255
This works as the locally attached Linux box replies to the ICMP echo request.  Here is the debug output I see when I generate the frame:
*Feb 26 05:17:17.329: CEF DPSS GRE: CEF rx pak: 0xADC1B458, size: 93
*Feb 26 05:17:17.331: CEF DPSS: Parsing inject message
*Feb 26 05:17:17.331: CEF DPSS: sync flowid: 4785, remote-flowid: 1, size: 13
*Feb 26 05:17:17.331: CEF DPSS: Cannot get flow tuple 4785, 1
*Feb 26 05:17:17.331: CEF DPSS: Injecting packet to OUTPUT(Ethernet0/1)


If you can provide more details of the packet being generated, I can try and test this locally.
 

Subject: RE: Re: New Message from Zach Seils in onePK - Troubleshooting: RE: Difficu
Replied by: Zach Seils on 26-02-2013 07:54:40 AM
I'm wondering if the IP header checksum is incorrect.  The output of sh ip cef switching stats would show this.
Zach

Subject: Re: New Message from Joseph Clarke in onePK - Troubleshooting: RE: Re: New
Replied by: Viktor S. Wold Eide on 26-02-2013 08:24:54 AM
Tanks a lot for the help, both Joseph and Zach.

Short version: Broadcast works with
ONEP_TARGET_LOCATION_HARDWARE_DEFINED_OUTPUT

Longer version and questions:  In order to avoid filling in layer 2 mac
addresses we've been using ONEP_TARGET_LOCATION_HARDWARE_DEFINED_PREROUTING
(as learnt via the forum). In combination with broadcast the debug output
on the router is as follows:
*Feb 26 12:40:30.575: CEF DPSS GRE: CEF rx pak: 0xEF797FE0, size: 109
*Feb 26 12:40:30.575: CEF DPSS: Parsing inject message
*Feb 26 12:40:30.575: CEF DPSS: sync flowid: 18, remote-flowid: 6, size: 13
*Feb 26 12:40:30.575: CEF DPSS: Could not attach CFT FO for flow 18
*Feb 26 12:40:30.575: CEF DPSS: Injected IPV4 packet punted to PS
What is the meaning of "punted to PS"?

When filling in the mac address information and using
ONEP_TARGET_LOCATION_HARDWARE_DEFINED_OUTPUT, the UDP broadcast packet
destined for 10.12.1.255 is sent out on the interface, as desired:
*Feb 26 12:40:52.528: CEF DPSS GRE: CEF rx pak: 0xEF797FE0, size: 109
*Feb 26 12:40:52.528: CEF DPSS: Parsing inject message
*Feb 26 12:40:52.528: CEF DPSS: sync flowid: 18, remote-flowid: 6, size: 13
*Feb 26 12:40:52.528: CEF DPSS: Could not attach CFT FO for flow 18
*Feb 26 12:40:52.528: CEF DPSS: Injecting packet to OUTPUT(Ethernet0/1)
The broadcast packet is also received by tcpdump / wireshark on the
interface, so it looks fine.

Have you tried the ONEP_TARGET_LOCATION_HARDWARE_DEFINED_PREROUTING option
in combination with broadcast, and in case, is it working for you?

Best regards
Viktor

Subject: RE: Re: New Message from Zach Seils in onePK - Troubleshooting: RE: Difficu
Replied by: Zach Seils on 26-02-2013 09:28:21 AM
The PS in "Injected IPV4 packet punted to PS" means process switched.  This is normal.  It just means that the injected packet is bypassing the input features.  I have some ideas I can test with Joe.  Let me do that and get back with you.
Thanks,
Zach
 

Subject: RE: Re: New Message from Zach Seils in onePK - Troubleshooting: RE: Difficu
Replied by: Zach Seils on 28-02-2013 06:19:47 PM
Viktor,
The issue with injecting a broadcast packet at INPUT or PREROUTING is that the router thinks the packet is destined for itself.  Here is the output from debug cef dpss and debug ip cef packet:
*Feb 28 16:56:41.404: CEF DPSS GRE: CEF rx pak: 0xB138B9A0, size: 74
*Feb 28 16:56:41.404: CEF DPSS: Parsing inject message
*Feb 28 16:56:41.404: CEF-Debug: Packet from 10.15.1.4 (Nu0) to 10.15.1.255, Packet for us
*Feb 28 16:56:41.404:   ihl=20, length=28, tos=0, ttl=255, checksum=42160, offset=0
*Feb 28 16:56:41.404:     UDP src=1025, dst=666, length=8, checksum=57890
*Feb 28 16:56:41.404: CEF DPSS: Injected IPV4 packet punted to PS
Now look at the difference when I specify a unicast destination:
*Feb 28 16:56:50.485: CEF DPSS GRE: CEF rx pak: 0xB138B9A0, size: 74
*Feb 28 16:56:50.485: CEF DPSS: Parsing inject message
*Feb 28 16:56:50.485: CEF DPSS: Packet injected to PRE-ROUTING(0)
Then I see a response from the destination:
*Feb 28 16:56:50.486: CEF-Debug: Packet from 10.15.1.1 (Et1/0) to 10.15.1.4, Packet for us
*Feb 28 16:56:50.486:   ihl=20, length=56, tos=192, ttl=254, checksum=42463, offset=0
*Feb 28 16:56:50.486:     ICMP type=3, code=3, checksum=4921
*Feb 28 16:56:50.486:          unreachable: port
To inject a packet with a broadcast destination, you'll need to use the OUTPUT target.
Please let us know if you have any questions.
Regards,
Zach

Subject: Re: New Message from Zach Seils in onePK - Troubleshooting: RE: Re: New Mes
Replied by: Viktor S. Wold Eide on 01-03-2013 08:00:24 AM
Thanks a lot for the feedback, Zach. This is consistent with what we have
experienced - OUTPUT target is required for IP broadcast.

Isn't it reasonable to expect the IP broadcast to be destined for both
itself (potential other processes on the router) and also being sent out on
the specific interface?

Is the current behaviour intended or something that you consider changing /
fixing in the future? It is not obvious that IP broadcast should not be
used in combination with PRE-ROUTING. And similarly, how about IP multicast?

Best regards
Viktor

Subject: RE: Re: New Message from Zach Seils in onePK - Troubleshooting: RE: Re: New
Replied by: Einar Nilsen-Nygaard on 01-03-2013 04:18:20 PM
Viktor, This is something we can consider enhancing. We would probably introduce a new target for this. We'll take your feedback as input to the engineering requirements for this.
 
Einar

Subject: RE: Difficulties injecting broadcast packets
Replied by: Joseph Clarke on 04-03-2013 01:32:33 PM
Viktor, I want to make sure I understand your use case correctly.  Would it be sufficient for you if there was an API such that you could pass an interface, packet type, and packet buffer at L3 and higher, and the network element would take care of adding the L2 portion based on the target interface and ethertype (assuming the dst MAC would be a broadcast MAC)?  We are discussing some things internally, and I want to make sure your use case is represented properly.  Thanks.

Subject: Re: New Message from Einar Nilsen-Nygaard in onePK - Troubleshooting: RE: R
Replied by: Viktor S. Wold Eide on 05-03-2013 10:10:49 AM
It's good to hear Einar.

During the EFT period, we provide feedback that hopefully is useful for you
to enhance the platform. We are currently in a testing and evaluation
phase, but we will try to point out functionality that is essential and
hence likely very hard to circumvent and realize by other means.

Best regards
Viktor

Subject: Re: New Message from Joseph Clarke in onePK - Troubleshooting: RE: Difficul
Replied by: Viktor S. Wold Eide on 06-03-2013 05:17:18 AM
Hi Joseph, let me take a step back here first.

We are doing some testing with a discovery protocol for auto-configuration
which should be running on nodes also before IPv4 and/or IPv6 is enabled on
the specific interfaces. In this case we need to send and receive packets
via onepk, encapsulated in L2 frames and where the frames are sent as link
layer broadcast (or link layer multicast). This is what we looked at
initially. To the best of our knowledge this is not currently supported in
onePK.

Would it be possible to allow any valid L2 frame to be sent via onePK,
having our custom protocol data encapsulated in the paylaod? We do not know
how you perform the "reject packet check" internally, neither in onePK nor
in the other router modules after that. Maybe it is possible, for example,
to have a onep configuration parameter in the router which disables the
discarding of non IP packets?

As an alternative in the current phase, we use IP broadcast which works
from within onePK, just to test the basic concepts.

Best regards
Viktor

Subject: RE: Difficulties injecting broadcast packets
Replied by: Joseph Clarke on 06-03-2013 04:09:36 PM
Thanks for this, Viktor.  This is very helpful to me for driving the L2 use cases.  I've passed your use case on to product marketing in the hopes that we can get L2 prioritized.
One of my colleagues mentioned to me that this might work on NX-OS today, but is not yet there on IOS.  IPv4 and v6 are the only allowed types.  There is a challenge to making this work on IOS, which is why we wanted a use case to help prioritize engineering resources.

Subject: RE: Re: New Message from Zach Seils in onePK - Troubleshooting: RE: Re: New
Replied by: Derek Fawcus on 10-06-2013 12:38:59 PM
Viktor S. Wold Eide:
 And similarly, how about IP multicast?


Have you actually tried injecting IP multicast packets?

Assuming a properly configured router,  I would expect it to just work - to a similar level that unicast works.

i.e. inject to output has to be fully encapsulated for the target group,  and for inject to input or prerouting the destination group address needs
to pass the MFIB input acceptance check,  or be of local interest to the router.
The acceptance check state is included in the ouput of 'show ip mfib'.

Attachments

    Outcomes