dpss low throughput when divert/re-inject in combination with inject

Document created by cdnadmin on Jan 25, 2014
Version 1Show Document
  • View in full screen mode
This document was generated from CDN thread

Created by: Viktor S. Wold Eide on 12-06-2013 01:43:10 PM
Hi,

As stated in the subject, we experience significantly reduced throughput when packets are diverted and re-inject in combination with injecting new, freshly generated packets. We send data from one Linux computer to another Linux computer, each connected directly to their respective onepk enabled router. The data transfer is TCP.

We have reproduced the problem described above with the CustomEncrytion sample application, patched to also inject packets (not only divert and re-inject). The patch is posted in another thread, titled "dpss segmentation faults":
http://developer.cisco.com/web/onepk/community/-/message_boards/message/15627789#_19_message_16012637
Note that port 23 is used in the ACL filter for punting in the patch.

With the patch, a new packet is sent every second, from startup. If we add, e.g., a one minute delay before starting to inject new packets (before entering the while loop in the code), TCP performance is OK until the first new packet is injected.

Best regards
Viktor

Subject: RE: dpss low throughput when divert/re-inject in combination with inject
Replied by: Joseph Clarke on 13-06-2013 03:54:14 PM
What do you observe as low throughput specifically?  How are you measuring it, and what do you see as the percentage of decrease?  I added a delay as you describe, but at least on an interactive basis, I do not observe any difference between the time prior to injection and then during injection.

Subject: Re: New Message from Joseph Clarke in onePK - Troubleshooting: RE: dpss low
Replied by: Viktor S. Wold Eide on 14-06-2013 03:35:19 AM
The observed drop in throughput is for bulk data transfer, such as TCP (or
UDP) transfer of a file from one computer to another via two onepk enabled
routers.

When the TCP packets are only diverted and re-injected, the throughput is
around 50 Mbps. As soon as the first new packet is injected the throughput
drops to well below 100 kbps, in other words the throughput drops to
roughly 1 / 1000 of the speed before the first new packet is injected. The
drop happens as soon as the first new packet is injected, as verified by
having an initial delay before injecting the first packet. The throughput
drop is observed via different monitoring tools (gnome-system-monitor,
iftop, etc) and from the change in size for the incoming bulk data, written
to file.

It is still possible to have an interactive connection (telnet or nc), but
for bulk transfer the throughput is basically unusable.

Restarting the file transfer or the onpk application does not help.
Restarting the dpss process gives a new fresh start, and the described
behavior can be repeated.

Note that even if a new packets is injected at a very low frequency, e.g.,
a one minute interval, the throughput remains low.  The DPSS process has to
be restarted.

Let me know if you observe the same kind of behavior.

Best regards
Viktor

2013/6/13 Cisco Developer Community Forums <cdicuser@developer.cisco.com>

> Joseph Clarke has created a new message in the forum "Troubleshooting":
> -------------------------------------------------------------- What do you
> observe as low throughput specifically?  How are you measuring it, and what
> do you see as the percentage of decrease?  I added a delay as you describe,
> but at least on an interactive basis, I do not observe any difference
> between the time prior to injection and then during injection.
> --
> To respond to this post, please click the following link:
> http://developer.cisco.com/web/onepk/community/-/message_boards/view_message/16182803or simply reply to this email.

Subject: RE: dpss low throughput when divert/re-inject in combination with inject
Replied by: Joseph Clarke on 27-06-2013 12:18:04 AM
Again, sorry for the delay.  I have been at CiscoLive! for the past week.  I had an email conversation with a DPSS developer, and they believe this, too, is fixed in GA.  They saw some performance issues in testing on your branch of code and that has since been fixed.  The general availability release this summer should fix this and your segfault issues.  Once you're able to get that code, we'd appreciate confirmation for your specific use case (as tests show good results locally).

Subject: RE: dpss low throughput when divert/re-inject in combination with inject
Replied by: Viktor S. Wold Eide on 09-10-2013 03:40:07 AM
Joseph Clarke:
Again, sorry for the delay.  I have been at CiscoLive! for the past week.  I had an email conversation with a DPSS developer, and they believe this, too, is fixed in GA.  They saw some performance issues in testing on your branch of code and that has since been fixed.  The general availability release this summer should fix this and your segfault issues.  Once you're able to get that code, we'd appreciate confirmation for your specific use case (as tests show good results locally).
Just wanted to let you know that we did some tests after the v1.0.0.84 release. The issue (low throughput when divert/re-inject in combination with inject) seems to be fixed and the throughput significantly improved. Thanks a lot.

Best regards
Viktor

Attachments

    Outcomes