AXL/EMAPI Memory Leak?

Version 1
    This document was generated from CDN thread

    Created by: JAMES DEPHILLIP II on 13-06-2013 05:46:10 PM
    Are there limits to the number of AXL get request and EMAPI requests?

    We have been having issues where it once a certain load is placed onto
    the Tomcat service (via EMAPI and AXL) the CallManager will start
    failing to respond to a TCP SYN handshake on 8443. We confirmed with a
    Wireshark capture on CallManager that some incoming TCP SYNs are not
    responded to at all. Even TCP SYN retransmits from the same source
    port are not responded to.

    When we piloted the application in our production environment, the
    load was fairly low (roughly 200-450 application requests per day) and
    we didn't encounter issue. Right as we implemented to all users the
    number of application requests jumped to about 12,000 per day, the
    problem started the same day.

    The percentage of failures increases daily. The first day in
    production, about 1% of our application requests had EMAPI/AXL
    failures. The second day it was about 3%. The percentage of failures
    increased every day (even on weekends when there was less load), until
    a week later when roughly 25% of requests were failing.

    We restarted the tomcat service but that didn't resolve the issue.
    When we rebooted the CallManager server, the error percentage dropped
    back to 0 but then it started slowly increasing just like the first
    day we put it in production.

    The application that I am working on does axl lookups for device
    logout profile and user profile. It will then do an EMAPI lookup for
    the logged in user and EMAPI login or logout and sometimes both.

    We've ruled out network issues, because we've confirmed in a Wireshark
    capture that the CallManager server isn't responding to some TCP SYN
    requests. Because TCP SYNs aren't being responded to, we think it must
    be something on CallManager. Also, because of the fact that the error
    percentage increases daily, it seems almost like a memory leak.

    Does anyone have any ideas why this would happen or what we could do
    to narrow down the problem?

    Thanks,
    Jim

    Subject: RE: AXL/EMAPI Memory Leak?
    Replied by: David Staudt on 14-06-2013 02:43:14 PM
    With AXL requests of any scale, it is important to re-use the session-IDs provided in the two response cookies

    POST https://10.88.131.141:8443/axl/ HTTP/1.1
    Accept-Encoding: gzip,deflate
    Content-Type: text/xml;charset=UTF-8
    Authorization: Basic QWRtaW5pc3RyYXRvcjpjaXNjbyExMjM=
    SOAPAction: "CUCMB ver=8.5"
    Content-Length: 249
    Host: 10.88.131.141:8443
    Connection: Keep-Alive
    User-Agent: Apache-HttpClient/4.1.1 (java 1.5)

    <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:ns="http://www.cisco.com/AXL/API/8.5">
       <soapenv:Header/>
       <soapenv:Body>
          <ns:getCCMVersion>
          </ns:getCCMVersion>
       </soapenv:Body>
    </soapenv:Envelope>
    ---------------------
    HTTP/1.1 200 OK
    Set-Cookie: JSESSIONIDSSO=C5374FFF9F8F25DA2A5C9DE1787D94B7; Path=/; Secure
    Set-Cookie: JSESSIONID=25CD2D2A71298D55118201EEF3DE0550; Path=/axl; Secure

    Content-Type: text/xml;charset=UTF-8
    Content-Length: 347
    Date: Fri, 14 Jun 2013 19:28:38 GMT
    Server:

    <?xml version='1.0' encoding='UTF-8'?><soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><ns:getCCMVersionResponse xmlns:ns="http://www.cisco.com/AXL/API/8.5"><return><componentVersion><version>8.5.1.10000(26)</version></componentVersion></return></ns:getCCMVersionResponse></soapenv:Body></soapenv:Envelope>

    ---------------------
    POST https://10.88.131.141:8443/axl/ HTTP/1.1
    Accept-Encoding: gzip,deflate
    Content-Type: text/xml;charset=UTF-8
    Cookie: JSESSIONIDSSO=C5374FFF9F8F25DA2A5C9DE1787D94B7; Path=/; Secure
    Cookie: JSESSIONID=25CD2D2A71298D55118201EEF3DE0550; Path=/axl; Secure

    SOAPAction: "CUCMB ver=8.5"
    Content-Length: 249
    Host: 10.88.131.141:8443
    Connection: Keep-Alive
    User-Agent: Apache-HttpClient/4.1.1 (java 1.5)

    <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:ns="http://www.cisco.com/AXL/API/8.5">
       <soapenv:Header/>
       <soapenv:Body>
          <ns:getCCMVersion>
          </ns:getCCMVersion>
       </soapenv:Body>
    </soapenv:Envelope>
    ---------------------
    HTTP/1.1 200 OK
    Content-Type: text/xml;charset=UTF-8
    Content-Length: 347
    Date: Fri, 14 Jun 2013 19:33:23 GMT
    Server:

    <?xml version='1.0' encoding='UTF-8'?><soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><ns:getCCMVersionResponse xmlns:ns="http://www.cisco.com/AXL/API/8.5"><return><componentVersion><version>8.5.1.10000(26)</version></componentVersion></return></ns:getCCMVersionResponse></soapenv:Body></soapenv:Envelope>

    Subject: RE: AXL/EMAPI Memory Leak?
    Replied by: JAMES DEPHILLIP II on 16-06-2013 09:53:51 AM
    Thanks David does this apply to EMAPI as well? We noticed more issues with EMAPI than AXL...

    Subject: RE: AXL/EMAPI Memory Leak?
    Replied by: David Staudt on 17-06-2013 11:11:13 AM
    All (most?) of the UCM web services APIs run through the on-board Tomcat session manager, so believe E/M would benefit as well.