Exception: Maximum AXL Memory Allocation Consumed

Version 1
    This document was generated from CDN thread

    Created by: Mayank Sharma on 24-05-2010 02:09:17 PM
    Hi,
     
    I have been using a program that I wrote using C#, which pulls data from some tables in CUCM. Everything was fine until recently, I started receiving this exception from the CUCM.
     
    Maximum AXL Memory Allocation Consumed.Please retry once requests in progress have completed
     
    This program runs once everyday, and I started receiving this message since Friday, 5/21/10. I am not aware of any other AXL programs running, and I am using throttling to limit the number of rows (by looking at the exception, which gives me the total rows and suggestive rows and then using skip in the query).  
     
    Now every time I run this program, I get this exception. It seems like AXL memory didn't get released somehow.
     
    Any suggestions on how to troubleshoot?
     
    Thanks,
    Max
     

    Subject: RE: Exception: Maximum AXL Memory Allocation Consumed
    Replied by: Mayank Sharma on 24-05-2010 08:29:52 PM
    Thanks for the response David.
     
    Actually, I am using throttling for the requests. I use the following Server Response to break one large request into multiple smaller requests. This is from the Cisco document for throttling AXL queries.
     
    Server response: ¿Query request too large. Total rows matched: <Matched Rows>. Suggested row fetch: less than <Number of Rows>¿
     
    Then I use the values of <Matched Rows> and <Number of Rows> to figure out the iterations.
     
    The problem now is that the server response is about maximum AXL memory usage instead of the one I mentioned here.
     
    Also, what do you mean by "detailed AXL service logs from a restart of the Publisher through observed symptoms."
     
    Do you mean I should restart the publisher and get the logs, or restart the service and get the logs, or don't restart anything and just get the logs.
     
    Thanks,
    Max
     

    Subject: RE: Exception: Maximum AXL Memory Allocation Consumed
    Replied by: David Staudt on 24-05-2010 07:46:06 PM
    See this thread for a similar issue: http://developer.cisco.com/web/axl/forums/-/message_boards/message/2175379
     
    See this blog entry for memory throttle details: http://developer.cisco.com/web/axl/blogroll/-/blogs/what-s-new-in-cisco-unified-communications-manager-6-1-3_1?_33_redirect=/web/axl/blogroll%3Fp_p_id%3D33%26p_p_lifecycle%3D0%26p_p_state%3Dnormal%26p_p_mode%3Dview%26p_p_col_id%3Dcolumn-1%26p_p_col_count%3D1%26_33_struts_action%3D%252Fblogs%252Fview%26_33_delta%3D5%26_33_keywords%3D%26_33_advancedSearch%3Dfalse%26_33_andOperator%3Dtrue%26cur%3D2
     
    I have had one previous report that seemed to suggest AXL memory tracking was not releasing properly, but it was unreproducible.  If you still feel this is a possibility, the best evidence would be detailed AXL service logs from a restart of the Publisher through observed symptoms.
     
    If you are retrieving large data sets (for example, all devices, or all users) you will need to consider using skip/first techniques to reduce response size.
     
     

    Subject: RE: Exception: Maximum AXL Memory Allocation Consumed
    Replied by: Mayank Sharma on 25-05-2010 06:40:47 PM
    OK, this is what I tried.
     
    Initially, I was using exception from the server to figure out the TotalRowCount and SuggestedRowCount. I thought may be if I get the TotalRowCount by doing "SELECT COUNT(*) FROM Table" query and set the SuggestedRowCount = 200, it might help.
     
    This is what happens now:
     
    TotalRowCount = 1980
    SuggestedRowCount = 200
    NumberOfIterations = 10
     
    It goes thru the first iteration and imports 200 records, but for the second iteration, I get the same "Maximum AXL Memory Allocation Consumed" exception. I have restarted the AXL service already but it did not help. 

    Subject: RE: Exception: Maximum AXL Memory Allocation Consumed
    Replied by: David Staudt on 25-05-2010 10:33:35 PM
    What kind of query are you running?  If possible pls post the SQL.  Unless there are a large number of joins, it seems 200 records should not be taxing.  If you lower the 'skip' to something like 100 do you still eventually encounter the problem?
     
    Are there other AXL applications operating on the system?  The max memory available is shared between all AXL apps.
     
     

    Subject: RE: Exception: Maximum AXL Memory Allocation Consumed
    Replied by: Mayank Sharma on 26-05-2010 12:16:11 PM
    Here's the query I am using,
     
    SELECT * FROM device WHERE tkclass = 1
     
    There are about 1980 records in this table. I don't think there's any other AXL application running and I get this error every time I execute this query. I attached a log file I got from RTMT. Is there a way to see what is holding up the memory?
     
    I tried with skipping 100 too, it goes through first iteration, gets 100 records, goes through second iteration, gets next 100 records and then it bombs. It seems like 200 is the limit.
     
    UPDATE: I tried something else, I removed the WHERE clause tkclass=1, server threw the usual error which mentions TotalRows and SuggestedRows. But when the code tried to SELECT suggested rows, it went back the error or maximum AXL memory allocation consumed.
     
    More Updates: I changed the query from using "*" to only 2-3 columns, which I actually need and no more exceptions. Although this would work for now, I am concerned about scenarios where people do need the whole tables. This one was an odd ball where throttling did not work as the documentation says it should.
     
    Thanks again,
    Max