[Mondrian] XMLA servlet running reporting out of memory, biserver then crashes, OS becomes unresponsive.

Brandon Jackson usbrandon at gmail.com
Thu May 2 21:38:22 EDT 2013


http://jira.pentaho.com/browse/MONDRIAN-1569

I found a reproduction path using SteelWheels and a PDI tranformation.
 Basically those mondrian.server.DynamicContentFinder$timer threads are
created in clumps each time a query hits the box and they never go away.
 The OS runs out of resources and everything dies.  The JVM Monitor made it
very clear what was happening.
A video of that experience is here http://www.screencast.com/t/P7WwB5Vd1

I was surprised that the queries took so long.  Makes me think caching was
not working for queries coming through XMLA, but I could be wrong.

Thank you for enduring so many emails to the thread.   Thanks for your
continued work on this great project.

Brandon




On Thu, May 2, 2013 at 4:21 PM, Brandon Jackson <usbrandon at gmail.com> wrote:

> In repeated query executions the result sets came back slower and slower.
> The performance monitor reported the following 1, 5, 15 minute utilization:
>
> 15.54, 8.28, 4.12
> 33.36, 13.31, 5.81
> 35.53, 14.54, 6.39
> 65.95, 26.31, 10.90
>
> http://screencast.com/t/bnY3ZJN7R
>
> Java is the process pegging out this little dev box.  The system does not
> appear to have freed up its resources after a few minutes.
>
>
>
>
> On Thu, May 2, 2013 at 4:13 PM, Brandon Jackson <usbrandon at gmail.com>wrote:
>
>> I ran this version you recommended before changing operating system
>> limits.  I think the relevant limit is nprocs, which by default for the
>> user was set to 1024.
>> The server quickly crashed out under the same XMLA query.  To circumvent
>> building mondrian.jar myself, I stole the copy found in the workbench
>> snapshot zip.
>>
>> I changed the OS limits to the ones recommended by Rossi.  We are testing
>> continuously under the new limits to see where it will pop.  By what you've
>> said, it should still pop.
>>
>>  I'm not sure what the implication is with the timing of GC that you
>> mentioned.  Maybe these resource limits raising up will allow the computer
>> to 'catch up' in a sense an prevent itself from going over a cliff.  If
>> that is not the case, then I would still consider this as still broken.
>>
>> Please let me know what I can report back that would be insightful for
>> solving the real problem.
>>
>> Thanks for the pointers so far.
>>
>> Brandon
>>
>>
>>
>>
>>
>> On Thu, May 2, 2013 at 3:59 PM, Luc Boudreau <lucboudreau at gmail.com>wrote:
>>
>>> The problem was that DynamicDataSourceServlet was creating a new
>>> DynamicContentFinder for each request creating a new Timer thread each time.
>>>
>>> It was fixed today in this commit:
>>> https://github.com/pentaho/mondrian/commit/32e653b3264a7f6af4f9d26816dbf04d772148fb
>>>
>>> This can be downloaded from CI at:
>>> http://ci.pentaho.com/view/Analysis/job/mondrian/lastSuccessfulBuild/artifact/dist/mondrian-TRUNK-SNAPSHOT.zip
>>>
>>> It should be stable enough to be tested, but this is the bleeding edge
>>> of our development. You are warned :)
>>>
>>> Limiting the number of threads at the OS level, as reported by Michele,
>>> only circumvents the issue. The ContentFinder implementations probably get
>>> garbage collected before the VM runs out of threads, but I wouldn't
>>> recommend this approach for multiple reasons, one being that the VM will
>>> slow down considerably because of all the threads reading the datasources
>>> file. The commit mentioned above actually fixed it by keeping a map of
>>> datasource path to finder object. It also cleans them up when the servlet
>>> shuts down.
>>>
>>> Luc
>>>
>>>
>>>
>>>
>>> On Thu, May 2, 2013 at 4:16 PM, Michele Rossi <michele.rossi at gmail.com>wrote:
>>>
>>>> hi,
>>>>
>>>> that happened to us too on a virtual Linux server and we solved it as
>>>> described below:
>>>>
>>>>
>>>>
>>>> /etc/security/limits.conf
>>>>
>>>> exampleUsername soft nofile 65536
>>>> exampleUsername hard nofile 65536
>>>> exampleUsername - nproc 2103295
>>>> exampleUsername hard core unlimited
>>>> exampleUsername soft core unlimited
>>>>
>>>>
>>>>
>>>> I am not 100% sure of which one of those settings fixed it, we left it
>>>> to our sys admins to sort out.
>>>>
>>>>
>>>> I was able to prove that the problem was with the Linux machine set up
>>>> by writing a simple Java program that just created Threads doing nothing
>>>> but Thread.sleep(). The machine previously only allowed the creation of 100
>>>> threads and after that it started failing with the error you reported.
>>>>
>>>>
>>>> Hope this helps.
>>>>
>>>> thanks,
>>>>
>>>> Michele
>>>>
>>>>
>>>>
>>>>
>>>> On 2 May 2013 21:47, Brandon Jackson <usbrandon at gmail.com> wrote:
>>>>
>>>>>  The last day or so after upgrading to 4.8.1-EE, I have encountered a
>>>>> really interesting problem.  After running an OLAP Input step in PDI which
>>>>> uses XMLA to communicate back to the server, the mondrian logs say the
>>>>> following before the whole system becomes toast.  The BI Server shuts down
>>>>> because of 'out of memory', but the OS (Centos 6.3 with Oracle Java
>>>>> 1.6.0_45) also becomes unstable mentioning that it has the inability to
>>>>> fork new processes.  The system must be shutdown via power button at that
>>>>> point (ACPI is still alive and the OS shuts down normally, but nothing new
>>>>> can happen in the OS).
>>>>>
>>>>> 2013-05-02 14:02:31,718 ERROR [mondrian.xmla.XmlaServlet] Errors when
>>>>> handling XML/A message
>>>>> mondrian.xmla.XmlaException: Mondrian Error:XMLA Discover unparse
>>>>> results error
>>>>>     at mondrian.xmla.XmlaHandler.discover(XmlaHandler.java:2867)
>>>>>     at mondrian.xmla.XmlaHandler.process(XmlaHandler.java:663)
>>>>>     at
>>>>> mondrian.xmla.impl.DefaultXmlaServlet.handleSoapBody(DefaultXmlaServlet.java:505)
>>>>>     at mondrian.xmla.XmlaServlet.doPost(XmlaServlet.java:317)
>>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
>>>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
>>>>>     at
>>>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
>>>>>     at
>>>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>>>>>     at
>>>>> org.pentaho.platform.web.http.filters.PentahoWebContextFilter.doFilter(PentahoWebContextFilter.java:142)
>>>>>     at
>>>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
>>>>>     at
>>>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>>>>>     at
>>>>> org.pentaho.platform.web.http.filters.PentahoRequestContextFilter.doFilter(PentahoRequestContextFilter.java:84)
>>>>>     at
>>>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
>>>>>     at
>>>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>>>>>     at
>>>>> org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:378)
>>>>>     at
>>>>> org.springframework.security.intercept.web.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:109)
>>>>>     at
>>>>> org.springframework.security.intercept.web.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:83)
>>>>>     at
>>>>> org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:390)
>>>>>     at
>>>>> org.springframework.security.ui.ExceptionTranslationFilter.doFilterHttp(ExceptionTranslationFilter.java:101)
>>>>>     at
>>>>> org.springframework.security.ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53)
>>>>>     at
>>>>> org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:390)
>>>>>     at
>>>>> org.pentaho.platform.web.http.security.SecurityStartupFilter.doFilter(SecurityStartupFilter.java:103)
>>>>>     at
>>>>> org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:390)
>>>>>     at
>>>>> org.springframework.security.providers.anonymous.AnonymousProcessingFilter.doFilterHttp(AnonymousProcessingFilter.java:105)
>>>>>     at
>>>>> org.springframework.security.ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53)
>>>>>     at
>>>>> org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:390)
>>>>>     at
>>>>> org.pentaho.platform.web.http.security.RequestParameterAuthenticationFilter.doFilter(RequestParameterAuthenticationFilter.java:169)
>>>>>     at
>>>>> org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:390)
>>>>>     at
>>>>> org.springframework.security.ui.basicauth.BasicProcessingFilter.doFilterHttp(BasicProcessingFilter.java:174)
>>>>>     at
>>>>> org.springframework.security.ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53)
>>>>>     at
>>>>> org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:390)
>>>>>     at
>>>>> org.springframework.security.ui.AbstractProcessingFilter.doFilterHttp(AbstractProcessingFilter.java:278)
>>>>>     at
>>>>> org.springframework.security.ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53)
>>>>>     at
>>>>> org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:390)
>>>>>     at
>>>>> org.springframework.security.ui.logout.LogoutFilter.doFilterHttp(LogoutFilter.java:89)
>>>>>     at
>>>>> org.springframework.security.ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53)
>>>>>     at
>>>>> org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:390)
>>>>>     at
>>>>> org.pentaho.platform.web.http.security.HttpSessionReuseDetectionFilter.doFilter(HttpSessionReuseDetectionFilter.java:134)
>>>>>     at
>>>>> org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:390)
>>>>>     at
>>>>> org.springframework.security.context.HttpSessionContextIntegrationFilter.doFilterHttp(HttpSessionContextIntegrationFilter.java:235)
>>>>>     at
>>>>> org.springframework.security.ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53)
>>>>>     at
>>>>> org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:390)
>>>>>     at
>>>>> org.springframework.security.wrapper.SecurityContextHolderAwareRequestFilter.doFilterHttp(SecurityContextHolderAwareRequestFilter.java:91)
>>>>>     at
>>>>> org.springframework.security.ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53)
>>>>>     at
>>>>> org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:390)
>>>>>     at
>>>>> org.springframework.security.util.FilterChainProxy.doFilter(FilterChainProxy.java:175)
>>>>>     at
>>>>> org.springframework.security.util.FilterToBeanProxy.doFilter(FilterToBeanProxy.java:99)
>>>>>     at
>>>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
>>>>>     at
>>>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>>>>>     at
>>>>> com.pentaho.ui.servlet.SystemStatusFilter.doFilter(SourceFile:72)
>>>>>     at
>>>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
>>>>>     at
>>>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>>>>>     at
>>>>> org.pentaho.platform.web.http.filters.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:113)
>>>>>     at
>>>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
>>>>>     at
>>>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>>>>>     at
>>>>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
>>>>>     at
>>>>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
>>>>>     at
>>>>> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:470)
>>>>>     at
>>>>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
>>>>>     at
>>>>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
>>>>>     at
>>>>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
>>>>>     at
>>>>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
>>>>>     at
>>>>> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:857)
>>>>>     at
>>>>> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
>>>>>     at
>>>>> org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
>>>>>     at java.lang.Thread.run(Thread.java:662)
>>>>> Caused by: java.lang.OutOfMemoryError: unable to create new native
>>>>> thread
>>>>>     at java.lang.Thread.start0(Native Method)
>>>>>     at java.lang.Thread.start(Thread.java:640)
>>>>>     at java.util.Timer.<init>(Timer.java:154)
>>>>>     at
>>>>> mondrian.util.UtilCompatibleJdk15.newTimer(UtilCompatibleJdk15.java:129)
>>>>>     at mondrian.olap.Util.newTimer(Util.java:2196)
>>>>>     at
>>>>> mondrian.server.DynamicContentFinder.<init>(DynamicContentFinder.java:60)
>>>>>     at
>>>>> org.pentaho.platform.web.servlet.PentahoXmlaServlet$1.<init>(PentahoXmlaServlet.java:86)
>>>>>     at
>>>>> org.pentaho.platform.web.servlet.PentahoXmlaServlet.makeContentFinder(PentahoXmlaServlet.java:86)
>>>>>     at
>>>>> org.pentaho.platform.web.servlet.PentahoXmlaServlet$3.getConnection(PentahoXmlaServlet.java:254)
>>>>>     at mondrian.xmla.XmlaHandler.getConnection(XmlaHandler.java:2939)
>>>>>     at mondrian.xmla.XmlaHandler.getConnection(XmlaHandler.java:175)
>>>>>     at mondrian.xmla.Rowset.populate(Rowset.java:218)
>>>>>     at mondrian.xmla.Rowset.unparse(Rowset.java:193)
>>>>>     at mondrian.xmla.XmlaHandler.discover(XmlaHandler.java:2861)
>>>>>     ... 65 more
>>>>>
>>>>> _______________________________________________
>>>>> Mondrian mailing list
>>>>> Mondrian at pentaho.org
>>>>> http://lists.pentaho.org/mailman/listinfo/mondrian
>>>>>
>>>>>
>>>>
>>>> _______________________________________________
>>>> Mondrian mailing list
>>>> Mondrian at pentaho.org
>>>> http://lists.pentaho.org/mailman/listinfo/mondrian
>>>>
>>>>
>>>
>>> _______________________________________________
>>> Mondrian mailing list
>>> Mondrian at pentaho.org
>>> http://lists.pentaho.org/mailman/listinfo/mondrian
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.pentaho.org/pipermail/mondrian/attachments/20130502/1324725a/attachment-0001.html 


More information about the Mondrian mailing list