[Mondrian] XMLA servlet running reporting out of memory, biserver then crashes, OS becomes unresponsive.

Luc Boudreau lucboudreau at gmail.com
Thu May 2 16:59:36 EDT 2013


The problem was that DynamicDataSourceServlet was creating a new
DynamicContentFinder for each request creating a new Timer thread each time.

It was fixed today in this commit:
https://github.com/pentaho/mondrian/commit/32e653b3264a7f6af4f9d26816dbf04d772148fb

This can be downloaded from CI at:
http://ci.pentaho.com/view/Analysis/job/mondrian/lastSuccessfulBuild/artifact/dist/mondrian-TRUNK-SNAPSHOT.zip

It should be stable enough to be tested, but this is the bleeding edge of
our development. You are warned :)

Limiting the number of threads at the OS level, as reported by Michele,
only circumvents the issue. The ContentFinder implementations probably get
garbage collected before the VM runs out of threads, but I wouldn't
recommend this approach for multiple reasons, one being that the VM will
slow down considerably because of all the threads reading the datasources
file. The commit mentioned above actually fixed it by keeping a map of
datasource path to finder object. It also cleans them up when the servlet
shuts down.

Luc




On Thu, May 2, 2013 at 4:16 PM, Michele Rossi <michele.rossi at gmail.com>wrote:

> hi,
>
> that happened to us too on a virtual Linux server and we solved it as
> described below:
>
>
>
> /etc/security/limits.conf
>
> exampleUsername soft nofile 65536
> exampleUsername hard nofile 65536
> exampleUsername - nproc 2103295
> exampleUsername hard core unlimited
> exampleUsername soft core unlimited
>
>
>
> I am not 100% sure of which one of those settings fixed it, we left it to
> our sys admins to sort out.
>
>
> I was able to prove that the problem was with the Linux machine set up by
> writing a simple Java program that just created Threads doing nothing but
> Thread.sleep(). The machine previously only allowed the creation of 100
> threads and after that it started failing with the error you reported.
>
>
> Hope this helps.
>
> thanks,
>
> Michele
>
>
>
>
> On 2 May 2013 21:47, Brandon Jackson <usbrandon at gmail.com> wrote:
>
>>  The last day or so after upgrading to 4.8.1-EE, I have encountered a
>> really interesting problem.  After running an OLAP Input step in PDI which
>> uses XMLA to communicate back to the server, the mondrian logs say the
>> following before the whole system becomes toast.  The BI Server shuts down
>> because of 'out of memory', but the OS (Centos 6.3 with Oracle Java
>> 1.6.0_45) also becomes unstable mentioning that it has the inability to
>> fork new processes.  The system must be shutdown via power button at that
>> point (ACPI is still alive and the OS shuts down normally, but nothing new
>> can happen in the OS).
>>
>> 2013-05-02 14:02:31,718 ERROR [mondrian.xmla.XmlaServlet] Errors when
>> handling XML/A message
>> mondrian.xmla.XmlaException: Mondrian Error:XMLA Discover unparse results
>> error
>>     at mondrian.xmla.XmlaHandler.discover(XmlaHandler.java:2867)
>>     at mondrian.xmla.XmlaHandler.process(XmlaHandler.java:663)
>>     at
>> mondrian.xmla.impl.DefaultXmlaServlet.handleSoapBody(DefaultXmlaServlet.java:505)
>>     at mondrian.xmla.XmlaServlet.doPost(XmlaServlet.java:317)
>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
>>     at
>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
>>     at
>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>>     at
>> org.pentaho.platform.web.http.filters.PentahoWebContextFilter.doFilter(PentahoWebContextFilter.java:142)
>>     at
>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
>>     at
>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>>     at
>> org.pentaho.platform.web.http.filters.PentahoRequestContextFilter.doFilter(PentahoRequestContextFilter.java:84)
>>     at
>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
>>     at
>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>>     at
>> org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:378)
>>     at
>> org.springframework.security.intercept.web.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:109)
>>     at
>> org.springframework.security.intercept.web.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:83)
>>     at
>> org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:390)
>>     at
>> org.springframework.security.ui.ExceptionTranslationFilter.doFilterHttp(ExceptionTranslationFilter.java:101)
>>     at
>> org.springframework.security.ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53)
>>     at
>> org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:390)
>>     at
>> org.pentaho.platform.web.http.security.SecurityStartupFilter.doFilter(SecurityStartupFilter.java:103)
>>     at
>> org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:390)
>>     at
>> org.springframework.security.providers.anonymous.AnonymousProcessingFilter.doFilterHttp(AnonymousProcessingFilter.java:105)
>>     at
>> org.springframework.security.ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53)
>>     at
>> org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:390)
>>     at
>> org.pentaho.platform.web.http.security.RequestParameterAuthenticationFilter.doFilter(RequestParameterAuthenticationFilter.java:169)
>>     at
>> org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:390)
>>     at
>> org.springframework.security.ui.basicauth.BasicProcessingFilter.doFilterHttp(BasicProcessingFilter.java:174)
>>     at
>> org.springframework.security.ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53)
>>     at
>> org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:390)
>>     at
>> org.springframework.security.ui.AbstractProcessingFilter.doFilterHttp(AbstractProcessingFilter.java:278)
>>     at
>> org.springframework.security.ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53)
>>     at
>> org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:390)
>>     at
>> org.springframework.security.ui.logout.LogoutFilter.doFilterHttp(LogoutFilter.java:89)
>>     at
>> org.springframework.security.ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53)
>>     at
>> org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:390)
>>     at
>> org.pentaho.platform.web.http.security.HttpSessionReuseDetectionFilter.doFilter(HttpSessionReuseDetectionFilter.java:134)
>>     at
>> org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:390)
>>     at
>> org.springframework.security.context.HttpSessionContextIntegrationFilter.doFilterHttp(HttpSessionContextIntegrationFilter.java:235)
>>     at
>> org.springframework.security.ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53)
>>     at
>> org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:390)
>>     at
>> org.springframework.security.wrapper.SecurityContextHolderAwareRequestFilter.doFilterHttp(SecurityContextHolderAwareRequestFilter.java:91)
>>     at
>> org.springframework.security.ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53)
>>     at
>> org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:390)
>>     at
>> org.springframework.security.util.FilterChainProxy.doFilter(FilterChainProxy.java:175)
>>     at
>> org.springframework.security.util.FilterToBeanProxy.doFilter(FilterToBeanProxy.java:99)
>>     at
>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
>>     at
>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>>     at com.pentaho.ui.servlet.SystemStatusFilter.doFilter(SourceFile:72)
>>     at
>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
>>     at
>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>>     at
>> org.pentaho.platform.web.http.filters.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:113)
>>     at
>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
>>     at
>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>>     at
>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
>>     at
>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
>>     at
>> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:470)
>>     at
>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
>>     at
>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
>>     at
>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
>>     at
>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
>>     at
>> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:857)
>>     at
>> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
>>     at
>> org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
>>     at java.lang.Thread.run(Thread.java:662)
>> Caused by: java.lang.OutOfMemoryError: unable to create new native thread
>>     at java.lang.Thread.start0(Native Method)
>>     at java.lang.Thread.start(Thread.java:640)
>>     at java.util.Timer.<init>(Timer.java:154)
>>     at
>> mondrian.util.UtilCompatibleJdk15.newTimer(UtilCompatibleJdk15.java:129)
>>     at mondrian.olap.Util.newTimer(Util.java:2196)
>>     at
>> mondrian.server.DynamicContentFinder.<init>(DynamicContentFinder.java:60)
>>     at
>> org.pentaho.platform.web.servlet.PentahoXmlaServlet$1.<init>(PentahoXmlaServlet.java:86)
>>     at
>> org.pentaho.platform.web.servlet.PentahoXmlaServlet.makeContentFinder(PentahoXmlaServlet.java:86)
>>     at
>> org.pentaho.platform.web.servlet.PentahoXmlaServlet$3.getConnection(PentahoXmlaServlet.java:254)
>>     at mondrian.xmla.XmlaHandler.getConnection(XmlaHandler.java:2939)
>>     at mondrian.xmla.XmlaHandler.getConnection(XmlaHandler.java:175)
>>     at mondrian.xmla.Rowset.populate(Rowset.java:218)
>>     at mondrian.xmla.Rowset.unparse(Rowset.java:193)
>>     at mondrian.xmla.XmlaHandler.discover(XmlaHandler.java:2861)
>>     ... 65 more
>>
>> _______________________________________________
>> Mondrian mailing list
>> Mondrian at pentaho.org
>> http://lists.pentaho.org/mailman/listinfo/mondrian
>>
>>
>
> _______________________________________________
> Mondrian mailing list
> Mondrian at pentaho.org
> http://lists.pentaho.org/mailman/listinfo/mondrian
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.pentaho.org/pipermail/mondrian/attachments/20130502/8440fa96/attachment-0001.html 


More information about the Mondrian mailing list