[Mondrian] XMLA servlet running reporting out of memory, biserver then crashes, OS becomes unresponsive.

Brandon Jackson usbrandon at gmail.com
Thu May 2 17:13:12 EDT 2013


I ran this version you recommended before changing operating system limits.
 I think the relevant limit is nprocs, which by default for the user was
set to 1024.
The server quickly crashed out under the same XMLA query.  To circumvent
building mondrian.jar myself, I stole the copy found in the workbench
snapshot zip.

I changed the OS limits to the ones recommended by Rossi.  We are testing
continuously under the new limits to see where it will pop.  By what you've
said, it should still pop.

I'm not sure what the implication is with the timing of GC that you
mentioned.  Maybe these resource limits raising up will allow the computer
to 'catch up' in a sense an prevent itself from going over a cliff.  If
that is not the case, then I would still consider this as still broken.

Please let me know what I can report back that would be insightful for
solving the real problem.

Thanks for the pointers so far.

Brandon





On Thu, May 2, 2013 at 3:59 PM, Luc Boudreau <lucboudreau at gmail.com> wrote:

> The problem was that DynamicDataSourceServlet was creating a new
> DynamicContentFinder for each request creating a new Timer thread each time.
>
> It was fixed today in this commit:
> https://github.com/pentaho/mondrian/commit/32e653b3264a7f6af4f9d26816dbf04d772148fb
>
> This can be downloaded from CI at:
> http://ci.pentaho.com/view/Analysis/job/mondrian/lastSuccessfulBuild/artifact/dist/mondrian-TRUNK-SNAPSHOT.zip
>
> It should be stable enough to be tested, but this is the bleeding edge of
> our development. You are warned :)
>
> Limiting the number of threads at the OS level, as reported by Michele,
> only circumvents the issue. The ContentFinder implementations probably get
> garbage collected before the VM runs out of threads, but I wouldn't
> recommend this approach for multiple reasons, one being that the VM will
> slow down considerably because of all the threads reading the datasources
> file. The commit mentioned above actually fixed it by keeping a map of
> datasource path to finder object. It also cleans them up when the servlet
> shuts down.
>
> Luc
>
>
>
>
> On Thu, May 2, 2013 at 4:16 PM, Michele Rossi <michele.rossi at gmail.com>wrote:
>
>> hi,
>>
>> that happened to us too on a virtual Linux server and we solved it as
>> described below:
>>
>>
>>
>> /etc/security/limits.conf
>>
>> exampleUsername soft nofile 65536
>> exampleUsername hard nofile 65536
>> exampleUsername - nproc 2103295
>> exampleUsername hard core unlimited
>> exampleUsername soft core unlimited
>>
>>
>>
>> I am not 100% sure of which one of those settings fixed it, we left it to
>> our sys admins to sort out.
>>
>>
>> I was able to prove that the problem was with the Linux machine set up by
>> writing a simple Java program that just created Threads doing nothing but
>> Thread.sleep(). The machine previously only allowed the creation of 100
>> threads and after that it started failing with the error you reported.
>>
>>
>> Hope this helps.
>>
>> thanks,
>>
>> Michele
>>
>>
>>
>>
>> On 2 May 2013 21:47, Brandon Jackson <usbrandon at gmail.com> wrote:
>>
>>>  The last day or so after upgrading to 4.8.1-EE, I have encountered a
>>> really interesting problem.  After running an OLAP Input step in PDI which
>>> uses XMLA to communicate back to the server, the mondrian logs say the
>>> following before the whole system becomes toast.  The BI Server shuts down
>>> because of 'out of memory', but the OS (Centos 6.3 with Oracle Java
>>> 1.6.0_45) also becomes unstable mentioning that it has the inability to
>>> fork new processes.  The system must be shutdown via power button at that
>>> point (ACPI is still alive and the OS shuts down normally, but nothing new
>>> can happen in the OS).
>>>
>>> 2013-05-02 14:02:31,718 ERROR [mondrian.xmla.XmlaServlet] Errors when
>>> handling XML/A message
>>> mondrian.xmla.XmlaException: Mondrian Error:XMLA Discover unparse
>>> results error
>>>     at mondrian.xmla.XmlaHandler.discover(XmlaHandler.java:2867)
>>>     at mondrian.xmla.XmlaHandler.process(XmlaHandler.java:663)
>>>     at
>>> mondrian.xmla.impl.DefaultXmlaServlet.handleSoapBody(DefaultXmlaServlet.java:505)
>>>     at mondrian.xmla.XmlaServlet.doPost(XmlaServlet.java:317)
>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
>>>     at
>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
>>>     at
>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>>>     at
>>> org.pentaho.platform.web.http.filters.PentahoWebContextFilter.doFilter(PentahoWebContextFilter.java:142)
>>>     at
>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
>>>     at
>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>>>     at
>>> org.pentaho.platform.web.http.filters.PentahoRequestContextFilter.doFilter(PentahoRequestContextFilter.java:84)
>>>     at
>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
>>>     at
>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>>>     at
>>> org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:378)
>>>     at
>>> org.springframework.security.intercept.web.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:109)
>>>     at
>>> org.springframework.security.intercept.web.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:83)
>>>     at
>>> org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:390)
>>>     at
>>> org.springframework.security.ui.ExceptionTranslationFilter.doFilterHttp(ExceptionTranslationFilter.java:101)
>>>     at
>>> org.springframework.security.ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53)
>>>     at
>>> org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:390)
>>>     at
>>> org.pentaho.platform.web.http.security.SecurityStartupFilter.doFilter(SecurityStartupFilter.java:103)
>>>     at
>>> org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:390)
>>>     at
>>> org.springframework.security.providers.anonymous.AnonymousProcessingFilter.doFilterHttp(AnonymousProcessingFilter.java:105)
>>>     at
>>> org.springframework.security.ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53)
>>>     at
>>> org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:390)
>>>     at
>>> org.pentaho.platform.web.http.security.RequestParameterAuthenticationFilter.doFilter(RequestParameterAuthenticationFilter.java:169)
>>>     at
>>> org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:390)
>>>     at
>>> org.springframework.security.ui.basicauth.BasicProcessingFilter.doFilterHttp(BasicProcessingFilter.java:174)
>>>     at
>>> org.springframework.security.ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53)
>>>     at
>>> org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:390)
>>>     at
>>> org.springframework.security.ui.AbstractProcessingFilter.doFilterHttp(AbstractProcessingFilter.java:278)
>>>     at
>>> org.springframework.security.ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53)
>>>     at
>>> org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:390)
>>>     at
>>> org.springframework.security.ui.logout.LogoutFilter.doFilterHttp(LogoutFilter.java:89)
>>>     at
>>> org.springframework.security.ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53)
>>>     at
>>> org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:390)
>>>     at
>>> org.pentaho.platform.web.http.security.HttpSessionReuseDetectionFilter.doFilter(HttpSessionReuseDetectionFilter.java:134)
>>>     at
>>> org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:390)
>>>     at
>>> org.springframework.security.context.HttpSessionContextIntegrationFilter.doFilterHttp(HttpSessionContextIntegrationFilter.java:235)
>>>     at
>>> org.springframework.security.ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53)
>>>     at
>>> org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:390)
>>>     at
>>> org.springframework.security.wrapper.SecurityContextHolderAwareRequestFilter.doFilterHttp(SecurityContextHolderAwareRequestFilter.java:91)
>>>     at
>>> org.springframework.security.ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53)
>>>     at
>>> org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:390)
>>>     at
>>> org.springframework.security.util.FilterChainProxy.doFilter(FilterChainProxy.java:175)
>>>     at
>>> org.springframework.security.util.FilterToBeanProxy.doFilter(FilterToBeanProxy.java:99)
>>>     at
>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
>>>     at
>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>>>     at com.pentaho.ui.servlet.SystemStatusFilter.doFilter(SourceFile:72)
>>>     at
>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
>>>     at
>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>>>     at
>>> org.pentaho.platform.web.http.filters.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:113)
>>>     at
>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
>>>     at
>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>>>     at
>>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
>>>     at
>>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
>>>     at
>>> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:470)
>>>     at
>>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
>>>     at
>>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
>>>     at
>>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
>>>     at
>>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
>>>     at
>>> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:857)
>>>     at
>>> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
>>>     at
>>> org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
>>>     at java.lang.Thread.run(Thread.java:662)
>>> Caused by: java.lang.OutOfMemoryError: unable to create new native thread
>>>     at java.lang.Thread.start0(Native Method)
>>>     at java.lang.Thread.start(Thread.java:640)
>>>     at java.util.Timer.<init>(Timer.java:154)
>>>     at
>>> mondrian.util.UtilCompatibleJdk15.newTimer(UtilCompatibleJdk15.java:129)
>>>     at mondrian.olap.Util.newTimer(Util.java:2196)
>>>     at
>>> mondrian.server.DynamicContentFinder.<init>(DynamicContentFinder.java:60)
>>>     at
>>> org.pentaho.platform.web.servlet.PentahoXmlaServlet$1.<init>(PentahoXmlaServlet.java:86)
>>>     at
>>> org.pentaho.platform.web.servlet.PentahoXmlaServlet.makeContentFinder(PentahoXmlaServlet.java:86)
>>>     at
>>> org.pentaho.platform.web.servlet.PentahoXmlaServlet$3.getConnection(PentahoXmlaServlet.java:254)
>>>     at mondrian.xmla.XmlaHandler.getConnection(XmlaHandler.java:2939)
>>>     at mondrian.xmla.XmlaHandler.getConnection(XmlaHandler.java:175)
>>>     at mondrian.xmla.Rowset.populate(Rowset.java:218)
>>>     at mondrian.xmla.Rowset.unparse(Rowset.java:193)
>>>     at mondrian.xmla.XmlaHandler.discover(XmlaHandler.java:2861)
>>>     ... 65 more
>>>
>>> _______________________________________________
>>> Mondrian mailing list
>>> Mondrian at pentaho.org
>>> http://lists.pentaho.org/mailman/listinfo/mondrian
>>>
>>>
>>
>> _______________________________________________
>> Mondrian mailing list
>> Mondrian at pentaho.org
>> http://lists.pentaho.org/mailman/listinfo/mondrian
>>
>>
>
> _______________________________________________
> Mondrian mailing list
> Mondrian at pentaho.org
> http://lists.pentaho.org/mailman/listinfo/mondrian
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.pentaho.org/pipermail/mondrian/attachments/20130502/8d6822c0/attachment-0001.html 


More information about the Mondrian mailing list