[Mondrian] XMLA servlet running reporting out of memory, biserver then crashes, OS becomes unresponsive.
Brandon Jackson
usbrandon at gmail.com
Thu May 2 17:21:15 EDT 2013
In repeated query executions the result sets came back slower and slower.
The performance monitor reported the following 1, 5, 15 minute utilization:
15.54, 8.28, 4.12
33.36, 13.31, 5.81
35.53, 14.54, 6.39
65.95, 26.31, 10.90
http://screencast.com/t/bnY3ZJN7R
Java is the process pegging out this little dev box. The system does not
appear to have freed up its resources after a few minutes.
On Thu, May 2, 2013 at 4:13 PM, Brandon Jackson <usbrandon at gmail.com> wrote:
> I ran this version you recommended before changing operating system
> limits. I think the relevant limit is nprocs, which by default for the
> user was set to 1024.
> The server quickly crashed out under the same XMLA query. To circumvent
> building mondrian.jar myself, I stole the copy found in the workbench
> snapshot zip.
>
> I changed the OS limits to the ones recommended by Rossi. We are testing
> continuously under the new limits to see where it will pop. By what you've
> said, it should still pop.
>
> I'm not sure what the implication is with the timing of GC that you
> mentioned. Maybe these resource limits raising up will allow the computer
> to 'catch up' in a sense an prevent itself from going over a cliff. If
> that is not the case, then I would still consider this as still broken.
>
> Please let me know what I can report back that would be insightful for
> solving the real problem.
>
> Thanks for the pointers so far.
>
> Brandon
>
>
>
>
>
> On Thu, May 2, 2013 at 3:59 PM, Luc Boudreau <lucboudreau at gmail.com>wrote:
>
>> The problem was that DynamicDataSourceServlet was creating a new
>> DynamicContentFinder for each request creating a new Timer thread each time.
>>
>> It was fixed today in this commit:
>> https://github.com/pentaho/mondrian/commit/32e653b3264a7f6af4f9d26816dbf04d772148fb
>>
>> This can be downloaded from CI at:
>> http://ci.pentaho.com/view/Analysis/job/mondrian/lastSuccessfulBuild/artifact/dist/mondrian-TRUNK-SNAPSHOT.zip
>>
>> It should be stable enough to be tested, but this is the bleeding edge of
>> our development. You are warned :)
>>
>> Limiting the number of threads at the OS level, as reported by Michele,
>> only circumvents the issue. The ContentFinder implementations probably get
>> garbage collected before the VM runs out of threads, but I wouldn't
>> recommend this approach for multiple reasons, one being that the VM will
>> slow down considerably because of all the threads reading the datasources
>> file. The commit mentioned above actually fixed it by keeping a map of
>> datasource path to finder object. It also cleans them up when the servlet
>> shuts down.
>>
>> Luc
>>
>>
>>
>>
>> On Thu, May 2, 2013 at 4:16 PM, Michele Rossi <michele.rossi at gmail.com>wrote:
>>
>>> hi,
>>>
>>> that happened to us too on a virtual Linux server and we solved it as
>>> described below:
>>>
>>>
>>>
>>> /etc/security/limits.conf
>>>
>>> exampleUsername soft nofile 65536
>>> exampleUsername hard nofile 65536
>>> exampleUsername - nproc 2103295
>>> exampleUsername hard core unlimited
>>> exampleUsername soft core unlimited
>>>
>>>
>>>
>>> I am not 100% sure of which one of those settings fixed it, we left it
>>> to our sys admins to sort out.
>>>
>>>
>>> I was able to prove that the problem was with the Linux machine set up
>>> by writing a simple Java program that just created Threads doing nothing
>>> but Thread.sleep(). The machine previously only allowed the creation of 100
>>> threads and after that it started failing with the error you reported.
>>>
>>>
>>> Hope this helps.
>>>
>>> thanks,
>>>
>>> Michele
>>>
>>>
>>>
>>>
>>> On 2 May 2013 21:47, Brandon Jackson <usbrandon at gmail.com> wrote:
>>>
>>>> The last day or so after upgrading to 4.8.1-EE, I have encountered a
>>>> really interesting problem. After running an OLAP Input step in PDI which
>>>> uses XMLA to communicate back to the server, the mondrian logs say the
>>>> following before the whole system becomes toast. The BI Server shuts down
>>>> because of 'out of memory', but the OS (Centos 6.3 with Oracle Java
>>>> 1.6.0_45) also becomes unstable mentioning that it has the inability to
>>>> fork new processes. The system must be shutdown via power button at that
>>>> point (ACPI is still alive and the OS shuts down normally, but nothing new
>>>> can happen in the OS).
>>>>
>>>> 2013-05-02 14:02:31,718 ERROR [mondrian.xmla.XmlaServlet] Errors when
>>>> handling XML/A message
>>>> mondrian.xmla.XmlaException: Mondrian Error:XMLA Discover unparse
>>>> results error
>>>> at mondrian.xmla.XmlaHandler.discover(XmlaHandler.java:2867)
>>>> at mondrian.xmla.XmlaHandler.process(XmlaHandler.java:663)
>>>> at
>>>> mondrian.xmla.impl.DefaultXmlaServlet.handleSoapBody(DefaultXmlaServlet.java:505)
>>>> at mondrian.xmla.XmlaServlet.doPost(XmlaServlet.java:317)
>>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
>>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
>>>> at
>>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
>>>> at
>>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>>>> at
>>>> org.pentaho.platform.web.http.filters.PentahoWebContextFilter.doFilter(PentahoWebContextFilter.java:142)
>>>> at
>>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
>>>> at
>>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>>>> at
>>>> org.pentaho.platform.web.http.filters.PentahoRequestContextFilter.doFilter(PentahoRequestContextFilter.java:84)
>>>> at
>>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
>>>> at
>>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>>>> at
>>>> org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:378)
>>>> at
>>>> org.springframework.security.intercept.web.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:109)
>>>> at
>>>> org.springframework.security.intercept.web.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:83)
>>>> at
>>>> org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:390)
>>>> at
>>>> org.springframework.security.ui.ExceptionTranslationFilter.doFilterHttp(ExceptionTranslationFilter.java:101)
>>>> at
>>>> org.springframework.security.ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53)
>>>> at
>>>> org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:390)
>>>> at
>>>> org.pentaho.platform.web.http.security.SecurityStartupFilter.doFilter(SecurityStartupFilter.java:103)
>>>> at
>>>> org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:390)
>>>> at
>>>> org.springframework.security.providers.anonymous.AnonymousProcessingFilter.doFilterHttp(AnonymousProcessingFilter.java:105)
>>>> at
>>>> org.springframework.security.ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53)
>>>> at
>>>> org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:390)
>>>> at
>>>> org.pentaho.platform.web.http.security.RequestParameterAuthenticationFilter.doFilter(RequestParameterAuthenticationFilter.java:169)
>>>> at
>>>> org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:390)
>>>> at
>>>> org.springframework.security.ui.basicauth.BasicProcessingFilter.doFilterHttp(BasicProcessingFilter.java:174)
>>>> at
>>>> org.springframework.security.ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53)
>>>> at
>>>> org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:390)
>>>> at
>>>> org.springframework.security.ui.AbstractProcessingFilter.doFilterHttp(AbstractProcessingFilter.java:278)
>>>> at
>>>> org.springframework.security.ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53)
>>>> at
>>>> org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:390)
>>>> at
>>>> org.springframework.security.ui.logout.LogoutFilter.doFilterHttp(LogoutFilter.java:89)
>>>> at
>>>> org.springframework.security.ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53)
>>>> at
>>>> org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:390)
>>>> at
>>>> org.pentaho.platform.web.http.security.HttpSessionReuseDetectionFilter.doFilter(HttpSessionReuseDetectionFilter.java:134)
>>>> at
>>>> org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:390)
>>>> at
>>>> org.springframework.security.context.HttpSessionContextIntegrationFilter.doFilterHttp(HttpSessionContextIntegrationFilter.java:235)
>>>> at
>>>> org.springframework.security.ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53)
>>>> at
>>>> org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:390)
>>>> at
>>>> org.springframework.security.wrapper.SecurityContextHolderAwareRequestFilter.doFilterHttp(SecurityContextHolderAwareRequestFilter.java:91)
>>>> at
>>>> org.springframework.security.ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53)
>>>> at
>>>> org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:390)
>>>> at
>>>> org.springframework.security.util.FilterChainProxy.doFilter(FilterChainProxy.java:175)
>>>> at
>>>> org.springframework.security.util.FilterToBeanProxy.doFilter(FilterToBeanProxy.java:99)
>>>> at
>>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
>>>> at
>>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>>>> at com.pentaho.ui.servlet.SystemStatusFilter.doFilter(SourceFile:72)
>>>> at
>>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
>>>> at
>>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>>>> at
>>>> org.pentaho.platform.web.http.filters.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:113)
>>>> at
>>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
>>>> at
>>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>>>> at
>>>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
>>>> at
>>>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
>>>> at
>>>> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:470)
>>>> at
>>>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
>>>> at
>>>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
>>>> at
>>>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
>>>> at
>>>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
>>>> at
>>>> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:857)
>>>> at
>>>> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
>>>> at
>>>> org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
>>>> at java.lang.Thread.run(Thread.java:662)
>>>> Caused by: java.lang.OutOfMemoryError: unable to create new native
>>>> thread
>>>> at java.lang.Thread.start0(Native Method)
>>>> at java.lang.Thread.start(Thread.java:640)
>>>> at java.util.Timer.<init>(Timer.java:154)
>>>> at
>>>> mondrian.util.UtilCompatibleJdk15.newTimer(UtilCompatibleJdk15.java:129)
>>>> at mondrian.olap.Util.newTimer(Util.java:2196)
>>>> at
>>>> mondrian.server.DynamicContentFinder.<init>(DynamicContentFinder.java:60)
>>>> at
>>>> org.pentaho.platform.web.servlet.PentahoXmlaServlet$1.<init>(PentahoXmlaServlet.java:86)
>>>> at
>>>> org.pentaho.platform.web.servlet.PentahoXmlaServlet.makeContentFinder(PentahoXmlaServlet.java:86)
>>>> at
>>>> org.pentaho.platform.web.servlet.PentahoXmlaServlet$3.getConnection(PentahoXmlaServlet.java:254)
>>>> at mondrian.xmla.XmlaHandler.getConnection(XmlaHandler.java:2939)
>>>> at mondrian.xmla.XmlaHandler.getConnection(XmlaHandler.java:175)
>>>> at mondrian.xmla.Rowset.populate(Rowset.java:218)
>>>> at mondrian.xmla.Rowset.unparse(Rowset.java:193)
>>>> at mondrian.xmla.XmlaHandler.discover(XmlaHandler.java:2861)
>>>> ... 65 more
>>>>
>>>> _______________________________________________
>>>> Mondrian mailing list
>>>> Mondrian at pentaho.org
>>>> http://lists.pentaho.org/mailman/listinfo/mondrian
>>>>
>>>>
>>>
>>> _______________________________________________
>>> Mondrian mailing list
>>> Mondrian at pentaho.org
>>> http://lists.pentaho.org/mailman/listinfo/mondrian
>>>
>>>
>>
>> _______________________________________________
>> Mondrian mailing list
>> Mondrian at pentaho.org
>> http://lists.pentaho.org/mailman/listinfo/mondrian
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.pentaho.org/pipermail/mondrian/attachments/20130502/a53da12d/attachment-0001.html
More information about the Mondrian
mailing list