<div dir="ltr">In repeated query executions the result sets came back slower and slower.<div>The performance monitor reported the following 1, 5, 15 minute utilization:</div><div><br><div>15.54, 8.28, 4.12</div></div><div>
33.36, 13.31, 5.81</div><div>35.53, 14.54, 6.39</div><div>65.95, 26.31, 10.90</div><div><br></div><div><a href="http://screencast.com/t/bnY3ZJN7R">http://screencast.com/t/bnY3ZJN7R</a><br></div><div><br></div><div style>Java is the process pegging out this little dev box. The system does not appear to have freed up its resources after a few minutes.</div>
<div><br></div><div><br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, May 2, 2013 at 4:13 PM, Brandon Jackson <span dir="ltr"><<a href="mailto:usbrandon@gmail.com" target="_blank">usbrandon@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">I ran this version you recommended before changing operating system limits. I think the relevant limit is nprocs, which by default for the user was set to 1024.<div>
The server quickly crashed out under the same XMLA query. To circumvent building mondrian.jar myself, I stole the copy found in the workbench snapshot zip.</div>
<div><div><br></div><div>I changed the OS limits to the ones recommended by Rossi. We are testing continuously under the new limits to see where it will pop. By what you've said, it should still pop.</div><div><br>
</div>
<div>I'm not sure what the implication is with the timing of GC that you mentioned. Maybe these resource limits raising up will allow the computer to 'catch up' in a sense an prevent itself from going over a cliff. If that is not the case, then I would still consider this as still broken.</div>
<div><br></div><div>Please let me know what I can report back that would be insightful for solving the real problem.</div><div><br></div><div>Thanks for the pointers so far.</div><span class="HOEnZb"><font color="#888888"><div>
<br></div>
<div>Brandon</div><div><br></div><div><br><div><br></div></div></font></span></div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, May 2, 2013 at 3:59 PM, Luc Boudreau <span dir="ltr"><<a href="mailto:lucboudreau@gmail.com" target="_blank">lucboudreau@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div><div>The problem was that DynamicDataSourceServlet was creating a new DynamicContentFinder for each request creating a new Timer thread each time.</div>
<div><br></div>It was fixed today in this commit: <a href="https://github.com/pentaho/mondrian/commit/32e653b3264a7f6af4f9d26816dbf04d772148fb" target="_blank">https://github.com/pentaho/mondrian/commit/32e653b3264a7f6af4f9d26816dbf04d772148fb</a><br>
<br></div>This can be downloaded from CI at: <a href="http://ci.pentaho.com/view/Analysis/job/mondrian/lastSuccessfulBuild/artifact/dist/mondrian-TRUNK-SNAPSHOT.zip" target="_blank">http://ci.pentaho.com/view/Analysis/job/mondrian/lastSuccessfulBuild/artifact/dist/mondrian-TRUNK-SNAPSHOT.zip</a><br>
<br></div>It should be stable enough to be tested, but this is the bleeding edge of our development. You are warned :)<br><br></div><div>Limiting the number of threads at the OS level, as reported by Michele, only circumvents the issue. The ContentFinder implementations probably get garbage collected before the VM runs out of threads, but I wouldn't recommend this approach for multiple reasons, one being that the VM will slow down considerably because of all the threads reading the datasources file. The commit mentioned above actually fixed it by keeping a map of datasource path to finder object. It also cleans them up when the servlet shuts down.<span><font color="#888888"><br>
</font></span></div><span><font color="#888888"><div><br></div>Luc<br><div><div><div><br><br></div></div></div></font></span></div><div><div><div class="gmail_extra"><br><br><div class="gmail_quote">
On Thu, May 2, 2013 at 4:16 PM, Michele Rossi <span dir="ltr"><<a href="mailto:michele.rossi@gmail.com" target="_blank">michele.rossi@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><p style="line-height:17px;font-size:13px;font-family:Arial,FreeSans,Helvetica,sans-serif;margin:0px;padding:0px">
hi,</p><p style="line-height:17px;font-size:13px;font-family:Arial,FreeSans,Helvetica,sans-serif;margin:0px;padding:0px">
that happened to us too on a virtual Linux server and we solved it as described below:</p><p style="line-height:17px;font-size:13px;font-family:Arial,FreeSans,Helvetica,sans-serif;margin:0px;padding:0px">
<br></p><p style="line-height:17px;font-size:13px;font-family:Arial,FreeSans,Helvetica,sans-serif;margin:0px;padding:0px"><br></p><p style="line-height:17px;font-size:13px;font-family:Arial,FreeSans,Helvetica,sans-serif;margin:0px;padding:0px">
/etc/security/limits.conf</p><p style="line-height:17px;font-size:13px;font-family:Arial,FreeSans,Helvetica,sans-serif;margin:16px 0px 0px;padding:0px">exampleUsername soft nofile 65536<br>exampleUsername hard nofile 65536<br>
exampleUsername - nproc 2103295<br>exampleUsername hard core unlimited<br>exampleUsername soft core unlimited</p><p style="line-height:17px;font-size:13px;font-family:Arial,FreeSans,Helvetica,sans-serif;margin:16px 0px 0px;padding:0px">
<br></p><p style="line-height:17px;font-size:13px;font-family:Arial,FreeSans,Helvetica,sans-serif;margin:16px 0px 0px;padding:0px"><br></p><p style="line-height:17px;font-size:13px;font-family:Arial,FreeSans,Helvetica,sans-serif;margin:16px 0px 0px;padding:0px">
I am not 100% sure of which one of those settings fixed it, we left it to our sys admins to sort out.</p><p style="line-height:17px;font-size:13px;font-family:Arial,FreeSans,Helvetica,sans-serif;margin:16px 0px 0px;padding:0px">
<br></p><p style="line-height:17px;font-size:13px;font-family:Arial,FreeSans,Helvetica,sans-serif;margin:16px 0px 0px;padding:0px">I was able to prove that the problem was with the Linux machine set up by writing a simple Java program that just created Threads doing nothing but Thread.sleep(). The machine previously only allowed the creation of 100 threads and after that it started failing with the error you reported.</p>
<p style="line-height:17px;font-size:13px;font-family:Arial,FreeSans,Helvetica,sans-serif;margin:16px 0px 0px;padding:0px"><br></p><p style="line-height:17px;font-size:13px;font-family:Arial,FreeSans,Helvetica,sans-serif;margin:16px 0px 0px;padding:0px">
Hope this helps.</p><p style="line-height:17px;font-size:13px;font-family:Arial,FreeSans,Helvetica,sans-serif;margin:16px 0px 0px;padding:0px">thanks,</p><p style="line-height:17px;font-size:13px;font-family:Arial,FreeSans,Helvetica,sans-serif;margin:16px 0px 0px;padding:0px">
Michele</p><p style="line-height:17px;font-size:13px;font-family:Arial,FreeSans,Helvetica,sans-serif;margin:16px 0px 0px;padding:0px"><br></p></div><div class="gmail_extra"><br><br><div class="gmail_quote"><div><div>
On 2 May 2013 21:47, Brandon Jackson <span dir="ltr"><<a href="mailto:usbrandon@gmail.com" target="_blank">usbrandon@gmail.com</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div><div>
<div dir="ltr">The last day or so after upgrading to 4.8.1-EE, I have encountered a really interesting problem. After running an OLAP Input step in PDI which uses XMLA to communicate back to the server, the mondrian logs say the following before the whole system becomes toast. The BI Server shuts down because of 'out of memory', but the OS (Centos 6.3 with Oracle Java 1.6.0_45) also becomes unstable mentioning that it has the inability to fork new processes. The system must be shutdown via power button at that point (ACPI is still alive and the OS shuts down normally, but nothing new can happen in the OS).<br>
<br>2013-05-02 14:02:31,718 ERROR [mondrian.xmla.XmlaServlet] Errors when handling XML/A message<br>mondrian.xmla.XmlaException: Mondrian Error:XMLA Discover unparse results error<br> at mondrian.xmla.XmlaHandler.discover(XmlaHandler.java:2867)<br>
at mondrian.xmla.XmlaHandler.process(XmlaHandler.java:663)<br> at mondrian.xmla.impl.DefaultXmlaServlet.handleSoapBody(DefaultXmlaServlet.java:505)<br> at mondrian.xmla.XmlaServlet.doPost(XmlaServlet.java:317)<br>
at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)<br> at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)<br> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)<br>
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)<br> at org.pentaho.platform.web.http.filters.PentahoWebContextFilter.doFilter(PentahoWebContextFilter.java:142)<br> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)<br>
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)<br> at org.pentaho.platform.web.http.filters.PentahoRequestContextFilter.doFilter(PentahoRequestContextFilter.java:84)<br>
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)<br> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)<br> at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:378)<br>
at org.springframework.security.intercept.web.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:109)<br> at org.springframework.security.intercept.web.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:83)<br>
at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:390)<br> at org.springframework.security.ui.ExceptionTranslationFilter.doFilterHttp(ExceptionTranslationFilter.java:101)<br>
at org.springframework.security.ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53)<br> at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:390)<br> at org.pentaho.platform.web.http.security.SecurityStartupFilter.doFilter(SecurityStartupFilter.java:103)<br>
at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:390)<br> at org.springframework.security.providers.anonymous.AnonymousProcessingFilter.doFilterHttp(AnonymousProcessingFilter.java:105)<br>
at org.springframework.security.ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53)<br> at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:390)<br> at org.pentaho.platform.web.http.security.RequestParameterAuthenticationFilter.doFilter(RequestParameterAuthenticationFilter.java:169)<br>
at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:390)<br> at org.springframework.security.ui.basicauth.BasicProcessingFilter.doFilterHttp(BasicProcessingFilter.java:174)<br>
at org.springframework.security.ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53)<br> at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:390)<br> at org.springframework.security.ui.AbstractProcessingFilter.doFilterHttp(AbstractProcessingFilter.java:278)<br>
at org.springframework.security.ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53)<br> at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:390)<br> at org.springframework.security.ui.logout.LogoutFilter.doFilterHttp(LogoutFilter.java:89)<br>
at org.springframework.security.ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53)<br> at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:390)<br> at org.pentaho.platform.web.http.security.HttpSessionReuseDetectionFilter.doFilter(HttpSessionReuseDetectionFilter.java:134)<br>
at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:390)<br> at org.springframework.security.context.HttpSessionContextIntegrationFilter.doFilterHttp(HttpSessionContextIntegrationFilter.java:235)<br>
at org.springframework.security.ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53)<br> at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:390)<br> at org.springframework.security.wrapper.SecurityContextHolderAwareRequestFilter.doFilterHttp(SecurityContextHolderAwareRequestFilter.java:91)<br>
at org.springframework.security.ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53)<br> at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:390)<br> at org.springframework.security.util.FilterChainProxy.doFilter(FilterChainProxy.java:175)<br>
at org.springframework.security.util.FilterToBeanProxy.doFilter(FilterToBeanProxy.java:99)<br> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)<br> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)<br>
at com.pentaho.ui.servlet.SystemStatusFilter.doFilter(SourceFile:72)<br> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)<br> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)<br>
at org.pentaho.platform.web.http.filters.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:113)<br> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)<br>
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)<br> at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)<br> at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)<br>
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:470)<br> at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)<br> at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)<br>
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)<br> at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)<br> at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:857)<br>
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)<br> at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)<br> at java.lang.Thread.run(Thread.java:662)<br>
Caused by: java.lang.OutOfMemoryError: unable to create new native thread<br> at java.lang.Thread.start0(Native Method)<br> at java.lang.Thread.start(Thread.java:640)<br> at java.util.Timer.<init>(Timer.java:154)<br>
at mondrian.util.UtilCompatibleJdk15.newTimer(UtilCompatibleJdk15.java:129)<br> at mondrian.olap.Util.newTimer(Util.java:2196)<br> at mondrian.server.DynamicContentFinder.<init>(DynamicContentFinder.java:60)<br>
at org.pentaho.platform.web.servlet.PentahoXmlaServlet$1.<init>(PentahoXmlaServlet.java:86)<br> at org.pentaho.platform.web.servlet.PentahoXmlaServlet.makeContentFinder(PentahoXmlaServlet.java:86)<br> at org.pentaho.platform.web.servlet.PentahoXmlaServlet$3.getConnection(PentahoXmlaServlet.java:254)<br>
at mondrian.xmla.XmlaHandler.getConnection(XmlaHandler.java:2939)<br> at mondrian.xmla.XmlaHandler.getConnection(XmlaHandler.java:175)<br> at mondrian.xmla.Rowset.populate(Rowset.java:218)<br> at mondrian.xmla.Rowset.unparse(Rowset.java:193)<br>
at mondrian.xmla.XmlaHandler.discover(XmlaHandler.java:2861)<br> ... 65 more<br></div>
<br></div></div><div>_______________________________________________<br>
Mondrian mailing list<br>
<a href="mailto:Mondrian@pentaho.org" target="_blank">Mondrian@pentaho.org</a><br>
<a href="http://lists.pentaho.org/mailman/listinfo/mondrian" target="_blank">http://lists.pentaho.org/mailman/listinfo/mondrian</a><br>
<br></div></blockquote></div><br></div>
<br>_______________________________________________<br>
Mondrian mailing list<br>
<a href="mailto:Mondrian@pentaho.org" target="_blank">Mondrian@pentaho.org</a><br>
<a href="http://lists.pentaho.org/mailman/listinfo/mondrian" target="_blank">http://lists.pentaho.org/mailman/listinfo/mondrian</a><br>
<br></blockquote></div><br></div>
</div></div><br>_______________________________________________<br>
Mondrian mailing list<br>
<a href="mailto:Mondrian@pentaho.org" target="_blank">Mondrian@pentaho.org</a><br>
<a href="http://lists.pentaho.org/mailman/listinfo/mondrian" target="_blank">http://lists.pentaho.org/mailman/listinfo/mondrian</a><br>
<br></blockquote></div><br></div>
</div></div></blockquote></div><br></div>