[Mondrian] Multi-threading SQL execution

Julian Hyde julianhyde at speakeasy.net
Tue Feb 13 19:00:55 EST 2007


I agree with the sentiment regarding threads. If we implement #2 (parallel
SQL query execution) I want mondrian to be able to run in a mode where
everything is done in one thread.
 
(If you read my blog post What's
<http://julianhyde.blogspot.com/2006/12/its-not-often-that-i-am-criticized-f
or.html> in a name, you will know that I am militantly agnostic on whether
mondrian is a server. A repository is one sure sign that a software library
is growing into a server, but a thread pool is another.) 
 
When we do allocate threads, we will use java.util.concurrent.ThreadFactory
- that should give enough flexibility for various environments. (I have a
feeling that ThreadFactory was introduced because people wanted a safe way
to create threads inside their EJBs, so we can use if for the same purpose.)
 
I want to challenge Michael's assertion that "almost no JDBC driver will
allow parallel queries". It may be true, but if so, it will cost us a lot of
effort to do things another way, so I want to see some evidence. Michael,
can you give an example of a JDBC driver which cannot execute two statements
in the same connection (and transaction context)?
 
Julian 
 
 


  _____  

From: mondrian-bounces at pentaho.org [mailto:mondrian-bounces at pentaho.org] On
Behalf Of michael bienstein
Sent: Tuesday, February 13, 2007 2:08 PM
To: Mondrian developer mailing list
Subject: Re : [Mondrian] Multi-threading SQL execution


Matt,

The reason that threads shouldn't be created in Servlets is conceptually
simple.  Servlets run in a managed environment.  The control over the usage
of system resources should be handled by the environment without
interference from applications being managed.  Threads are one of those
resources.  To give an idea of this, there is actually no restriction on the
number of Threads you create in Java.  However if you go over 8 threads per
CPU you get serious performance degradation because of the switching the CPU
has to do between the threads (at least on Windows Sun JDK 1.4).  So it's
better if you set up your application server and configure how many threads
get used for all the applications in the same JVM and no application changes
this.

More generally, Java has very bad semantics for enforcing this sort of
control over system resources.  Consider memory.  I didn't know that Java 5
allowed some control over this as Richard has shown us recently.  Look at
the problems though: Derby runs in the same JVM as Mondrian and all of a
sudden we get lots of failures because we are told that we're running out of
memory.  How can you limit how much memory a WAR can use?  You can't.  With
File access you probably could do it with a custom SecurityManager.  With
Threads that's probably true too, but no major app server does it.

The second reason has nothing to do with Mondrian.  EJB 1 and 2 focussed on
being able to call business logic without worrying about where that logic
was implemented.  It could be in a different JVM or even on another
computer.  Using threads (or static variables) is therefore forbidden even
though if you know you are only running locally then it will still all work.

Lastly, I know commercial BI software that *breaks* this rule and launches a
thread in a Servlet init() method.  It is truly astounding how many bad
things pop up when the language or framework doesn't constrain you.

Michael

PS: Congrats on 2.3 feature complete.  Really happy to see cache flushing in
there.  But did Bart's bug on virtual cube get fixed?  I'm really curious as
to what causes it.


  _____  

Découvrez une nouvelle façon d'obtenir des réponses à toutes vos questions !
Profitez des connaissances, des opinions et des expériences des internautes
sur Yahoo!  <http://fr.rd.yahoo.com/evt=42054/*http://fr.answers.yahoo.com>
Questions/Réponses.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.pentaho.org/pipermail/mondrian/attachments/20070213/e7108d03/attachment.html 


More information about the Mondrian mailing list