[Mondrian] Multithreading, Parallel Batch / Query Execution

Tushar Apshankar tapshank at thoughtworks.com
Fri Apr 27 10:04:20 EDT 2007


Hi,
To achieve Parallel Batch / Query Execution, we need to give execution 
tasks to separate threads.
We could think of 2 options to achieve it. Both of them have their pros 
and cons in terms of performance and ThreadLocal variable management.

1) New Thread Creation:
        Pros: We change the thread local variable to inheritable thread 
local and the child thread automatically get the thread local values of 
the parent thread.
        Cons: Creating new threads is not a good idea. Too many threads 
running in a system should not be a problem though, as we can limit the 
number of Threads created using java.util.concurrent.Semaphore/BigInteger 
etc.
2) Thread Pool:
        Pros: It would be faster (It looks like a valid assumption.)
        Cons:  We will have to do some manipulation to pass the thread 
local variables to worker threads of Thread Pool. All ThreadLocal 
variables of a Thread are stored in a ThreadLocalMap. Thread class does 
not provide getter/setter for ThreadLocalMap. We could still get/set the 
ThreadLocalMap using reflection. This solution is working fine.

Is there any other option? If not, which one of the above looks better 
approach to you?

Please let us know your thoughts.
Thanks and regards,
Tushar
 





Tushar Apshankar <tapshank at thoughtworks.com> 
Sent by: mondrian-bounces at pentaho.org
04/25/2007 06:27 PM
Please respond to
Mondrian developer mailing list <mondrian at pentaho.org>


To
Mondrian developer mailing list <mondrian at pentaho.org>
cc

Subject
Re : [Mondrian] Multithreading, Parallel Batch / Query Execution







Hi, 

To improve performance, we started looking at ReentrantReadWriteLock and 
ConcurrentHashMap (from java.util.concurrent) package. 
We are also looking at ThreadPoolExecutor and related classes to implment 
parallel query / batch execution (most of these classes are also available 
in java.util.concurrent as well as backport jar). 
Since compatibility with Java 1.4 is essential, we started looking at the 
build script retroweave target that uses Retroweaver 1.2.4. 

Retroweaver without our changes works fine. We ran into problems while 
using the current retroweaver version and build target after our changes. 

We have a simple class LockTest to reproduce the problem: 
============================================================================================= 

package mondrian; 

import java.util.concurrent.ConcurrentHashMap; 
import java.util.concurrent.locks.ReentrantReadWriteLock; 
import java.util.concurrent.locks.Lock; 

public class LockTest { 
    ConcurrentHashMap<String, String> cache = new 
ConcurrentHashMap<String, String>(); 
    ReentrantReadWriteLock lock; 
    Lock r ; 
    Lock w ; 
    public LockTest() 
    { 
        cache = new ConcurrentHashMap(); 
        lock = new ReentrantReadWriteLock(); 
        System.out.println("Before lock = " + lock); 
        w = lock.writeLock(); 
        r = lock.readLock(); 
        System.out.println("After lock = " + lock); 
    } 

    public static void main(String[] args) { 
        LockTest lt = new LockTest(); 
        System.out.println("i am inside main method"); 
    } 
} 
============================================================================================= 

When we retroweave it with retroweaver 1.2.4 (using ANT as well as 
retroweaver UI) and run the weaved class using Java 1.4.2, we get 
java.lang.NoSuchMethodError. Stack is: 

java.lang.NoSuchMethodError: 
edu.emory.mathcs.backport.java.util.concurrent.locks.ReentrantReadWriteLock.writeLock() 

Ledu/emory/mathcs/backport/java/util/concurrent/locks/ReentrantReadWriteLock$WriteLock; 

[Loaded java.lang.StackTraceElement from C:\j2sdk1.4.2_14\jre\lib\rt.jar] 
        at mondrian.LockTest.<init>(LockTest.java:26) 
        at mondrian.LockTest.main(LockTest.java:34): 

We have backport jar in our class path (please note that we are NOT 
getting ClassNotFoundExcecption). 
Did someone face this issue earlier? Please let us know if we are missing 
something here. 

We also tried out retroweaver 2.0 without success. 

As an alternative, we tried Retrotranslator(
http://retrotranslator.sourceforge.net) and everything worked in first 
attempt. We propose to create a new target for Retrotranslator and add the 
related libraries. Please let us know your thoughts on this 

Thanks and regards, 
Tushar, Ajit 
_______________________________________________
Mondrian mailing list
Mondrian at pentaho.org
http://lists.pentaho.org/mailman/listinfo/mondrian

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.pentaho.org/pipermail/mondrian/attachments/20070427/460e5ffa/attachment.html 


More information about the Mondrian mailing list