[Mondrian] improving schema load concurrency

Julian Hyde julianhyde at gmail.com
Fri Jul 15 14:00:58 EDT 2016


Can you describe why schemas are taking so long to load? Maybe we can defer some parts of the load process.

Replacing an actor (which single-threads critical processing) with a parallel process is never going to be straightforward.

Another approach is to keep the actor but reduce the amount of processing that occurs in the actor. Which may end up looking like the suggestion I made in the first paragraph.

Julian

> On Jul 15, 2016, at 10:51 AM, Wright, Jeff <jeff.s.wright at truvenhealth.com> wrote:
> 
> When we were working on our recent deadlock issue, we had initially hoped that the change last year in RolapSchemaPool to use a ReentrantReadWriteLock might fix that problem. Although it turned out to require a different fix, my colleague Jon Rand that was working on this came up with an idea about how to possibly allow even more concurrency with schema loading.
>  
> The basic proposition is to have the two maps that maintain the schema pool store <Future<RolapSchema>> objects rather than storing completed <RolapSchema> objects.  The Future can be added and the lock released before the schema even begins to load.
>  
> Jon actually did prototypes of 2 different approaches, here’s how he described them:
>  
> 1.       Schema loading occurs in a thread managed by an ExecutorService.  This makes it easy to control the maximum number of schemas that can load concurrently simply by limiting the size of the thread pool managed by the ExecutorService.  I’ve run a few reports with this version but no extensive testing has been performed.
> 2.       Schema loading occurs in the thread that’s calling RolapSchemaPool.get() using a FutureTask.  I haven’t had a chance to test this version.
>  
> One thing that appeals to me a lot about this is I think this would also reduce some issues with the SegmentCacheManager$ACTOR thread running schema loads, since this can be a long running process (minutes), and holds up the SegmentCacheManager.
>  
> Any thoughts on this, would this be worth capturing in JIRA? I’m not sure I can get anybody from our team to work on this in the short term, but I wanted to pass along the idea…
>  
> --jeff
>  
> _______________________________________________
> Mondrian mailing list
> Mondrian at pentaho.org <mailto:Mondrian at pentaho.org>
> http://lists.pentaho.org/mailman/listinfo/mondrian <http://lists.pentaho.org/mailman/listinfo/mondrian>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.pentaho.org/pipermail/mondrian/attachments/20160715/218abe7f/attachment-0001.html 


More information about the Mondrian mailing list