[Mondrian] improving schema load concurrency

[Wright, Jeff]

When we were working on our recent deadlock issue, we had initially hoped that the change last year in RolapSchemaPool to use a ReentrantReadWriteLock might fix that problem. Although it turned out to require a different fix, my colleague Jon Rand that was working on this came up with an idea about how to possibly allow even more concurrency with schema loading.

The basic proposition is to have the two maps that maintain the schema pool store <Future<RolapSchema>> objects rather than storing completed <RolapSchema> objects.  The Future can be added and the lock released before the schema even begins to load.

Jon actually did prototypes of 2 different approaches, here's how he described them:


1.       Schema loading occurs in a thread managed by an ExecutorService.  This makes it easy to control the maximum number of schemas that can load concurrently simply by limiting the size of the thread pool managed by the ExecutorService.  I've run a few reports with this version but no extensive testing has been performed.

2.       Schema loading occurs in the thread that's calling RolapSchemaPool.get() using a FutureTask.  I haven't had a chance to test this version.

One thing that appeals to me a lot about this is I think this would also reduce some issues with the SegmentCacheManager$ACTOR thread running schema loads, since this can be a long running process (minutes), and holds up the SegmentCacheManager.

Any thoughts on this, would this be worth capturing in JIRA? I'm not sure I can get anybody from our team to work on this in the short term, but I wanted to pass along the idea...

--jeff

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.pentaho.org/pipermail/mondrian/attachments/20160715/7507067e/attachment.html