[Mondrian] Improvements for High Cardinality

Luis F. Canals luis.canals at stratebi.com
Tue Jan 15 11:59:39 EST 2008


Hi all,

as you may remember, we were working (since november 2007) to give
Mondrian capability to deal with high cardinality dimensions.
In this time, we have improved load process from database with the
capablity of executing several parallel threads to get information as
fast as databases are able to.

Ooook. Here, we have our first approach to the solution.
We have made a lot of changes, so we send attached a patch to be applied
to mondrian 2.4.2-9831.
Main modifications are:
    1.- changed Member[] to List<Member> (there is no memory and
performance impact and let us to move towards an iterable list)
    2.- add new boolean property to .xml describing schema:
highCardinality, for dimensions
    3.- add new properties into mondrian.properties to manage high
cardinality issues:
             mondrian.result.highCardChunkSize to indicate how many
cells are read at a the same time when highcardinality dimensions are
involved
             mondrian.rolap.MaximumParallelThreads to indicate how many
parallel threads are allowed to get results from an MDX query

There are many improvements on performance (basically because even for
low dimensions results needn't to be read and managed as a "big block"
causing the allocation and deallocation of big amounts of bytes).

Changes to API are described in patch file. (Aply it using:
       patch -p0 < diff)

Please, we are very interested in including these changes into the next
release.
If you have any question, send us. (javier.gimenez at stratebi.com,
jorge.lopez at stratebi.com, luis.canals at stratebi.com)
We have opened questions: for instance, which functions can be used with
high cardinality dimensions? (At this point, only "head" and "tail" are
fully implemented for high cardinality.) We will be very happy if you
find more!

Best regards,

- javier, jorge & luis
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: diffs
Url: http://lists.pentaho.org/pipermail/mondrian/attachments/20080115/443301bb/attachment.pl 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 252 bytes
Desc: OpenPGP digital signature
Url : http://lists.pentaho.org/pipermail/mondrian/attachments/20080115/443301bb/attachment.bin 


More information about the Mondrian mailing list