[Mondrian] hierarchical sorting
thejoe at gmail.com
Tue Jan 11 17:37:13 EST 2011
On Tue, Jan 11, 2011 at 12:55 PM, Julian Hyde <jhyde at pentaho.com> wrote:
> Thinking about the sorting performance issues you are trying to solve.
> Are a lot of your sorts hierarchical (i.e. use ASC or DESC flags, as opposed
> to BASC and BDESC)? I am thinking that we have been doing hierarchical sorts
> inefficiently. If two members in the list are not siblings, we should even
> try to sort them; we should just leave them in the incoming order. But I
> think the current algorithm spends a significant amount of our time figuring
> out whether members are ancestors, siblings, cousins etc. and this is wasted
> effort. Plus it leads to sorting one big list as opposed to a number of much
> smaller lists.
> If your sorts use BASC and BDESC (or indeed TopCount etc.) then ignore -- we
> do have to compare all members.
Unfortunately, all our sorts are Top/Bottom counts (and were
Order(set, measure, BASC/BDESC) before we started using those
functions), so optimizing the hierarchical sort won't help for our
queries, at least.
A not-really related optimization (but the word hierarchical reminds
me of it) that we have looked at is trying to optimize away the
Hierarchize() call in
FunUtil.levelMembers()/FunUtil.hierarchyMembers() by having the member
readers hierarchize as they read. We actually ran with a version of
that against 2.4 for a while. When porting our changes over to 3.2,
we left it out, due to noticing that:
a) caused _lots_ of tests to fail
b) didn't actually affect overall query times, despite profilers
showing a significant portion of time being spent in Hierarchize()...
c) the implementation we threw together for highCardinalityDimensions
was unlikely to perform well at all
due to (b), didn't really look into (a) or (c) in a lot of detail, but
still seems like there should be a way to save work by hierarchizing
when loading the members from the db, rather than every time
Level/Hierarchy.Members is accessed...
Anyways, just one more thing to possibly consider...
More information about the Mondrian