[Mondrian] RE: Test failures with high-cardinality dimensions

Julian Hyde jhyde at pentaho.com
Mon Jul 14 16:10:34 EDT 2008


Looks like we have very different opinions on this issue. The way I see it,
we have a half implemented feature. I think that most mondrian users would
see it that way too.
 
(I'd like to hear about other developers on the list feel about this.)
 
My decision principle in mondrian has been orthogonality: every feature has
to work with every other feature. It's not possible for feature X to work in
environment Y, then the implementation should fall back to the basic
implementation which works, and things are no worse than they would have
been without feature X.
 
Looking at the whole picture, of end-user perceptions, maintainability, and
simplicity, I would in general rather leave out a feature than have it half
implemented.
 
The mondrian.test.highCardDimensions flag is a neat trick to ensure
orthogonality. It shows up losts of test failures because high-cardinality
dimensions are partially implemented. The solution is not to remove the
mondrian.test.highCardDimensions test, but to fix (or remove)
high-cardinality dimensions support. You appeal for the community to help
implement high-cardinality support, but I don't see that happening, at least
not in the time scales we need.
 
One idea that may help us fix it. There's a lot in common between
high-cardinality dimensions and the ITERABLE result style (see
http://mondrian.pentaho.org/api/src-html/mondrian/calc/ResultStyle.html),
which causes functions to return their results as iterators rather than
lists. Richard Emberson did a lot of work on that feature, and it is now a
fully orthogonal part of the architecture; if a function isn't implemented
in the ITERABLE style, mondrian creates an adapter to implement it in the
LIST style. In other words, the system fails gracefull. Every so often,
someone contributes an implementation of another function in ITERABLE style.
 
Is there a way to leverage this in order to get high-cardinality dimensions
to the status of a fully-implemented feature?
 
Julian


  _____  

From: Luis F. Canals [mailto:luisf.canals at gmail.com] On Behalf Of Luis F.
Canals
Sent: Sunday, July 13, 2008 11:47 PM
To: jhyde at pentaho.com
Cc: jorge.lopez at stratebi.com; javier.gimenez at stratebi.com;
mondrian at pentaho.org
Subject: Re: Test failures with high-cardinality dimensions


Hi all,

about this question, problem arises when using functions with high
cardinality dimensions not implemented (or not implementable) for this kind
of dimensions.
Since not every function or query will work for high cardinality dimensions,
I think it's not a good idea to use "mondrian.test.highCardDimensions=...".
The solution would be to use mondrian/rolap/HighDimensionsTest (or simiar)
and include any tests related with high dimensions into it instead of this
kind of "blind" tests for all functions.

So the first thing to do is: move from testing every functions against high
card. dimensions to detailed tests function by function for suitable ones.

The key question here is the limitations for the meaning of "high cardility"
feature: high cardinality dimensions are able to be managed. But the use of
functions with these kind of dimensions can't be equivalent to usual
dimensions. Mainly, all functions that require the whole set of elements to
produce a result will not work for high cardinality dimensions (obviously).

At this time, we have no human sructure to refactor everything and implement
all possible functions to work with high dimension property (Mondrian is a
very big project). 
It's a continuous task, open for everyone, ans sould be done by everyone.

The first (and most difficult) step has been done: Mondrian can read high
cardinality dimensions; the second step (implement all suitable functions to
work for high cardimality dimensions) is drafted, with "head", "count",
"filter" and "non empty" functions already running. 

I see here an opportunity to "transfer knowledge": it would be better that
someone (different than us) made propper changes to implement "Ancestor"
function (if has sense) for high cardilanity dimensions. For example, for
"Ancestor" function, the approach would be:
   The first question: Would Ancestor work for high cardinality dimensions? 
   Second question: How would it work for h.c. dimensions?
   Then, implement it.


Best regards!



- Luis F. Canals
  CISM
  luis.canals at stratebi.com



On 06/07/2008, at 19:48, Julian Hyde wrote:


Luis, Jorge, Javier,

There are still some errors with high-cardinality dimensions. See below; and
also search for 'execHighCardTest' in
http://www.hydromatic.net/buildlogs/mondrian-marmalade.hydromatic.net-200807
05-change11263.log.bz2.

Do you know about these? Any idea what the cause is? When can they be fixed?

Julian

Running test with JDK=jdk1.6 retroweave= database=oracle props={
mondrian.test.highCardDimensions=Store}

[java] 1)
testAncestorNumeric(mondrian.olap.fun.FunctionTest)mondrian.olap.MondrianExc
eption: Mondrian Error:Failed to parse query 'select
{Ancestor([Store].[USA].[Washington], 1)} on columns from [Sales Ragged]'

[java] 2)
testAncestorWithHiddenParent(mondrian.olap.fun.FunctionTest)mondrian.olap.Mo
ndrianException: Mondrian Error:Failed to parse query 'select
{Ancestor([Store].[All Stores].[Israel].[Haifa], [Store].[Store Country])}
on columns from [Sales Ragged]'

[java] 3)
testOrdinal(mondrian.olap.fun.FunctionTest)mondrian.olap.MondrianException:
Mondrian Error:Failed to parse query 'with member [Measures].[Foo] as
'[Store].[All Stores].[USA].[Washington].ordinal' select {[Measures].[Foo]}
on columns from [Sales Ragged]'

[java] 4)
testLead(mondrian.test.RaggedHierarchyTest)mondrian.olap.MondrianException:
Mondrian Error:Failed to parse query 'select {[Store].[All
Stores].[Mexico].[DF].Lead(1)} on columns from [Sales Ragged]'

[java] 5)
testParentOfHaifa(mondrian.test.RaggedHierarchyTest)mondrian.olap.MondrianEx
ception: Mondrian Error:Failed to parse query 'select
{[Store].[Israel].[Haifa].Parent} on columns from [Sales Ragged]'

[java] 6)
testPrevMemberOfHaifa(mondrian.test.RaggedHierarchyTest)mondrian.olap.Mondri
anException: Mondrian Error:Failed to parse query 'select
{[Store].[Israel].[Haifa].PrevMember} on columns from [Sales Ragged]'

[java] 7)
testNextMemberOfTelAviv(mondrian.test.RaggedHierarchyTest)mondrian.olap.Mond
rianException: Mondrian Error:Failed to parse query 'select
{[Store].[Israel].[Tel Aviv].NextMember} on columns from [Sales Ragged]'

[java] 8)
testNextMemberOfBC(mondrian.test.RaggedHierarchyTest)mondrian.olap.MondrianE
xception: Mondrian Error:Failed to parse query 'select {[Store].[All
Stores].[Canada].[BC].NextMember} on columns from [Sales Ragged]'

[java] 9)
testAncestorOfHaifa(mondrian.test.RaggedHierarchyTest)mondrian.olap.Mondrian
Exception: Mondrian Error:Failed to parse query 'select
{Ancestor([Store].[Israel].[Haifa], [Store].[Store State])} on columns from
[Sales Ragged]'

[java] 10)
testDataSourceChangeListenerPlugin(mondrian.rolap.DataSourceChangeListenerTe
st)java.lang.ClassCastException: mondrian.rolap.NoCacheMemberReader cannot
be cast to mondrian.rolap.SmartMemberReader

[java] 11)
testLookupMemberCache(mondrian.rolap.NonEmptyTest)java.lang.ClassCastExcepti
on: mondrian.rolap.NoCacheMemberReader cannot be cast to
mondrian.rolap.SmartMemberReader

[java] 12)
testLookupMember2(mondrian.rolap.NonEmptyTest)mondrian.olap.MondrianExceptio
n: Mondrian Error:Failed to parse query 'select {[Store].[USA].[Washington]}
on columns from [Sales Ragged]'

[java] 1)
testCrossJoinAsteriskQuery(mondrian.olap.fun.FunctionTest)junit.framework.Co
mparisonFailure: Expected:

[java] 2)
testCalculatedMemberInCube(mondrian.test.TestCalculatedMembers)junit.framewo
rk.ComparisonFailure: Expected:

[java] 3)
testHierarchize(mondrian.test.RaggedHierarchyTest)junit.framework.Comparison
Failure: Expected:

[java] 4)
testDescendantsOfRootAtCity(mondrian.test.RaggedHierarchyTest)junit.framewor
k.ComparisonFailure: Expected:

[java] Tests run: 1856, Failures: 4, Errors: 12


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.pentaho.org/pipermail/mondrian/attachments/20080714/0fb8da73/attachment.html 


More information about the Mondrian mailing list