[Mondrian] RE: Eigenbase perforce change 13367 for review

Kurtis.Walker at thomsonreuters.com Kurtis.Walker at thomsonreuters.com
Wed Feb 24 09:41:25 EST 2010


Hi Julian,
   I implemented nonAllMembers as a Map because the keys will be very
sparse compared to possible values.  If it was an array, it would have
to be the full length of the currentMembers, and we'd have to filter out
all the nulls whenever we wanted to use it.  Do you see another way?

Since TreeMap.clone does a shallow copy, and I expect nonAllMembersMap
usually has a very small number of members in it, I figured it would be
fast.  I'll do some additional profiling to see what kind of time we are
spending in the push.  My previous profiling runs did not surface it as
a hot spot, but I also wasn't specifically looking at that method.

Kurt

-----Original Message-----
From: mondrian-bounces at pentaho.org [mailto:mondrian-bounces at pentaho.org]
On Behalf Of Julian Hyde
Sent: Wednesday, February 24, 2010 5:52 AM
To: Campbell, Matthew (Healthcare USA)
Cc: 'Mondrian developer mailing list'
Subject: [Mondrian] RE: Eigenbase perforce change 13367 for review

This is a smart fix, and I'm sure that it helps for really large
schemas,
but is it always a win? We need RolapEvaluator.push() to be efficient,
and
copying a treemap isn't that cheap: it involves an non-trivial iterator,
and
lots of memory allocations. A map from integer to members can obviously
be
implemented as an array. I want to be sure that we haven't made a
critical
piece of code much more expensive in the usual just to improve
algorithmic
complexity.

Julian

> -----Original Message-----
> From: Matt Campbell [mailto:Matthew.Campbell at thomson.com] 
> Sent: Wednesday, February 03, 2010 1:28 PM
> To: Aaron Phillips; Ezequiel Cuellar; Julian Hyde; John V. 
> Sichi; Will Gorman
> Subject: Eigenbase perforce change 13367 for review
> 
> http://p4web.eigenbase.org/@md=d&c=6PU@//13367?ac=10
> 
> Change 13367 by mkambol at guest_AA-5501 on 2010/02/03 13:26:50
> 
> 	MONDRIAN: performance enhancment.  RolapEvaluator now 
> has a map that contains all members in the context minus any 
> "all" members in context.  The new list is used in 
> getProperty and other places to significantly trim down the 
> number of iterations needed for each cell.  The amount of 
> performance gain is proportional to the number of cells in 
> the results times the number of dimensions in the schema.
> 
> Affected files ...
> 
> ... 
> //open/mondrian-release/3.2/src/main/mondrian/olap/Evaluator.j
> ava#2 edit
> ... 
> //open/mondrian-release/3.2/src/main/mondrian/rolap/RolapAggre
> gationManager.java#2 edit
> ... 
> //open/mondrian-release/3.2/src/main/mondrian/rolap/RolapEvalu
> ator.java#2 edit
> ... 
> //open/mondrian-release/3.2/src/main/mondrian/rolap/RolapEvalu
> atorRoot.java#2 edit
> ... 
> //open/mondrian-release/3.2/src/main/mondrian/rolap/SqlConstra
> intUtils.java#2 edit
> ... 
> //open/mondrian-release/3.2/src/main/mondrian/rolap/SqlMemberS
> ource.java#2 edit
> ... 
> //open/mondrian-release/3.2/src/main/mondrian/rolap/SqlTupleRe
> ader.java#2 edit
> ... 
> //open/mondrian-release/3.2/testsrc/main/mondrian/test/Perform
> anceTest.java#2 edit
> 
> Differences ...
> 
> ==== 
> //open/mondrian-release/3.2/src/main/mondrian/olap/Evaluator.j
> ava#2 (ktext) ====
> 
> 2c2
> < // $Id: 
> //open/mondrian-release/3.2/src/main/mondrian/olap/Evaluator.java#1 $
> ---
> > // $Id: 
> //open/mondrian-release/3.2/src/main/mondrian/olap/Evaluator.java#2 $
> 27c27
> <  * @version $Id: 
> //open/mondrian-release/3.2/src/main/mondrian/olap/Evaluator.java#1 $
> ---
> >  * @version $Id: 
> //open/mondrian-release/3.2/src/main/mondrian/olap/Evaluator.java#2 $
> 227a228,232
> >      * Returns an array of the non-All members which make 
> up the current context
> >      */
> >     Member[] getNonAllMembers();
> > 
> >     /**
> 
> ==== 
> //open/mondrian-release/3.2/src/main/mondrian/rolap/RolapAggre
> gationManager.java#2 (ktext) ====
> 
> 2c2
> < // $Id: 
> //open/mondrian-release/3.2/src/main/mondrian/rolap/RolapAggre
> gationManager.java#1 $
> ---
> > // $Id: 
> //open/mondrian-release/3.2/src/main/mondrian/rolap/RolapAggre
> gationManager.java#2 $
> 32c32
> <  * @version $Id: 
> //open/mondrian-release/3.2/src/main/mondrian/rolap/RolapAggre
> gationManager.java#1 $
> ---
> >  * @version $Id: 
> //open/mondrian-release/3.2/src/main/mondrian/rolap/RolapAggre
> gationManager.java#2 $
> 98c98
> <         final Member[] currentMembers = evaluator.getMembers();
> ---
> >         final Member[] currentMembers = 
> evaluator.getNonAllMembers();
> 
> ==== 
> //open/mondrian-release/3.2/src/main/mondrian/rolap/RolapEvalu
> ator.java#2 (ktext) ====
> 
> 2c2
> < // $Id: 
> //open/mondrian-release/3.2/src/main/mondrian/rolap/RolapEvalu
> ator.java#1 $
> ---
> > // $Id: 
> //open/mondrian-release/3.2/src/main/mondrian/rolap/RolapEvalu
> ator.java#2 $
> 45c45
> <  * @version $Id: 
> //open/mondrian-release/3.2/src/main/mondrian/rolap/RolapEvalu
> ator.java#1 $
> ---
> >  * @version $Id: 
> //open/mondrian-release/3.2/src/main/mondrian/rolap/RolapEvalu
> ator.java#2 $
> 55a56
> >     private TreeMap<Integer, RolapMember> nonAllMemberMap;
> 112a114,115
> >         nonAllMemberMap =
> >             (TreeMap<Integer, RolapMember>) 
> parent.nonAllMemberMap.clone();
> 140a144,145
> >         nonAllMemberMap =
> >             (TreeMap<Integer, RolapMember>) 
> root.nonAllDefaultMembers.clone();
> 223a229,233
> >     public final Member[] getNonAllMembers() {
> >         final Collection<RolapMember> members = 
> nonAllMemberMap.values();
> >         return members.toArray(new Member[members.size()]);
> >     }
> > 
> 383a394,398
> >         if (m.isAll()) {
> >             
> nonAllMemberMap.remove(m.getHierarchy().getOrdinalInCube());
> >         } else {
> >             
> nonAllMemberMap.put(m.getHierarchy().getOrdinalInCube(), m);
> >         }
> 600,602c615,617
> <         for (int i = 0; i < currentMembers.length; i++) {
> <             final Member member = currentMembers[i];
> < 
> ---
> >         int i = -1;
> >         for (RolapMember member : nonAllMemberMap.values()) {
> >             i++;
> 
> ==== 
> //open/mondrian-release/3.2/src/main/mondrian/rolap/RolapEvalu
> atorRoot.java#2 (ktext) ====
> 
> 2c2
> < // $Id: 
> //open/mondrian-release/3.2/src/main/mondrian/rolap/RolapEvalu
> atorRoot.java#1 $
> ---
> > // $Id: 
> //open/mondrian-release/3.2/src/main/mondrian/rolap/RolapEvalu
> atorRoot.java#2 $
> 28c28
> <  * @version $Id: 
> //open/mondrian-release/3.2/src/main/mondrian/rolap/RolapEvalu
> atorRoot.java#1 $
> ---
> >  * @version $Id: 
> //open/mondrian-release/3.2/src/main/mondrian/rolap/RolapEvalu
> atorRoot.java#2 $
> 47a48
> >     final TreeMap<Integer, RolapMember> nonAllDefaultMembers;
> 66a68
> >         nonAllDefaultMembers = new TreeMap<Integer, RolapMember>();
> 91a94,97
> >             if (!defaultMember.isAll()) {
> >                 nonAllDefaultMembers.put(
> >                     hierarchy.getOrdinalInCube(), defaultMember);
> >             }
> 
> ==== 
> //open/mondrian-release/3.2/src/main/mondrian/rolap/SqlConstra
> intUtils.java#2 (ktext) ====
> 
> 2c2
> < // $Id: 
> //open/mondrian-release/3.2/src/main/mondrian/rolap/SqlConstra
> intUtils.java#1 $
> ---
> > // $Id: 
> //open/mondrian-release/3.2/src/main/mondrian/rolap/SqlConstra
> intUtils.java#2 $
> 32c32
> <  * @version $Id: 
> //open/mondrian-release/3.2/src/main/mondrian/rolap/SqlConstra
> intUtils.java#1 $
> ---
> >  * @version $Id: 
> //open/mondrian-release/3.2/src/main/mondrian/rolap/SqlConstra
> intUtils.java#2 $
> 57c57
> <         Member[] members = evaluator.getMembers();
> ---
> >         Member[] members = evaluator.getNonAllMembers();
> 
> ==== 
> //open/mondrian-release/3.2/src/main/mondrian/rolap/SqlMemberS
> ource.java#2 (ktext) ====
> 
> 2c2
> < // $Id: 
> //open/mondrian-release/3.2/src/main/mondrian/rolap/SqlMemberS
> ource.java#1 $
> ---
> > // $Id: 
> //open/mondrian-release/3.2/src/main/mondrian/rolap/SqlMemberS
> ource.java#2 $
> 39c39
> <  * @version $Id: 
> //open/mondrian-release/3.2/src/main/mondrian/rolap/SqlMemberS
> ource.java#1 $
> ---
> >  * @version $Id: 
> //open/mondrian-release/3.2/src/main/mondrian/rolap/SqlMemberS
> ource.java#2 $
> 688c688
> <         final Member[] members = evaluator.getMembers();
> ---
> >         final Member[] members = evaluator.getNonAllMembers();
> 
> ==== 
> //open/mondrian-release/3.2/src/main/mondrian/rolap/SqlTupleRe
> ader.java#2 (ktext) ====
> 
> 65c65
> <  * @version $Id: 
> //open/mondrian-release/3.2/src/main/mondrian/rolap/SqlTupleRe
> ader.java#1 $
> ---
> >  * @version $Id: 
> //open/mondrian-release/3.2/src/main/mondrian/rolap/SqlTupleRe
> ader.java#2 $
> 93c93
> <      * @version $Id: 
> //open/mondrian-release/3.2/src/main/mondrian/rolap/SqlTupleRe
> ader.java#1 $
> ---
> >      * @version $Id: 
> //open/mondrian-release/3.2/src/main/mondrian/rolap/SqlTupleRe
> ader.java#2 $
> 1132c1132
> <         final Member[] members = evaluator.getMembers();
> ---
> >         final Member[] members = evaluator.getNonAllMembers();
> 
> ==== 
> //open/mondrian-release/3.2/testsrc/main/mondrian/test/Perform
> anceTest.java#2 (ktext) ====
> 
> 2c2
> < // $Id: 
> //open/mondrian-release/3.2/testsrc/main/mondrian/test/Perform
> anceTest.java#1 $
> ---
> > // $Id: 
> //open/mondrian-release/3.2/testsrc/main/mondrian/test/Perform
> anceTest.java#2 $
> 22c22
> <  * @version $Id: 
> //open/mondrian-release/3.2/testsrc/main/mondrian/test/Perform
> anceTest.java#1 $
> ---
> >  * @version $Id: 
> //open/mondrian-release/3.2/testsrc/main/mondrian/test/Perform
> anceTest.java#2 $
> 214a215,354
> >     /***
> >      * Tests performance of a larger schema with a large 
> number of result cells
> >      * Runs in 12.2 seconds when RolapEvaluator.getProperty 
> uses currentMemmber
> >      * Runs in 7.5 seconds when RolapEvaluator.getProperty 
> uses nonAllMemberMap
> >      * The performance boost gets more significant as the 
> schema size grows
> >      */
> >     public void testBigResultsWithBigSchemaPerforms() {
> >         TestContext testContext = 
> TestContext.createSubstitutingCube(
> >             "Sales",
> >             "<Dimension name=\"Gender2\" 
> foreignKey=\"customer_id\">\n"
> >             + "<Hierarchy hasAll=\"true\" 
> allMemberName=\"All Gender\" primaryKey=\"customer_id\">\n"
> >             + "      <Table name=\"customer\"/>\n"
> >             + "      <Level name=\"Gender\" 
> column=\"gender\" uniqueMembers=\"true\"/>\n"
> >             + "    </Hierarchy>"
> >             + "</Dimension>"
> >             + "<Dimension name=\"Gender3\" 
> foreignKey=\"customer_id\">\n"
> >             + "<Hierarchy hasAll=\"true\" 
> allMemberName=\"All Gender\" primaryKey=\"customer_id\">\n"
> >             + "      <Table name=\"customer\"/>\n"
> >             + "      <Level name=\"Gender\" 
> column=\"gender\" uniqueMembers=\"true\"/>\n"
> >             + "    </Hierarchy>"
> >             + "</Dimension>"
> >             + "<Dimension name=\"Gender4\" 
> foreignKey=\"customer_id\">\n"
> >             + "<Hierarchy hasAll=\"true\" 
> allMemberName=\"All Gender\" primaryKey=\"customer_id\">\n"
> >             + "      <Table name=\"customer\"/>\n"
> >             + "      <Level name=\"Gender\" 
> column=\"gender\" uniqueMembers=\"true\"/>\n"
> >             + "    </Hierarchy>"
> >             + "</Dimension>"
> >             + "<Dimension name=\"Gender5\" 
> foreignKey=\"customer_id\">\n"
> >             + "<Hierarchy hasAll=\"true\" 
> allMemberName=\"All Gender\" primaryKey=\"customer_id\">\n"
> >             + "      <Table name=\"customer\"/>\n"
> >             + "      <Level name=\"Gender\" 
> column=\"gender\" uniqueMembers=\"true\"/>\n"
> >             + "    </Hierarchy>"
> >             + "</Dimension>"
> >             + "<Dimension name=\"Gender6\" 
> foreignKey=\"customer_id\">\n"
> >             + "<Hierarchy hasAll=\"true\" 
> allMemberName=\"All Gender\" primaryKey=\"customer_id\">\n"
> >             + "      <Table name=\"customer\"/>\n"
> >             + "      <Level name=\"Gender\" 
> column=\"gender\" uniqueMembers=\"true\"/>\n"
> >             + "    </Hierarchy>"
> >             + "</Dimension>"
> >             + "<Dimension name=\"Gender7\" 
> foreignKey=\"customer_id\">\n"
> >             + "<Hierarchy hasAll=\"true\" 
> allMemberName=\"All Gender\" primaryKey=\"customer_id\">\n"
> >             + "      <Table name=\"customer\"/>\n"
> >             + "      <Level name=\"Gender\" 
> column=\"gender\" uniqueMembers=\"true\"/>\n"
> >             + "    </Hierarchy>"
> >             + "</Dimension>"
> >             + "<Dimension name=\"Gender8\" 
> foreignKey=\"customer_id\">\n"
> >             + "<Hierarchy hasAll=\"true\" 
> allMemberName=\"All Gender\" primaryKey=\"customer_id\">\n"
> >             + "      <Table name=\"customer\"/>\n"
> >             + "      <Level name=\"Gender\" 
> column=\"gender\" uniqueMembers=\"true\"/>\n"
> >             + "    </Hierarchy>"
> >             + "</Dimension>"
> >             + "<Dimension name=\"Gender9\" 
> foreignKey=\"customer_id\">\n"
> >             + "<Hierarchy hasAll=\"true\" 
> allMemberName=\"All Gender\" primaryKey=\"customer_id\">\n"
> >             + "      <Table name=\"customer\"/>\n"
> >             + "      <Level name=\"Gender\" 
> column=\"gender\" uniqueMembers=\"true\"/>\n"
> >             + "    </Hierarchy>"
> >             + "</Dimension>"
> >             + "<Dimension name=\"Gender10\" 
> foreignKey=\"customer_id\">\n"
> >             + "<Hierarchy hasAll=\"true\" 
> allMemberName=\"All Gender\" primaryKey=\"customer_id\">\n"
> >             + "      <Table name=\"customer\"/>\n"
> >             + "      <Level name=\"Gender\" 
> column=\"gender\" uniqueMembers=\"true\"/>\n"
> >             + "    </Hierarchy>"
> >             + "</Dimension>"
> >             + "<Dimension name=\"Gender11\" 
> foreignKey=\"customer_id\">\n"
> >             + "<Hierarchy hasAll=\"true\" 
> allMemberName=\"All Gender\" primaryKey=\"customer_id\">\n"
> >             + "      <Table name=\"customer\"/>\n"
> >             + "      <Level name=\"Gender\" 
> column=\"gender\" uniqueMembers=\"true\"/>\n"
> >             + "    </Hierarchy>"
> >             + "</Dimension>"
> >             + "<Dimension name=\"Gender12\" 
> foreignKey=\"customer_id\">\n"
> >             + "<Hierarchy hasAll=\"true\" 
> allMemberName=\"All Gender\" primaryKey=\"customer_id\">\n"
> >             + "      <Table name=\"customer\"/>\n"
> >             + "      <Level name=\"Gender\" 
> column=\"gender\" uniqueMembers=\"true\"/>\n"
> >             + "    </Hierarchy>"
> >             + "</Dimension>"
> >             + "<Dimension name=\"Gender13\" 
> foreignKey=\"customer_id\">\n"
> >             + "<Hierarchy hasAll=\"true\" 
> allMemberName=\"All Gender\" primaryKey=\"customer_id\">\n"
> >             + "      <Table name=\"customer\"/>\n"
> >             + "      <Level name=\"Gender\" 
> column=\"gender\" uniqueMembers=\"true\"/>\n"
> >             + "    </Hierarchy>"
> >             + "</Dimension>"
> >             + "<Dimension name=\"Gender14\" 
> foreignKey=\"customer_id\">\n"
> >             + "<Hierarchy hasAll=\"true\" 
> allMemberName=\"All Gender\" primaryKey=\"customer_id\">\n"
> >             + "      <Table name=\"customer\"/>\n"
> >             + "      <Level name=\"Gender\" 
> column=\"gender\" uniqueMembers=\"true\"/>\n"
> >             + "    </Hierarchy>"
> >             + "</Dimension>"
> >             + "<Dimension name=\"Gender15\" 
> foreignKey=\"customer_id\">\n"
> >             + "<Hierarchy hasAll=\"true\" 
> allMemberName=\"All Gender\" primaryKey=\"customer_id\">\n"
> >             + "      <Table name=\"customer\"/>\n"
> >             + "      <Level name=\"Gender\" 
> column=\"gender\" uniqueMembers=\"true\"/>\n"
> >             + "    </Hierarchy>"
> >             + "</Dimension>"
> >             + "<Dimension name=\"Gender16\" 
> foreignKey=\"customer_id\">\n"
> >             + "<Hierarchy hasAll=\"true\" 
> allMemberName=\"All Gender\" primaryKey=\"customer_id\">\n"
> >             + "      <Table name=\"customer\"/>\n"
> >             + "      <Level name=\"Gender\" 
> column=\"gender\" uniqueMembers=\"true\"/>\n"
> >             + "    </Hierarchy>"
> >             + "</Dimension>"
> >             + "<Dimension name=\"Gender17\" 
> foreignKey=\"customer_id\">\n"
> >             + "<Hierarchy hasAll=\"true\" 
> allMemberName=\"All Gender\" primaryKey=\"customer_id\">\n"
> >             + "      <Table name=\"customer\"/>\n"
> >             + "      <Level name=\"Gender\" 
> column=\"gender\" uniqueMembers=\"true\"/>\n"
> >             + "    </Hierarchy>"
> >             + "</Dimension>"
> >             + "<Dimension name=\"Gender18\" 
> foreignKey=\"customer_id\">\n"
> >             + "<Hierarchy hasAll=\"true\" 
> allMemberName=\"All Gender\" primaryKey=\"customer_id\">\n"
> >             + "      <Table name=\"customer\"/>\n"
> >             + "      <Level name=\"Gender\" 
> column=\"gender\" uniqueMembers=\"true\"/>\n"
> >             + "    </Hierarchy>"
> >             + "</Dimension>"
> >             + "<Dimension name=\"Gender19\" 
> foreignKey=\"customer_id\">\n"
> >             + "<Hierarchy hasAll=\"true\" 
> allMemberName=\"All Gender\" primaryKey=\"customer_id\">\n"
> >             + "      <Table name=\"customer\"/>\n"
> >             + "      <Level name=\"Gender\" 
> column=\"gender\" uniqueMembers=\"true\"/>\n"
> >             + "    </Hierarchy>"
> >             + "</Dimension>"
> >             + "<Dimension name=\"Gender20\" 
> foreignKey=\"customer_id\">\n"
> >             + "<Hierarchy hasAll=\"true\" 
> allMemberName=\"All Gender\" primaryKey=\"customer_id\">\n"
> >             + "      <Table name=\"customer\"/>\n"
> >             + "      <Level name=\"Gender\" 
> column=\"gender\" uniqueMembers=\"true\"/>\n"
> >             + "    </Hierarchy>"
> >             + "</Dimension>",
> >             null);
> >         String mdx = "with "
> >                      + " member [Measures].[one] as '1'"
> >                      + " member [Measures].[two] as '2'"
> >                      + " member [Measures].[three] as '3'"
> >                      + " member [Measures].[four] as '4'"
> >                      + " member [Measures].[five] as '5'"
> >                      + " select "
> >                      + 
> "{[Measures].[one],[Measures].[two],[Measures].[three],[Measur
> es].[four],[Measures].[five]}"
> >                      + " on 0, "
> >                      + 
> "Crossjoin([Customers].[name].members,[Store].[Store Name].members)"
> >                      + " on 1 from sales";
> >         long start = System.currentTimeMillis();
> >         testContext.executeQuery(mdx);
> >         printDuration("getProperty taking a long time", start);
> >     }
> > 
> 

_______________________________________________
Mondrian mailing list
Mondrian at pentaho.org
http://lists.pentaho.org/mailman/listinfo/mondrian




More information about the Mondrian mailing list