[Mondrian] RE: Eigenbase perforce change 13367 for review

Julian Hyde jhyde at pentaho.com
Wed Feb 24 05:52:15 EST 2010


This is a smart fix, and I'm sure that it helps for really large schemas,
but is it always a win? We need RolapEvaluator.push() to be efficient, and
copying a treemap isn't that cheap: it involves an non-trivial iterator, and
lots of memory allocations. A map from integer to members can obviously be
implemented as an array. I want to be sure that we haven't made a critical
piece of code much more expensive in the usual just to improve algorithmic
complexity.

Julian

> -----Original Message-----
> From: Matt Campbell [mailto:Matthew.Campbell at thomson.com] 
> Sent: Wednesday, February 03, 2010 1:28 PM
> To: Aaron Phillips; Ezequiel Cuellar; Julian Hyde; John V. 
> Sichi; Will Gorman
> Subject: Eigenbase perforce change 13367 for review
> 
> http://p4web.eigenbase.org/@md=d&c=6PU@//13367?ac=10
> 
> Change 13367 by mkambol at guest_AA-5501 on 2010/02/03 13:26:50
> 
> 	MONDRIAN: performance enhancment.  RolapEvaluator now 
> has a map that contains all members in the context minus any 
> "all" members in context.  The new list is used in 
> getProperty and other places to significantly trim down the 
> number of iterations needed for each cell.  The amount of 
> performance gain is proportional to the number of cells in 
> the results times the number of dimensions in the schema.
> 
> Affected files ...
> 
> ... 
> //open/mondrian-release/3.2/src/main/mondrian/olap/Evaluator.j
> ava#2 edit
> ... 
> //open/mondrian-release/3.2/src/main/mondrian/rolap/RolapAggre
> gationManager.java#2 edit
> ... 
> //open/mondrian-release/3.2/src/main/mondrian/rolap/RolapEvalu
> ator.java#2 edit
> ... 
> //open/mondrian-release/3.2/src/main/mondrian/rolap/RolapEvalu
> atorRoot.java#2 edit
> ... 
> //open/mondrian-release/3.2/src/main/mondrian/rolap/SqlConstra
> intUtils.java#2 edit
> ... 
> //open/mondrian-release/3.2/src/main/mondrian/rolap/SqlMemberS
> ource.java#2 edit
> ... 
> //open/mondrian-release/3.2/src/main/mondrian/rolap/SqlTupleRe
> ader.java#2 edit
> ... 
> //open/mondrian-release/3.2/testsrc/main/mondrian/test/Perform
> anceTest.java#2 edit
> 
> Differences ...
> 
> ==== 
> //open/mondrian-release/3.2/src/main/mondrian/olap/Evaluator.j
> ava#2 (ktext) ====
> 
> 2c2
> < // $Id: 
> //open/mondrian-release/3.2/src/main/mondrian/olap/Evaluator.java#1 $
> ---
> > // $Id: 
> //open/mondrian-release/3.2/src/main/mondrian/olap/Evaluator.java#2 $
> 27c27
> <  * @version $Id: 
> //open/mondrian-release/3.2/src/main/mondrian/olap/Evaluator.java#1 $
> ---
> >  * @version $Id: 
> //open/mondrian-release/3.2/src/main/mondrian/olap/Evaluator.java#2 $
> 227a228,232
> >      * Returns an array of the non-All members which make 
> up the current context
> >      */
> >     Member[] getNonAllMembers();
> > 
> >     /**
> 
> ==== 
> //open/mondrian-release/3.2/src/main/mondrian/rolap/RolapAggre
> gationManager.java#2 (ktext) ====
> 
> 2c2
> < // $Id: 
> //open/mondrian-release/3.2/src/main/mondrian/rolap/RolapAggre
> gationManager.java#1 $
> ---
> > // $Id: 
> //open/mondrian-release/3.2/src/main/mondrian/rolap/RolapAggre
> gationManager.java#2 $
> 32c32
> <  * @version $Id: 
> //open/mondrian-release/3.2/src/main/mondrian/rolap/RolapAggre
> gationManager.java#1 $
> ---
> >  * @version $Id: 
> //open/mondrian-release/3.2/src/main/mondrian/rolap/RolapAggre
> gationManager.java#2 $
> 98c98
> <         final Member[] currentMembers = evaluator.getMembers();
> ---
> >         final Member[] currentMembers = 
> evaluator.getNonAllMembers();
> 
> ==== 
> //open/mondrian-release/3.2/src/main/mondrian/rolap/RolapEvalu
> ator.java#2 (ktext) ====
> 
> 2c2
> < // $Id: 
> //open/mondrian-release/3.2/src/main/mondrian/rolap/RolapEvalu
> ator.java#1 $
> ---
> > // $Id: 
> //open/mondrian-release/3.2/src/main/mondrian/rolap/RolapEvalu
> ator.java#2 $
> 45c45
> <  * @version $Id: 
> //open/mondrian-release/3.2/src/main/mondrian/rolap/RolapEvalu
> ator.java#1 $
> ---
> >  * @version $Id: 
> //open/mondrian-release/3.2/src/main/mondrian/rolap/RolapEvalu
> ator.java#2 $
> 55a56
> >     private TreeMap<Integer, RolapMember> nonAllMemberMap;
> 112a114,115
> >         nonAllMemberMap =
> >             (TreeMap<Integer, RolapMember>) 
> parent.nonAllMemberMap.clone();
> 140a144,145
> >         nonAllMemberMap =
> >             (TreeMap<Integer, RolapMember>) 
> root.nonAllDefaultMembers.clone();
> 223a229,233
> >     public final Member[] getNonAllMembers() {
> >         final Collection<RolapMember> members = 
> nonAllMemberMap.values();
> >         return members.toArray(new Member[members.size()]);
> >     }
> > 
> 383a394,398
> >         if (m.isAll()) {
> >             
> nonAllMemberMap.remove(m.getHierarchy().getOrdinalInCube());
> >         } else {
> >             
> nonAllMemberMap.put(m.getHierarchy().getOrdinalInCube(), m);
> >         }
> 600,602c615,617
> <         for (int i = 0; i < currentMembers.length; i++) {
> <             final Member member = currentMembers[i];
> < 
> ---
> >         int i = -1;
> >         for (RolapMember member : nonAllMemberMap.values()) {
> >             i++;
> 
> ==== 
> //open/mondrian-release/3.2/src/main/mondrian/rolap/RolapEvalu
> atorRoot.java#2 (ktext) ====
> 
> 2c2
> < // $Id: 
> //open/mondrian-release/3.2/src/main/mondrian/rolap/RolapEvalu
> atorRoot.java#1 $
> ---
> > // $Id: 
> //open/mondrian-release/3.2/src/main/mondrian/rolap/RolapEvalu
> atorRoot.java#2 $
> 28c28
> <  * @version $Id: 
> //open/mondrian-release/3.2/src/main/mondrian/rolap/RolapEvalu
> atorRoot.java#1 $
> ---
> >  * @version $Id: 
> //open/mondrian-release/3.2/src/main/mondrian/rolap/RolapEvalu
> atorRoot.java#2 $
> 47a48
> >     final TreeMap<Integer, RolapMember> nonAllDefaultMembers;
> 66a68
> >         nonAllDefaultMembers = new TreeMap<Integer, RolapMember>();
> 91a94,97
> >             if (!defaultMember.isAll()) {
> >                 nonAllDefaultMembers.put(
> >                     hierarchy.getOrdinalInCube(), defaultMember);
> >             }
> 
> ==== 
> //open/mondrian-release/3.2/src/main/mondrian/rolap/SqlConstra
> intUtils.java#2 (ktext) ====
> 
> 2c2
> < // $Id: 
> //open/mondrian-release/3.2/src/main/mondrian/rolap/SqlConstra
> intUtils.java#1 $
> ---
> > // $Id: 
> //open/mondrian-release/3.2/src/main/mondrian/rolap/SqlConstra
> intUtils.java#2 $
> 32c32
> <  * @version $Id: 
> //open/mondrian-release/3.2/src/main/mondrian/rolap/SqlConstra
> intUtils.java#1 $
> ---
> >  * @version $Id: 
> //open/mondrian-release/3.2/src/main/mondrian/rolap/SqlConstra
> intUtils.java#2 $
> 57c57
> <         Member[] members = evaluator.getMembers();
> ---
> >         Member[] members = evaluator.getNonAllMembers();
> 
> ==== 
> //open/mondrian-release/3.2/src/main/mondrian/rolap/SqlMemberS
> ource.java#2 (ktext) ====
> 
> 2c2
> < // $Id: 
> //open/mondrian-release/3.2/src/main/mondrian/rolap/SqlMemberS
> ource.java#1 $
> ---
> > // $Id: 
> //open/mondrian-release/3.2/src/main/mondrian/rolap/SqlMemberS
> ource.java#2 $
> 39c39
> <  * @version $Id: 
> //open/mondrian-release/3.2/src/main/mondrian/rolap/SqlMemberS
> ource.java#1 $
> ---
> >  * @version $Id: 
> //open/mondrian-release/3.2/src/main/mondrian/rolap/SqlMemberS
> ource.java#2 $
> 688c688
> <         final Member[] members = evaluator.getMembers();
> ---
> >         final Member[] members = evaluator.getNonAllMembers();
> 
> ==== 
> //open/mondrian-release/3.2/src/main/mondrian/rolap/SqlTupleRe
> ader.java#2 (ktext) ====
> 
> 65c65
> <  * @version $Id: 
> //open/mondrian-release/3.2/src/main/mondrian/rolap/SqlTupleRe
> ader.java#1 $
> ---
> >  * @version $Id: 
> //open/mondrian-release/3.2/src/main/mondrian/rolap/SqlTupleRe
> ader.java#2 $
> 93c93
> <      * @version $Id: 
> //open/mondrian-release/3.2/src/main/mondrian/rolap/SqlTupleRe
> ader.java#1 $
> ---
> >      * @version $Id: 
> //open/mondrian-release/3.2/src/main/mondrian/rolap/SqlTupleRe
> ader.java#2 $
> 1132c1132
> <         final Member[] members = evaluator.getMembers();
> ---
> >         final Member[] members = evaluator.getNonAllMembers();
> 
> ==== 
> //open/mondrian-release/3.2/testsrc/main/mondrian/test/Perform
> anceTest.java#2 (ktext) ====
> 
> 2c2
> < // $Id: 
> //open/mondrian-release/3.2/testsrc/main/mondrian/test/Perform
> anceTest.java#1 $
> ---
> > // $Id: 
> //open/mondrian-release/3.2/testsrc/main/mondrian/test/Perform
> anceTest.java#2 $
> 22c22
> <  * @version $Id: 
> //open/mondrian-release/3.2/testsrc/main/mondrian/test/Perform
> anceTest.java#1 $
> ---
> >  * @version $Id: 
> //open/mondrian-release/3.2/testsrc/main/mondrian/test/Perform
> anceTest.java#2 $
> 214a215,354
> >     /***
> >      * Tests performance of a larger schema with a large 
> number of result cells
> >      * Runs in 12.2 seconds when RolapEvaluator.getProperty 
> uses currentMemmber
> >      * Runs in 7.5 seconds when RolapEvaluator.getProperty 
> uses nonAllMemberMap
> >      * The performance boost gets more significant as the 
> schema size grows
> >      */
> >     public void testBigResultsWithBigSchemaPerforms() {
> >         TestContext testContext = 
> TestContext.createSubstitutingCube(
> >             "Sales",
> >             "<Dimension name=\"Gender2\" 
> foreignKey=\"customer_id\">\n"
> >             + "<Hierarchy hasAll=\"true\" 
> allMemberName=\"All Gender\" primaryKey=\"customer_id\">\n"
> >             + "      <Table name=\"customer\"/>\n"
> >             + "      <Level name=\"Gender\" 
> column=\"gender\" uniqueMembers=\"true\"/>\n"
> >             + "    </Hierarchy>"
> >             + "</Dimension>"
> >             + "<Dimension name=\"Gender3\" 
> foreignKey=\"customer_id\">\n"
> >             + "<Hierarchy hasAll=\"true\" 
> allMemberName=\"All Gender\" primaryKey=\"customer_id\">\n"
> >             + "      <Table name=\"customer\"/>\n"
> >             + "      <Level name=\"Gender\" 
> column=\"gender\" uniqueMembers=\"true\"/>\n"
> >             + "    </Hierarchy>"
> >             + "</Dimension>"
> >             + "<Dimension name=\"Gender4\" 
> foreignKey=\"customer_id\">\n"
> >             + "<Hierarchy hasAll=\"true\" 
> allMemberName=\"All Gender\" primaryKey=\"customer_id\">\n"
> >             + "      <Table name=\"customer\"/>\n"
> >             + "      <Level name=\"Gender\" 
> column=\"gender\" uniqueMembers=\"true\"/>\n"
> >             + "    </Hierarchy>"
> >             + "</Dimension>"
> >             + "<Dimension name=\"Gender5\" 
> foreignKey=\"customer_id\">\n"
> >             + "<Hierarchy hasAll=\"true\" 
> allMemberName=\"All Gender\" primaryKey=\"customer_id\">\n"
> >             + "      <Table name=\"customer\"/>\n"
> >             + "      <Level name=\"Gender\" 
> column=\"gender\" uniqueMembers=\"true\"/>\n"
> >             + "    </Hierarchy>"
> >             + "</Dimension>"
> >             + "<Dimension name=\"Gender6\" 
> foreignKey=\"customer_id\">\n"
> >             + "<Hierarchy hasAll=\"true\" 
> allMemberName=\"All Gender\" primaryKey=\"customer_id\">\n"
> >             + "      <Table name=\"customer\"/>\n"
> >             + "      <Level name=\"Gender\" 
> column=\"gender\" uniqueMembers=\"true\"/>\n"
> >             + "    </Hierarchy>"
> >             + "</Dimension>"
> >             + "<Dimension name=\"Gender7\" 
> foreignKey=\"customer_id\">\n"
> >             + "<Hierarchy hasAll=\"true\" 
> allMemberName=\"All Gender\" primaryKey=\"customer_id\">\n"
> >             + "      <Table name=\"customer\"/>\n"
> >             + "      <Level name=\"Gender\" 
> column=\"gender\" uniqueMembers=\"true\"/>\n"
> >             + "    </Hierarchy>"
> >             + "</Dimension>"
> >             + "<Dimension name=\"Gender8\" 
> foreignKey=\"customer_id\">\n"
> >             + "<Hierarchy hasAll=\"true\" 
> allMemberName=\"All Gender\" primaryKey=\"customer_id\">\n"
> >             + "      <Table name=\"customer\"/>\n"
> >             + "      <Level name=\"Gender\" 
> column=\"gender\" uniqueMembers=\"true\"/>\n"
> >             + "    </Hierarchy>"
> >             + "</Dimension>"
> >             + "<Dimension name=\"Gender9\" 
> foreignKey=\"customer_id\">\n"
> >             + "<Hierarchy hasAll=\"true\" 
> allMemberName=\"All Gender\" primaryKey=\"customer_id\">\n"
> >             + "      <Table name=\"customer\"/>\n"
> >             + "      <Level name=\"Gender\" 
> column=\"gender\" uniqueMembers=\"true\"/>\n"
> >             + "    </Hierarchy>"
> >             + "</Dimension>"
> >             + "<Dimension name=\"Gender10\" 
> foreignKey=\"customer_id\">\n"
> >             + "<Hierarchy hasAll=\"true\" 
> allMemberName=\"All Gender\" primaryKey=\"customer_id\">\n"
> >             + "      <Table name=\"customer\"/>\n"
> >             + "      <Level name=\"Gender\" 
> column=\"gender\" uniqueMembers=\"true\"/>\n"
> >             + "    </Hierarchy>"
> >             + "</Dimension>"
> >             + "<Dimension name=\"Gender11\" 
> foreignKey=\"customer_id\">\n"
> >             + "<Hierarchy hasAll=\"true\" 
> allMemberName=\"All Gender\" primaryKey=\"customer_id\">\n"
> >             + "      <Table name=\"customer\"/>\n"
> >             + "      <Level name=\"Gender\" 
> column=\"gender\" uniqueMembers=\"true\"/>\n"
> >             + "    </Hierarchy>"
> >             + "</Dimension>"
> >             + "<Dimension name=\"Gender12\" 
> foreignKey=\"customer_id\">\n"
> >             + "<Hierarchy hasAll=\"true\" 
> allMemberName=\"All Gender\" primaryKey=\"customer_id\">\n"
> >             + "      <Table name=\"customer\"/>\n"
> >             + "      <Level name=\"Gender\" 
> column=\"gender\" uniqueMembers=\"true\"/>\n"
> >             + "    </Hierarchy>"
> >             + "</Dimension>"
> >             + "<Dimension name=\"Gender13\" 
> foreignKey=\"customer_id\">\n"
> >             + "<Hierarchy hasAll=\"true\" 
> allMemberName=\"All Gender\" primaryKey=\"customer_id\">\n"
> >             + "      <Table name=\"customer\"/>\n"
> >             + "      <Level name=\"Gender\" 
> column=\"gender\" uniqueMembers=\"true\"/>\n"
> >             + "    </Hierarchy>"
> >             + "</Dimension>"
> >             + "<Dimension name=\"Gender14\" 
> foreignKey=\"customer_id\">\n"
> >             + "<Hierarchy hasAll=\"true\" 
> allMemberName=\"All Gender\" primaryKey=\"customer_id\">\n"
> >             + "      <Table name=\"customer\"/>\n"
> >             + "      <Level name=\"Gender\" 
> column=\"gender\" uniqueMembers=\"true\"/>\n"
> >             + "    </Hierarchy>"
> >             + "</Dimension>"
> >             + "<Dimension name=\"Gender15\" 
> foreignKey=\"customer_id\">\n"
> >             + "<Hierarchy hasAll=\"true\" 
> allMemberName=\"All Gender\" primaryKey=\"customer_id\">\n"
> >             + "      <Table name=\"customer\"/>\n"
> >             + "      <Level name=\"Gender\" 
> column=\"gender\" uniqueMembers=\"true\"/>\n"
> >             + "    </Hierarchy>"
> >             + "</Dimension>"
> >             + "<Dimension name=\"Gender16\" 
> foreignKey=\"customer_id\">\n"
> >             + "<Hierarchy hasAll=\"true\" 
> allMemberName=\"All Gender\" primaryKey=\"customer_id\">\n"
> >             + "      <Table name=\"customer\"/>\n"
> >             + "      <Level name=\"Gender\" 
> column=\"gender\" uniqueMembers=\"true\"/>\n"
> >             + "    </Hierarchy>"
> >             + "</Dimension>"
> >             + "<Dimension name=\"Gender17\" 
> foreignKey=\"customer_id\">\n"
> >             + "<Hierarchy hasAll=\"true\" 
> allMemberName=\"All Gender\" primaryKey=\"customer_id\">\n"
> >             + "      <Table name=\"customer\"/>\n"
> >             + "      <Level name=\"Gender\" 
> column=\"gender\" uniqueMembers=\"true\"/>\n"
> >             + "    </Hierarchy>"
> >             + "</Dimension>"
> >             + "<Dimension name=\"Gender18\" 
> foreignKey=\"customer_id\">\n"
> >             + "<Hierarchy hasAll=\"true\" 
> allMemberName=\"All Gender\" primaryKey=\"customer_id\">\n"
> >             + "      <Table name=\"customer\"/>\n"
> >             + "      <Level name=\"Gender\" 
> column=\"gender\" uniqueMembers=\"true\"/>\n"
> >             + "    </Hierarchy>"
> >             + "</Dimension>"
> >             + "<Dimension name=\"Gender19\" 
> foreignKey=\"customer_id\">\n"
> >             + "<Hierarchy hasAll=\"true\" 
> allMemberName=\"All Gender\" primaryKey=\"customer_id\">\n"
> >             + "      <Table name=\"customer\"/>\n"
> >             + "      <Level name=\"Gender\" 
> column=\"gender\" uniqueMembers=\"true\"/>\n"
> >             + "    </Hierarchy>"
> >             + "</Dimension>"
> >             + "<Dimension name=\"Gender20\" 
> foreignKey=\"customer_id\">\n"
> >             + "<Hierarchy hasAll=\"true\" 
> allMemberName=\"All Gender\" primaryKey=\"customer_id\">\n"
> >             + "      <Table name=\"customer\"/>\n"
> >             + "      <Level name=\"Gender\" 
> column=\"gender\" uniqueMembers=\"true\"/>\n"
> >             + "    </Hierarchy>"
> >             + "</Dimension>",
> >             null);
> >         String mdx = "with "
> >                      + " member [Measures].[one] as '1'"
> >                      + " member [Measures].[two] as '2'"
> >                      + " member [Measures].[three] as '3'"
> >                      + " member [Measures].[four] as '4'"
> >                      + " member [Measures].[five] as '5'"
> >                      + " select "
> >                      + 
> "{[Measures].[one],[Measures].[two],[Measures].[three],[Measur
> es].[four],[Measures].[five]}"
> >                      + " on 0, "
> >                      + 
> "Crossjoin([Customers].[name].members,[Store].[Store Name].members)"
> >                      + " on 1 from sales";
> >         long start = System.currentTimeMillis();
> >         testContext.executeQuery(mdx);
> >         printDuration("getProperty taking a long time", start);
> >     }
> > 
> 




More information about the Mondrian mailing list