[Mondrian] Refactor to change Member by List<Member>
remberson at edgedynamics.com
Tue Nov 13 14:48:24 EST 2007
I saw my name mentioned :-)
A while ago an additional mondrian.calc.ResultStyle was
added: ITERABLE. An ITERABLE implements the Iterable
interface but is not necessarily a List.
A List (such as an ArrayList) has all of its data
in memory (as elements of an array). On the other hand,
an Iterable may not have it members "realize" or
"materialized" in memory. Lists are generated and loaded
at the point of creation while an Iterable returns its
values dynamically on demand - the memory foot print of
a crossjoin that returns an Iterable is much smaller than
a crossjoin that is materialized fully at creation.
All of this was to reduce memory usage (while keeping an
eye on performance). A RolapResult uses RolapAxis (as the
Axis implementation) and for these Axes if the calling
application is careful only to iterate across the Axes and
their Positions, then with Iterables the foot print can
be kept small. If the calling application makes a call
that is "non-iterable" like getting an element at a
given position (list.get(4)) then the RolapAxis, materializes
its content (possibly greatly increasing memory usage).
One could imagine that all data returned could be dynamically
generated since JDBC ResultSets are basically iterators, and all
subsequent evaluations could also return Iterables that
encapsulate the operations on the data. This may be more than is
needed for all but the largest datasets. Right now, Mondrian only
targets crossjoin operations and a couple other where Iterables
can be returned rather than a materialized list.
Julian Hyde wrote:
>> Luis Canals wrote:
>> After taking a look at SqlMemberSource, we have discovered that all
>> elementes from a ResultSet from a JDBC query are loading into memory
>> and stored into an array of Member objects....
>> As we are wanting to deal with very very very big dimensions, the
>> first thing we need is not to overload the memory, so we are thinking
>> of changing the type returned by SqlMemberSource.getMembers from a
>> Member to a List<Member> to be able to change it, in future, to a
>> more intelligent List supported by an Iterator which takes elements
>> from ResultSet non demand.
>> Yes, we know this is going to move us to change a lot elements into
>> code, but using a List or an Iterator instead of an array of Members
>> is, in almost all places, a better approach.
>> We would like to know your oppinions and suggestions.
> Sounds like a good idea. Further, I have found that passing in a list is
> often better than returning a list, because this way the caller gets to have
> a say how the list is represented; and can easily make multiple calls and
> append to an existing list.
> That said, I think Will is making some sweeping changes in that area which
> he has not yet checked in. Will, please comment whether Luis's proposed
> change is likely to conflict with your pending change. If so, I will ask
> Luis to make the change only after you have checked in.
> It is possible that this change affects public APIs. Please post a summary
> of any changes to public APIs (or which people might conceivably have
> implemented/overridden) after you have made your change. I will add this to
> the release notes.
> When you have completd this API change and come to use novel implementations
> of lists, note that Richard Emberson has done some work to (a) use
> memory-sensitive lists, and (b) use iterators. Please refer to his work and
> use the same data structures if possible.
> Mondrian mailing list
> Mondrian at pentaho.org
Quis custodiet ipsos custodes:
This email message is for the sole use of the intended recipient(s) and
may contain confidential information. Any unauthorized review, use,
disclosure or distribution is prohibited. If you are not the intended
recipient, please contact the sender by reply email and destroy all
copies of the original message.
More information about the Mondrian