<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML xmlns:o = "urn:schemas-microsoft-com:office:office"><HEAD>
<META http-equiv=Content-Type content="text/html; charset=iso-8859-1">
<STYLE type=text/css>DIV {
MARGIN: 0px
}
</STYLE>
<META content="MSHTML 6.00.5730.11" name=GENERATOR></HEAD>
<BODY>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=343513209-21032007>Julian,</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=343513209-21032007></SPAN></FONT> </DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=343513209-21032007>The last few weeks, I have been busy thinking about
ways to contribute a test suite that works as an insurance policy for me.
Since I can only contribute a small part of my time to mondrian, I cannot watch
each change to ensure my application is still working. All extra
suggestions you have made are definitely very
helpful. </SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=343513209-21032007></SPAN></FONT> </DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=343513209-21032007>1) Test suite :</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=343513209-21032007></SPAN></FONT> </DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=343513209-21032007>My application now has a schema of more than 5000
lines. It uses many (almost all) features that are present in
mondrian. And it combines them, in a way I am not sure all of them are
covered by existing regression tests. It does also things that are not
covered by the foodmart test case (like combining two cubes of identical layout
in one virtual cube).</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=343513209-21032007></SPAN></FONT> </DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=343513209-21032007>I think I will try to do three things
:</SPAN></FONT></DIV>
<UL dir=ltr>
<LI>
<DIV align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=343513209-21032007>Try to check against the foodmart database whether
the features I use are supported by a test, if not, add one to the standard
regression test suite. This is a hard one, since most features are
already tested, but not in every combination with other features. I
noticed in the past, most thing that silently fail in my application (without
triggering the regression tests) are mostly due to combination of many facts
(virtual cubes, properties, user defined functions, cache turned on/off,
complex format expressions,...)<BR></SPAN></FONT></DIV></LI>
<LI>
<DIV align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=343513209-21032007>Create a regression test suite that runs
here, using my schema and my database. This in combination with
continuous integration could alarm me if anything is
failing.<BR></SPAN></FONT></DIV></LI>
<LI>
<DIV align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=343513209-21032007>Try to contribute a database and schema to test
against a dynamic database. While the test will never be able to
simulate a real dynamic environment, I will try to change the database in
between two queries. I will try out the change listener plug-in, in
order to see if things are flushing and results are correct. Maybe a
good place to test your new cache control part.<BR><BR>I think I would
implement it using an database that is copied before the test (so the test
starts with the same view each time) and doing jdbc calls to fill the
database. If you have a better idea, please tell me. I am not sure
what kind of database I should choose (access, ...). I noticed that the
test suite contains something to load a DB from csv or xml, but I am not sure
how this is working and how I can modify the data on the
fly?</SPAN></FONT></DIV></LI></UL>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=343513209-21032007>2) QA partner</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=343513209-21032007></SPAN></FONT> </DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=343513209-21032007>Sounds like a good idea</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=343513209-21032007></SPAN></FONT> </DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=343513209-21032007>3) Continuous integration</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=343513209-21032007></SPAN></FONT> </DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=343513209-21032007>Looking forward to see this, I hope the setup
environment for e.g. CruiseControl is made available, so I can use it to setup
up an environment to test my own cube and regression tests.</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=343513209-21032007></SPAN></FONT> </DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=343513209-21032007>4) Dynamic database</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=343513209-21032007></SPAN></FONT> </DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=343513209-21032007>Indeed, this is very complex to test. But one
needs to start somewhere, so I will try to contribute a test environment (see
1)) that will behave the same way as my application does.</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=343513209-21032007></SPAN></FONT> </DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=343513209-21032007>And yes, there are many pitfalls, but due to a specific
setup of the database most things are working for me, with some known issues
that are acceptable in my kind of application.</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=343513209-21032007></SPAN></FONT> </DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=343513209-21032007>Most things I did for 2.3 are about making it possible
to :</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=343513209-21032007></SPAN></FONT> </DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=343513209-21032007>* Let cubes not maintaining cache, not mess with the
global cache.</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=343513209-21032007></SPAN></FONT> </DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=343513209-21032007>* Cubes loading cache, will not interfere with
concurrent threads (only check into global cache when all other threads are
done).</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=343513209-21032007></SPAN></FONT> </DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=343513209-21032007>* Being able to flush both aggregate and member cache
using a plug-in.</SPAN></FONT><FONT face=Arial color=#0000ff size=2><SPAN
class=343513209-21032007> </SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=343513209-21032007></SPAN></FONT><FONT face=Arial color=#0000ff
size=2><SPAN class=343513209-21032007></SPAN></FONT> </DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=343513209-21032007>Bart</SPAN></FONT></DIV><FONT
style="BACKGROUND-COLOR: #ffffff" face=Arial color=#0000ff size=2></FONT><FONT
style="BACKGROUND-COLOR: #ffffff" face=Arial color=#0000ff size=2></FONT><FONT
style="BACKGROUND-COLOR: #ffffff" face=Arial color=#0000ff size=2></FONT><FONT
style="BACKGROUND-COLOR: #ffffff" face=Arial color=#0000ff size=2></FONT><FONT
style="BACKGROUND-COLOR: #ffffff" face=Arial color=#0000ff size=2></FONT><FONT
style="BACKGROUND-COLOR: #ffffff" face=Arial color=#0000ff size=2></FONT><FONT
style="BACKGROUND-COLOR: #ffffff" face=Arial color=#0000ff size=2></FONT><FONT
style="BACKGROUND-COLOR: #ffffff" face=Arial color=#0000ff size=2></FONT><FONT
style="BACKGROUND-COLOR: #ffffff" face=Arial color=#0000ff size=2></FONT><FONT
style="BACKGROUND-COLOR: #ffffff" face=Arial color=#0000ff size=2></FONT><FONT
style="BACKGROUND-COLOR: #ffffff" face=Arial color=#0000ff size=2></FONT><FONT
style="BACKGROUND-COLOR: #ffffff" face=Arial color=#0000ff size=2></FONT><FONT
style="BACKGROUND-COLOR: #ffffff" face=Arial color=#0000ff size=2></FONT><FONT
style="BACKGROUND-COLOR: #ffffff" face=Arial color=#0000ff size=2></FONT><FONT
style="BACKGROUND-COLOR: #ffffff" face=Arial color=#0000ff size=2></FONT><FONT
style="BACKGROUND-COLOR: #ffffff" face=Arial color=#0000ff size=2></FONT><FONT
style="BACKGROUND-COLOR: #ffffff" face=Arial color=#0000ff size=2></FONT><FONT
style="BACKGROUND-COLOR: #ffffff" face=Arial color=#0000ff size=2></FONT><FONT
style="BACKGROUND-COLOR: #ffffff" face=Arial color=#0000ff size=2></FONT><FONT
style="BACKGROUND-COLOR: #ffffff" face=Arial color=#0000ff size=2></FONT><FONT
style="BACKGROUND-COLOR: #ffffff" face=Arial color=#0000ff size=2></FONT><FONT
style="BACKGROUND-COLOR: #ffffff" face=Arial color=#0000ff size=2></FONT><FONT
style="BACKGROUND-COLOR: #ffffff" face=Arial color=#0000ff size=2></FONT><FONT
style="BACKGROUND-COLOR: #ffffff" face=Arial color=#0000ff size=2></FONT><FONT
style="BACKGROUND-COLOR: #ffffff" face=Arial color=#0000ff size=2></FONT><FONT
style="BACKGROUND-COLOR: #ffffff" face=Arial color=#0000ff size=2></FONT><FONT
style="BACKGROUND-COLOR: #ffffff" face=Arial color=#0000ff size=2></FONT><FONT
style="BACKGROUND-COLOR: #ffffff" face=Arial color=#0000ff size=2></FONT><FONT
style="BACKGROUND-COLOR: #ffffff" face=Arial color=#0000ff size=2></FONT><FONT
style="BACKGROUND-COLOR: #ffffff" face=Arial color=#0000ff size=2></FONT><FONT
style="BACKGROUND-COLOR: #ffffff" face=Arial color=#0000ff size=2></FONT><FONT
style="BACKGROUND-COLOR: #ffffff" face=Arial color=#0000ff size=2></FONT><FONT
style="BACKGROUND-COLOR: #ffffff" face=Arial color=#0000ff size=2></FONT><FONT
style="BACKGROUND-COLOR: #ffffff" face=Arial color=#0000ff size=2></FONT><FONT
style="BACKGROUND-COLOR: #ffffff" face=Arial color=#0000ff size=2></FONT><FONT
style="BACKGROUND-COLOR: #ffffff" face=Arial color=#0000ff size=2></FONT><FONT
style="BACKGROUND-COLOR: #ffffff" face=Arial color=#0000ff size=2></FONT><FONT
style="BACKGROUND-COLOR: #ffffff" face=Arial color=#0000ff size=2></FONT><FONT
style="BACKGROUND-COLOR: #ffffff" face=Arial color=#0000ff size=2></FONT><BR>
<DIV class=OutlookMessageHeader lang=en-us dir=ltr align=left>
<HR tabIndex=-1>
<FONT face=Tahoma size=2><B>From:</B> mondrian-bounces@pentaho.org
[mailto:mondrian-bounces@pentaho.org] <B>On Behalf Of </B>Julian
Hyde<BR><B>Sent:</B> vrijdag 16 maart 2007 18:04<BR><B>To:</B> 'Mondrian
developer mailing list'<BR><B>Subject:</B> RE: [Mondrian] Multithreading
etc<BR></FONT><BR></DIV>
<DIV></DIV>
<DIV dir=ltr align=left><SPAN class=836262316-16032007><FONT face=Verdana
color=#000080 size=2>If changes to mondrian are breaking your application, I
sympathize. How can we prevent that from happening? Unless we restrict ourselves
to trivial enhancements, the we obviously need to test the new functionality
against existing apps, or at least tests which exercise existing apps'
requirements.</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=836262316-16032007><FONT face=Verdana
color=#000080 size=2></FONT></SPAN> </DIV>
<DIV dir=ltr align=left><SPAN class=836262316-16032007><FONT face=Verdana
color=#000080 size=2>Ideally, these tests would be in the standard regression
suite. But since some tests are too complicated to be in the standard suite, to
testing which cannot be done nightly at least has to be done once per
release.</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=836262316-16032007><FONT face=Verdana
color=#000080 size=2></FONT></SPAN> </DIV>
<DIV dir=ltr align=left><SPAN class=836262316-16032007><SPAN
class=836262316-16032007><FONT face=Verdana color=#000080 size=2>We already have
a process in place for much of this. For example:</FONT></SPAN></DIV><SPAN
class=836262316-16032007><FONT face=Verdana color=#000080 size=2>
<UL dir=ltr>
<LI>
<DIV align=left><SPAN class=836262316-16032007><FONT face=Verdana
color=#000080 size=2>Developers ensure that code changes don't break the
regression suite.</FONT></SPAN></DIV>
<LI>
<DIV align=left><SPAN class=836262316-16032007>I run the regression suite
nightly, in a wide set of configurations, and let the developers know next day
if things break.</SPAN></DIV>
<LI>
<DIV align=left><SPAN class=836262316-16032007><SPAN
class=836262316-16032007><FONT face=Verdana color=#000080 size=2>All code
changes - bug fixes, enhancements and ad hoc feature - must be accompanied by
a regression test which exercises the change. (That means it should fail if
the change is not present.)</FONT></SPAN></SPAN></DIV></LI></UL></FONT></SPAN>
<DIV dir=ltr align=left><SPAN class=836262316-16032007><FONT face=Verdana
color=#000080 size=2>The extra things we need to do:</FONT></SPAN></DIV>
<UL dir=ltr>
<LI>
<DIV align=left><SPAN class=836262316-16032007><FONT face=Verdana
color=#000080 size=2>If you are using mondrian in your apps, contribute test
suites which exercise your app's functionality. LucidEra have already done
thist (ClearTestSuite) and Thomson-Medstat are working on it. Sure it's a lot
of work, but it's less work than taking a new release where things have
stopped working. It's an insurance policy.</FONT></SPAN></DIV>
<LI>
<DIV align=left><SPAN class=836262316-16032007><FONT face=Verdana
color=#000080 size=2>Would it help if we invented the notion of 'QA partner'
companies? We could run through the test suite of each QA partner before each
release, as a pre-condition for making that release.</FONT></SPAN></DIV>
<LI>
<DIV align=left><SPAN class=836262316-16032007><FONT face=Verdana
color=#000080 size=2>Write better tests for new features. Complex features
require complex tests. </FONT></SPAN></DIV>
<LI>
<DIV align=left><SPAN class=836262316-16032007><FONT face=Verdana
color=#000080 size=2>Set up continuous integration (E.g. CruiseControl) to
ensure that the code always builds and compiles. Thiyagu is already working on
this.</FONT></SPAN></DIV></LI></UL>
<DIV dir=ltr align=left><SPAN class=836262316-16032007><FONT face=Verdana
color=#000080 size=2>Bart, On point #3: <SPAN class=836262316-16032007><FONT
face=Verdana color=#000080 size=2>The feature you are interested in, for working
on top of dynamic databases, is VERY complex to test. We have spent a lot of
time discussing how to implement this feature, but very little time designing a
testing infrastructure. It should be little surprise that the feature is fragile
at this point.</FONT></SPAN></FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=836262316-16032007><FONT face=Verdana
color=#000080 size=2><SPAN
class=836262316-16032007></SPAN></FONT></SPAN> </DIV>
<DIV dir=ltr align=left><SPAN class=836262316-16032007><FONT face=Verdana
color=#000080 size=2><SPAN class=836262316-16032007></SPAN></FONT></SPAN><SPAN
class=836262316-16032007><FONT face=Verdana color=#000080
size=2>Julian</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=836262316-16032007><FONT face=Verdana
color=#000080 size=2></FONT></SPAN> </DIV></SPAN><FONT face=Verdana
color=#000080 size=2></FONT><FONT face=Verdana color=#000080 size=2></FONT><FONT
face=Verdana color=#000080 size=2></FONT><FONT face=Verdana color=#000080
size=2></FONT><FONT face=Verdana color=#000080 size=2></FONT><FONT face=Verdana
color=#000080 size=2></FONT><FONT face=Verdana color=#000080 size=2></FONT><FONT
style="BACKGROUND-COLOR: #ffffff" face=Verdana color=#000080 size=2></FONT><FONT
style="BACKGROUND-COLOR: #ffffff" face=Verdana color=#000080 size=2></FONT><BR>
<BLOCKQUOTE
style="PADDING-LEFT: 5px; MARGIN-LEFT: 5px; BORDER-LEFT: #000080 2px solid; MARGIN-RIGHT: 0px">
<DIV class=OutlookMessageHeader lang=en-us dir=ltr align=left>
<HR tabIndex=-1>
<FONT face=Tahoma size=2><B>From:</B> mondrian-bounces@pentaho.org
[mailto:mondrian-bounces@pentaho.org] <B>On Behalf Of </B>Pappyn
Bart<BR><B>Sent:</B> Tuesday, March 13, 2007 1:34 AM<BR><B>To:</B> Mondrian
developer mailing list<BR><B>Subject:</B> RE: [Mondrian] Multithreading
etc<BR></FONT><BR></DIV>
<DIV></DIV>
<DIV dir=ltr align=left><SPAN class=890593607-13032007><FONT face=Arial
color=#0000ff size=2>
<P class=MsoNormal style="MARGIN: 0cm 0cm 0pt"><SPAN
style="FONT-SIZE: 10pt; COLOR: blue; FONT-FAMILY: Arial">Hi
Michael,</SPAN><o:p></o:p></P>
<P class=MsoNormal style="MARGIN: 0cm 0cm 0pt"><FONT size=3><FONT
color=#000000><FONT
face="Times New Roman"> <o:p></o:p></FONT></FONT></FONT></P>
<P class=MsoNormal style="MARGIN: 0cm 0cm 0pt"><SPAN
style="FONT-SIZE: 10pt; COLOR: blue; FONT-FAMILY: Arial">I don't see any
problems for you to make changes to mondrian. But I have some
concerns:</SPAN><o:p></o:p></P>
<P class=MsoNormal style="MARGIN: 0cm 0cm 0pt"><FONT size=3><FONT
color=#000000><FONT
face="Times New Roman"> <o:p></o:p></FONT></FONT></FONT></P>
<P class=MsoNormal style="MARGIN: 0cm 0cm 0pt"><SPAN
style="FONT-SIZE: 10pt; COLOR: blue; FONT-FAMILY: Arial">I think the changes
you are about to make are quite huge and will have an impact of how mondrian
will behave. Since this is the first source contribution you are about
to make, I urge you not to check anything into perforce before it is actually
working and passing all regression tests.</SPAN><o:p></o:p></P>
<P class=MsoNormal style="MARGIN: 0cm 0cm 0pt"><FONT size=3><FONT
color=#000000><FONT
face="Times New Roman"> <o:p></o:p></FONT></FONT></FONT></P>
<P class=MsoNormal style="MARGIN: 0cm 0cm 0pt"><SPAN
style="FONT-SIZE: 10pt; COLOR: blue; FONT-FAMILY: Arial">I think most
developers of mondrian have ongoing projects that are using mondrian, I think
this is</SPAN><o:p></o:p></P>
<P class=MsoNormal style="MARGIN: 0cm 0cm 0pt"><SPAN
style="FONT-SIZE: 10pt; COLOR: blue; FONT-FAMILY: Arial">becoming more and
more an important issue.</SPAN><o:p></o:p></P>
<P class=MsoNormal style="MARGIN: 0cm 0cm 0pt"><FONT size=3><FONT
color=#000000><FONT
face="Times New Roman"> <o:p></o:p></FONT></FONT></FONT></P>
<P class=MsoNormal style="MARGIN: 0cm 0cm 0pt"><SPAN
style="FONT-SIZE: 10pt; COLOR: blue; FONT-FAMILY: Arial">For me: it must be
able to flush aggregates and member cache using the plug-in and cubes
not</SPAN><FONT face="Times New Roman" color=#000000 size=3> </FONT><SPAN
style="FONT-SIZE: 10pt; COLOR: blue; FONT-FAMILY: Arial">maintaining cache
should be able to load their own data, without messing with global
cache.</SPAN><o:p></o:p></P>
<P class=MsoNormal style="MARGIN: 0cm 0cm 0pt"><FONT size=3><FONT
color=#000000><FONT
face="Times New Roman"> <o:p></o:p></FONT></FONT></FONT></P>
<P class=MsoNormal style="MARGIN: 0cm 0cm 0pt"><SPAN
style="FONT-SIZE: 10pt; COLOR: blue; FONT-FAMILY: Arial">And since a dynamic
database cannot easily be simulated in a regression test, I think if you
are</SPAN><FONT face="Times New Roman" color=#000000 size=3> </FONT><SPAN
style="FONT-SIZE: 10pt; COLOR: blue; FONT-FAMILY: Arial">serious about
tackling the read-consistency for near-real-time data, you need a realistic
(dynamic) database to test against. And the database must be large
enough to be able to see realistic performance.<SPAN
style="mso-spacerun: yes"> </SPAN>It is also advised to test with
virtual cubes, cubes maintaining cache (with aggregate tables) in combination
with cubes not maintaining cache (without aggregate tables), shared dimensions
and so on…</SPAN><o:p></o:p></P>
<P class=MsoNormal style="MARGIN: 0cm 0cm 0pt"><FONT size=3><FONT
color=#000000><FONT
face="Times New Roman"> <o:p></o:p></FONT></FONT></FONT></P>
<P class=MsoNormal style="MARGIN: 0cm 0cm 0pt"><SPAN
style="FONT-SIZE: 10pt; COLOR: blue; FONT-FAMILY: Arial">I released my project
4 weeks ago, not even using the latest version of perforce, since at a given
point mondrian-head was completely broken for me. While I know software
always has some bugs that need to be patched, things that are not tested and
are breaking mondrian should not be checked in. All too often I had to
sync with perforce to solve a bug and this ended up in a nightmare, spending
most of my time finding out what change was causing mondrian to
break.</SPAN><o:p></o:p></P>
<P class=MsoNormal style="MARGIN: 0cm 0cm 0pt"><FONT size=3><FONT
color=#000000><FONT
face="Times New Roman"> <o:p></o:p></FONT></FONT></FONT></P>
<P class=MsoNormal style="MARGIN: 0cm 0cm 0pt"><SPAN
style="FONT-SIZE: 10pt; COLOR: blue; FONT-FAMILY: Arial">When mondrian 2.3
will be released, it is most likely that there will be some 2.3.x version
containing some patches. I think it must be possible to make those
patches without having to drag new huge features along.</SPAN><o:p></o:p></P>
<P class=MsoNormal style="MARGIN: 0cm 0cm 0pt"><FONT size=3><FONT
color=#000000><FONT
face="Times New Roman"> <o:p></o:p></FONT></FONT></FONT></P>
<P class=MsoNormal style="MARGIN: 0cm 0cm 0pt"><SPAN
style="FONT-SIZE: 10pt; COLOR: blue; FONT-FAMILY: Arial">Bart</SPAN><o:p></o:p></P></FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=890593607-13032007><FONT face=Arial
color=#0000ff size=2></FONT></SPAN> </DIV>
<DIV dir=ltr align=left>
<HR tabIndex=-1>
</DIV>
<DIV dir=ltr align=left><FONT face=Tahoma size=2><B>From:</B>
mondrian-bounces@pentaho.org [mailto:mondrian-bounces@pentaho.org] <B>On
Behalf Of </B>michael bienstein<BR><B>Sent:</B> vrijdag 9 maart 2007
11:36<BR><B>To:</B> Mondrian developer mailing list<BR><B>Subject:</B>
[Mondrian] Multithreading etc<BR></FONT><BR></DIV>
<DIV></DIV>
<DIV
style="FONT-SIZE: 12pt; FONT-FAMILY: times new roman, new york, times, serif">
<DIV>I sent this to the list but it gets bounced because I attached the code
in a zip file. How do I send code through without checking it in because
it is still orthogonal to the codebase?<BR><BR>Michael<BR>----<BR><BR>Well, I
have code that works for multi-threading infrastructure so I would like to
know if it is worth continuing with this or not.<BR><BR>As for ROLLUP/CUBE my
thoughts are:<BR>1) Either we keep the codebase simple by sticking to a
standard (SQL2003) even if this standard is not yet implemented widely and
certain databases have better special features than others, or we allow a
per-database SQL generation system. The argument for the second makes
sense only if the developer resources to write and maintain each dialect comes
from the database vendor or their community. Mondrian is probably at a
stage that such discussions can be undertaken with the database vendors.<BR>2)
Architecturally this implies loading multiple Aggregations from one SQL
query. That requires a rethink of the way the cell cache loading is done
because at the moment an Aggregation is loaded one at a time and in a
synchronized block on the Aggregation. Similar concerns have to be dealt
with for in-memory rollups. I think that synchronized is too
forceful. We need something more like a Lock from java.util.concurrent
so we can do tryLock(). Look at the TxLock idea I have in the code I'm
attaching.<BR><BR>As for multi-threading:<BR>I have only written most of the
base infrastructure, not the cell loading. To integrate would require a
significant amount of work in Mondrian's code to pass all interaction with
Mondrian through TxSystem.runWithTx(). <BR><BR>Basic concerns are:<BR>
<DIV class=MsoNormal
style="MARGIN-LEFT: 36pt; TEXT-INDENT: -18pt"><SPAN><SPAN>1)<SPAN
style="FONT-WEIGHT: normal; FONT-SIZE: 7pt; LINE-HEIGHT: normal; FONT-STYLE: normal; FONT-VARIANT: normal; font-size-adjust: none; font-stretch: normal">
</SPAN></SPAN></SPAN><SPAN>Threads should be able to share data related to the
request across the threads.</SPAN></DIV>
<DIV class=MsoNormal
style="MARGIN-LEFT: 36pt; TEXT-INDENT: -18pt"><SPAN><SPAN>2)<SPAN
style="FONT-WEIGHT: normal; FONT-SIZE: 7pt; LINE-HEIGHT: normal; FONT-STYLE: normal; FONT-VARIANT: normal; font-size-adjust: none; font-stretch: normal">
</SPAN></SPAN></SPAN><SPAN>A Thread should be loaned to a request and returned
in a way that is well-nigh fail-safe (i.e. the thread shouldn’t keep running
of the request fails in some way).</SPAN></DIV>
<DIV class=MsoNormal
style="MARGIN-LEFT: 36pt; TEXT-INDENT: -18pt"><SPAN><SPAN>3)<SPAN
style="FONT-WEIGHT: normal; FONT-SIZE: 7pt; LINE-HEIGHT: normal; FONT-STYLE: normal; FONT-VARIANT: normal; font-size-adjust: none; font-stretch: normal">
</SPAN></SPAN></SPAN><SPAN>We should be able in a parameter of some sort
decide to NOT use threads at all.</SPAN></DIV>
<DIV class=MsoNormal
style="MARGIN-LEFT: 36pt; TEXT-INDENT: -18pt"><SPAN><SPAN>4)<SPAN
style="FONT-WEIGHT: normal; FONT-SIZE: 7pt; LINE-HEIGHT: normal; FONT-STYLE: normal; FONT-VARIANT: normal; font-size-adjust: none; font-stretch: normal">
</SPAN></SPAN></SPAN><SPAN>The number of threads should be
configurable.</SPAN></DIV>
<DIV class=MsoNormal
style="MARGIN-LEFT: 36pt; TEXT-INDENT: -18pt"><SPAN><SPAN>5)<SPAN
style="FONT-WEIGHT: normal; FONT-SIZE: 7pt; LINE-HEIGHT: normal; FONT-STYLE: normal; FONT-VARIANT: normal; font-size-adjust: none; font-stretch: normal">
</SPAN></SPAN></SPAN><SPAN>There should be an independence from the rest of
the code base.</SPAN></DIV>
<DIV class=MsoNormal
style="MARGIN-LEFT: 36pt; TEXT-INDENT: -18pt"><SPAN><SPAN>6)<SPAN
style="FONT-WEIGHT: normal; FONT-SIZE: 7pt; LINE-HEIGHT: normal; FONT-STYLE: normal; FONT-VARIANT: normal; font-size-adjust: none; font-stretch: normal">
</SPAN></SPAN></SPAN><SPAN>We should be able to make use of custom thread
pools or use managed thread pools from the application server.</SPAN></DIV>
<DIV class=MsoNormal
style="MARGIN-LEFT: 36pt; TEXT-INDENT: -18pt"><SPAN><SPAN>7)<SPAN
style="FONT-WEIGHT: normal; FONT-SIZE: 7pt; LINE-HEIGHT: normal; FONT-STYLE: normal; FONT-VARIANT: normal; font-size-adjust: none; font-stretch: normal">
</SPAN></SPAN></SPAN><SPAN>Then there is a relatively minor issue with
read-consistency for near-real-time data that turns out to be a real
head-ache.<SPAN> </SPAN>This can be done by either: using the
transaction semantics of the underlying data store <U>or</U> modifying all SQL
requests and cache interactions with a timestamp and/or transaction id of some
sort.<SPAN> </SPAN>E.g. when an MDX requests begins it asks the
underlying data store for the id of the last completed transaction that
modified data and keeps this in a request-scope available to all
threads.<SPAN> </SPAN>Then it appends “changedTxId <=
${lastTxIdWhenFirstEntered}” to each WHERE clause.<SPAN> </SPAN>If
however we use the underlying data store’s transactions then we must keep open
the JDBC Connection for the duration of the request reusing it on the same
thread for each interaction with that data store.</SPAN></DIV>
<DIV class=MsoNormal><SPAN>Now, I think that the best way to take advantage of
multiple threads in the storage system is NOT launching multiple SQLs on the
same star schema but different aggregations but rather to use
<U>partitioning</U> of data.<SPAN> </SPAN>That is to segment the cell
data (and maybe dimension data) based on values of certain
columns.<SPAN> </SPAN>For example year<2007 and year=2007
in<SPAN> </SPAN>two different partitions.<SPAN> </SPAN>This can be
introduced slowly by simply making a RolapStar one Partition for the
moment.<SPAN> </SPAN>Having said that aggregation tables are also a type
of Partition and hitting two of them at once should be quite
easy.</SPAN></DIV>So the design I am introducing has the following
features:<BR>1) A scope for "request" or "interaction" that is larger than the
Thread that begins it. Since this is similar to a transaction I've
called it a Tx. See the mondrian.tx package. Each sub-system in
Mondrian can enlist a representation of itself in the Tx.<BR>2) Break up the
different tasks performed into Task objects that can be run potentially in
parallel. Allow a set of Tasks to be tied to the same Thread so that the
same JDBC Connection can be used for all of them for read-consistency and
cleaned up at the end of the Tx. This is done declaratively so the
implementation can be changed easily. The implementation can also ensure
that the J2EE context is passed onto separate threads (JNDI, context class
loader etc).<BR>3) A system of fail-quick locks at the Tx scope rather than
just Thread scope. <BR><BR>If this is worth persuing as a design for the
next version then good. If not I'll stop
now.<BR><BR>Michael</DIV></DIV><BR>
<HR SIZE=1>
Découvrez une nouvelle façon d'obtenir des réponses à toutes vos questions !
Profitez des connaissances, des opinions et des expériences des internautes
sur <A
href="http://fr.rd.yahoo.com/evt=42054/*http://fr.answers.yahoo.com">Yahoo!
Questions/Réponses</A>.
<BR>______________________________________________________________________<BR>This
email has been scanned by the Email Security
System.<BR>______________________________________________________________________<BR></BLOCKQUOTE><BR>______________________________________________________________________<BR>This
email has been scanned by the Email Security
System.<BR>______________________________________________________________________<BR></BODY></HTML>