[Mondrian] Testing a Hive dialect

Fu Hongwei fuhongwei2006 at 163.com
Sun Feb 13 10:15:14 EST 2011


Thanks, please commit the files for me. Hive-1922 contains some stuff I consider easier to fix on the Hive side, but, yes, I think it's better if we have a non hive patch version.

The failures are caused by some small problems. It's running beautifully now.

I've fixed the join issue. You would also want to add a VM parameter when running the tests or it will report some errors. 
-javax.xml.parsers.DocumentBuilderFactory=com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderFactoryImpl 

Can I have some kind of commit right? It looks like I have a lot of files to submit.

BTW, could somebody tell me how to make a patch in perforce like in svn? Thanks.
2011-02-13 



Fu Hongwei 



发件人: Julian Hyde 
发送时间: 2011-02-12  01:59:33 
收件人: 'Mondrian developer mailing list' 
抄送: 
主题: RE: RE: [Mondrian] Testing a Hive dialect 
 
If it is difficult to fix HIVE-1922, we could possibly workaround issue by changing Dialect.generateOrderItem. (Every dialect tends to have different rules for how to generate an ORDER BY clause -- order by column name, order by ordinal, order by expression, order by ordinal only when applied to a set operation such as union, etc. -- so there's no harm having yet another behavior.)

Someone also mentioned that Hive only supports ANSI join syntax 'FROM t1 JOIN t2 ON t1.x = t2.y', whereas mondrian only generates 'FROM t1, t2 WHERE t1.x = t2.y'. Is this still an issue? We would need to fix either Hive or Mondrian's dialect. Mondrian's dialect it probably easier.

Since other people have expressed an interest in a Hive dialect, it would be useful if you check in what you have right now, even though there are many failures. Send me the files and I will check them in.

Julian





From: mondrian-bounces at pentaho.org [mailto:mondrian-bounces at pentaho.org] On Behalf Of Fu Hongwei
Sent: Friday, February 11, 2011 3:53 AM
To: 'Mondrian developer mailing list'
Subject: Re: RE: [Mondrian] Testing a Hive dialect


Hi Julian,
Thanks for the prompt reply.
The test is running on Hive 0.7.0, but a patch will be needed. I've started a jira on Hive and will submit it soon too.
https://issues.apache.org/jira/browse/HIVE-1922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12987548#action_12987548 

There are too many failures in the test suite.  I'm still working on it, but I will submit the DialectTest.java and HiveDialect.java, so people interested can work on it together.
I think the point is first to make it running then speed it up.
2011-02-11 



Fu Hongwei 



发件人: Julian Hyde 
发送时间: 2011-02-11  00:19:26 
收件人: 'Mondrian developer mailing list' 
抄送: 
主题: RE: [Mondrian] Testing a Hive dialect 
I am well aware of the compromises with Hive. There is certainly an impedance mismatch between hadoop and the real-time analysis, and that is reflected in query response time. It is still useful  to have a Hive dialect, because as you say, Hive is improving all the time. And Pentaho is thinking about ways to bridge the impedance mismatch.

Can please you attach your code to the jira case as a patch? I will submit it.

Also please attach the output of the test suite, and describe the version of Hive you are running against. That will be a reference point for others who are working on Hive.

Julian




From: mondrian-bounces at pentaho.org [mailto:mondrian-bounces at pentaho.org] On Behalf Of Fu Hongwei
Sent: Thursday, February 10, 2011 1:29 AM
To: mondrian
Subject: [Mondrian] Testing a Hive dialect 


Hi,
I'm new to the community. I don't know who to ask this for. 
I've written a Hive dialect for mondrian and been testing it.
The DialectTest is passed, but it's still failing other tests.

From what I can see it's still a long way to make an integration
of Hive and Mondrian practical.
1. It's really slow. It takes 1 or 2 days to run through all the tests, 
a low end estimation. The latency is too big for most applications.
It might take some major architectural change on the Hive side to solve
this problem.
2. Hive ql is still at a rather immature stage, partly because it's not 
really intended to be a full featured relational database. There are some
 bugs too, like the join behavior is incorrect at the current trunk.

I know there is already a jira about it.
http://jira.pentaho.com/browse/MONDRIAN-789 
How can I submit the code? Thanks
2011-02-10 



Fu Hongwei 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.pentaho.org/pipermail/mondrian/attachments/20110213/9349c29b/attachment.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: SqlQuery.java
Type: application/octet-stream
Size: 29830 bytes
Desc: not available
Url : http://lists.pentaho.org/pipermail/mondrian/attachments/20110213/9349c29b/attachment.obj 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: DialectTest.java
Type: application/octet-stream
Size: 33411 bytes
Desc: not available
Url : http://lists.pentaho.org/pipermail/mondrian/attachments/20110213/9349c29b/attachment-0001.obj 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: FoodMart.xml
Type: application/octet-stream
Size: 34917 bytes
Desc: not available
Url : http://lists.pentaho.org/pipermail/mondrian/attachments/20110213/9349c29b/attachment-0002.obj 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: HiveDialect.java
Type: application/octet-stream
Size: 5181 bytes
Desc: not available
Url : http://lists.pentaho.org/pipermail/mondrian/attachments/20110213/9349c29b/attachment-0003.obj 


More information about the Mondrian mailing list