[Mondrian] Thread hangs with RolapStar.clearAggregationRequests

Richard Emberson remberson at edgedynamics.com
Sat Feb 24 16:24:14 EST 2007


QA testing on a linux box with 1.4.2_13 x86 with the
Mondrian and JPivot head, the Java
process pegged at 99% for 2 1/2 hours (before I
was called and I looked into it). Using 'kill -3 pid',
I got the following thread dumps:


INFO   | jvm 1    | 2007/02/24 11:46:10 |      [java] "TP-Processor2" 
daemon prio=1 tid=0x6bab7638 nid=0x781 runnable [0x69433000..0x69435228]
INFO   | jvm 1    | 2007/02/24 11:46:10 |      [java]   at 
java.util.AbstractList$Itr.remove(AbstractList.java:436)
INFO   | jvm 1    | 2007/02/24 11:46:10 |      [java]   at 
java.util.AbstractCollection.remove(AbstractCollection.java:255)
INFO   | jvm 1    | 2007/02/24 11:46:10 |      [java]   at 
mondrian.rolap.RolapStar.clearAggregationRequests(RolapStar.java:578)
INFO   | jvm 1    | 2007/02/24 11:46:10 |      [java]   - locked 
<0x841ec4c8> (a java.util.ArrayList)
INFO   | jvm 1    | 2007/02/24 11:46:10 |      [java]   at 
mondrian.rolap.RolapStar.pushAggregateModificationsToGlobalCache(RolapStar.java:506)
INFO   | jvm 1    | 2007/02/24 11:46:10 |      [java]   - locked 
<0x841ec398> (a mondrian.rolap.RolapStar)
INFO   | jvm 1    | 2007/02/24 11:46:10 |      [java]   at 
mondrian.rolap.RolapCube.pushAggregateModificationsToGlobalCache(RolapCube.java:1127)


INFO   | jvm 1    | 2007/02/24 12:02:20 |      [java] "TP-Processor2" 
daemon prio=1 tid=0x6bab7638 nid=0x781 runnable [0x69433000..0x69435228]
INFO   | jvm 1    | 2007/02/24 12:02:20 |      [java]   at 
java.util.AbstractList$Itr.remove(AbstractList.java:436)
INFO   | jvm 1    | 2007/02/24 12:02:20 |      [java]   at 
java.util.AbstractCollection.remove(AbstractCollection.java:255)
INFO   | jvm 1    | 2007/02/24 12:02:20 |      [java]   at 
mondrian.rolap.RolapStar.clearAggregationRequests(RolapStar.java:578)
INFO   | jvm 1    | 2007/02/24 12:02:20 |      [java]   - locked 
<0x841ec4c8> (a java.util.ArrayList)
INFO   | jvm 1    | 2007/02/24 12:02:20 |      [java]   at 
mondrian.rolap.RolapStar.pushAggregateModificationsToGlobalCache(RolapStar.java:506)
INFO   | jvm 1    | 2007/02/24 12:02:20 |      [java]   - locked 
<0x841ec398> (a mondrian.rolap.RolapStar)
INFO   | jvm 1    | 2007/02/24 12:02:20 |      [java]   at 
mondrian.rolap.RolapCube.pushAggregateModificationsToGlobalCache(RolapCube.java:1127)

INFO   | jvm 1    | 2007/02/24 12:17:29 |      [java] "TP-Processor2" 
daemon prio=1 tid=0x6bab7638 nid=0x781 runnable [0x69433000..0x69435228]
INFO   | jvm 1    | 2007/02/24 12:17:29 |      [java]   at 
mondrian.rolap.RolapStar.clearAggregationRequests(RolapStar.java:578)
INFO   | jvm 1    | 2007/02/24 12:17:29 |      [java]   - waiting to 
lock <0x841ec4c8> (a java.util.ArrayList)
INFO   | jvm 1    | 2007/02/24 12:17:29 |      [java]   at 
mondrian.rolap.RolapStar.pushAggregateModificationsToGlobalCache(RolapStar.java:506)

INFO   | jvm 1    | 2007/02/24 12:29:42 |      [java] "TP-Processor2" 
daemon prio=1 tid=0x6bab7638 nid=0x781 runnable [0x69433000..0x69435228]
INFO   | jvm 1    | 2007/02/24 12:29:42 |      [java]   at 
mondrian.rolap.RolapStar.clearAggregationRequests(RolapStar.java:578)
INFO   | jvm 1    | 2007/02/24 12:29:42 |      [java]   - waiting to 
lock <0x841ec4c8> (a java.util.ArrayList)
INFO   | jvm 1    | 2007/02/24 12:29:42 |      [java]   at 
mondrian.rolap.RolapStar.pushAggregateModificationsToGlobalCache(RolapStar.java:506)
INFO   | jvm 1    | 2007/02/24 12:47:03 |      [java] "TP-Processor2" 
daemon prio=1 tid=0x6bab7638 nid=0x781 runnable [0x69433000..0x69435228]
INFO   | jvm 1    | 2007/02/24 12:47:03 |      [java]   at 
mondrian.rolap.RolapStar.clearAggregationRequests(RolapStar.java:578)
INFO   | jvm 1    | 2007/02/24 12:47:03 |      [java]   - waiting to 
lock <0x841ec4c8> (a java.util.ArrayList)
INFO   | jvm 1    | 2007/02/24 12:47:03 |      [java]   at 
mondrian.rolap.RolapStar.pushAggregateModificationsToGlobalCache(RolapStar.java:506)

INFO   | jvm 1    | 2007/02/24 12:56:11 |      [java] "TP-Processor2" 
daemon prio=1 tid=0x6bab7638 nid=0x781 runnable [0x69433000..0x69435228]
INFO   | jvm 1    | 2007/02/24 12:56:11 |      [java]   at 
java.util.AbstractList.iterator(AbstractList.java:336)
INFO   | jvm 1    | 2007/02/24 12:56:11 |      [java]   at 
java.util.AbstractCollection.remove(AbstractCollection.java:244)
INFO   | jvm 1    | 2007/02/24 12:56:11 |      [java]   at 
mondrian.rolap.RolapStar.clearAggregationRequests(RolapStar.java:578)
INFO   | jvm 1    | 2007/02/24 12:56:11 |      [java]   - locked 
<0x841ec4c8> (a java.util.ArrayList)
INFO   | jvm 1    | 2007/02/24 12:56:11 |      [java]   at 
mondrian.rolap.RolapStar.pushAggregateModificationsToGlobalCache(RolapStar.java:506)


INFO   | jvm 1    | 2007/02/24 12:56:39 |      [java] "TP-Processor2" 
daemon prio=1 tid=0x6bab7638 nid=0x781 runnable [0x69433000..0x69435228]
INFO   | jvm 1    | 2007/02/24 12:56:39 |      [java]   at 
mondrian.rolap.RolapStar.clearAggregationRequests(RolapStar.java:578)
INFO   | jvm 1    | 2007/02/24 12:56:39 |      [java]   - waiting to 
lock <0x841ec4c8> (a java.util.ArrayList)
INFO   | jvm 1    | 2007/02/24 12:56:39 |      [java]   at 
mondrian.rolap.RolapStar.pushAggregateModificationsToGlobalCache(RolapStar.java:506)
INFO   | jvm 1    | 2007/02/24 12:56:39 |      [java]   - locked 
<0x841ec398> (a mondrian.rolap.RolapStar)
INFO   | jvm 1    | 2007/02/24 12:56:39 |      [java]   at 
mondrian.rolap.RolapCube.pushAggregateModificationsToGlobalCache(RolapCube.java:1127)


INFO   | jvm 1    | 2007/02/24 12:56:40 |      [java] "TP-Processor2" 
daemon prio=1 tid=0x6bab7638 nid=0x781 runnable [0x69433000..0x69435228]
INFO   | jvm 1    | 2007/02/24 12:56:40 |      [java]   at 
java.util.AbstractList$Itr.hasNext(AbstractList.java:415)
INFO   | jvm 1    | 2007/02/24 12:56:40 |      [java]   at 
mondrian.rolap.RolapStar.clearAggregationRequests(RolapStar.java:575)
INFO   | jvm 1    | 2007/02/24 12:56:40 |      [java]   - locked 
<0x841ec4c8> (a java.util.ArrayList)
INFO   | jvm 1    | 2007/02/24 12:56:40 |      [java]   at 
mondrian.rolap.RolapStar.pushAggregateModificationsToGlobalCache(RolapStar.java:506)
INFO   | jvm 1    | 2007/02/24 12:56:40 |      [java]   - locked 
<0x841ec398> (a mondrian.rolap.RolapStar)

INFO   | jvm 1    | 2007/02/24 12:56:41 |      [java] "TP-Processor2" 
daemon prio=1 tid=0x6bab7638 nid=0x781 runnable [0x69433000..0x69435228]
INFO   | jvm 1    | 2007/02/24 12:56:41 |      [java]   at 
java.util.AbstractList$Itr.remove(AbstractList.java:436)
INFO   | jvm 1    | 2007/02/24 12:56:41 |      [java]   at 
java.util.AbstractCollection.remove(AbstractCollection.java:255)
INFO   | jvm 1    | 2007/02/24 12:56:41 |      [java]   at 
mondrian.rolap.RolapStar.clearAggregationRequests(RolapStar.java:578)
INFO   | jvm 1    | 2007/02/24 12:56:41 |      [java]   - locked 
<0x841ec4c8> (a java.util.ArrayList)
INFO   | jvm 1    | 2007/02/24 12:56:41 |      [java]   at 
mondrian.rolap.RolapStar.pushAggregateModificationsToGlobalCache(RolapStar.java:506)
INFO   | jvm 1    | 2007/02/24 12:56:41 |      [java]   - locked 
<0x841ec398> (a mondrian.rolap.RolapStar)
INFO   | jvm 1    | 2007/02/24 12:56:41 |      [java]   at 
mondrian.rolap.RolapCube.pushAggregateModificationsToGlobalCache(RolapCube.java:1127)

INFO   | jvm 1    | 2007/02/24 12:56:41 |      [java] "TP-Processor2" 
daemon prio=1 tid=0x6bab7638 nid=0x781 runnable [0x69433000..0x69435228]
INFO   | jvm 1    | 2007/02/24 12:56:41 |      [java]   at 
mondrian.rolap.RolapStar.clearAggregationRequests(RolapStar.java:578)
INFO   | jvm 1    | 2007/02/24 12:56:41 |      [java]   - waiting to 
lock <0x841ec4c8> (a java.util.ArrayList)
INFO   | jvm 1    | 2007/02/24 12:56:41 |      [java]   at 
mondrian.rolap.RolapStar.pushAggregateModificationsToGlobalCache(RolapStar.java:506)
INFO   | jvm 1    | 2007/02/24 12:56:41 |      [java]   - locked 
<0x841ec398> (a mondrian.rolap.RolapStar)
INFO   | jvm 1    | 2007/02/24 12:56:41 |      [java]   at 
mondrian.rolap.RolapCube.pushAggregateModificationsToGlobalCache(RolapCube.java:1127)
INFO   | jvm 1    | 2007/02/24 12:56:42 |      [java] "TP-Processor2" 
daemon prio=1 tid=0x6bab7638 nid=0x781 runnable [0x69433000..0x69435228]
INFO   | jvm 1    | 2007/02/24 12:56:42 |      [java]   at 
java.util.AbstractList$Itr.remove(AbstractList.java:436)
INFO   | jvm 1    | 2007/02/24 12:56:42 |      [java]   at 
java.util.AbstractCollection.remove(AbstractCollection.java:255)
INFO   | jvm 1    | 2007/02/24 12:56:42 |      [java]   at 
mondrian.rolap.RolapStar.clearAggregationRequests(RolapStar.java:578)
INFO   | jvm 1    | 2007/02/24 12:56:42 |      [java]   - locked 
<0x841ec4c8> (a java.util.ArrayList)
INFO   | jvm 1    | 2007/02/24 12:56:42 |      [java]   at 
mondrian.rolap.RolapStar.pushAggregateModificationsToGlobalCache(RolapStar.java:506)
INFO   | jvm 1    | 2007/02/24 12:56:42 |      [java]   - locked 
<0x841ec398> (a mondrian.rolap.RolapStar)
INFO   | jvm 1    | 2007/02/24 12:56:42 |      [java]   at 
mondrian.rolap.RolapCube.pushAggregateModificationsToGlobalCache(RolapCube.java:1127)


INFO   | jvm 1    | 2007/02/24 12:56:43 |      [java] "TP-Processor2" 
daemon prio=1 tid=0x6bab7638 nid=0x781 runnable [0x69433000..0x69435228]
INFO   | jvm 1    | 2007/02/24 12:56:43 |      [java]   at 
java.util.AbstractList$Itr.remove(AbstractList.java:436)
INFO   | jvm 1    | 2007/02/24 12:56:43 |      [java]   at 
java.util.AbstractCollection.remove(AbstractCollection.java:255)
INFO   | jvm 1    | 2007/02/24 12:56:43 |      [java]   at 
mondrian.rolap.RolapStar.clearAggregationRequests(RolapStar.java:578)
INFO   | jvm 1    | 2007/02/24 12:56:43 |      [java]   - locked 
<0x841ec4c8> (a java.util.ArrayList)
INFO   | jvm 1    | 2007/02/24 12:56:43 |      [java]   at 
mondrian.rolap.RolapStar.pushAggregateModificationsToGlobalCache(RolapStar.java:506)
INFO   | jvm 1    | 2007/02/24 12:56:43 |      [java]   - locked 
<0x841ec398> (a mondrian.rolap.RolapStar)
INFO   | jvm 1    | 2007/02/24 12:56:43 |      [java]   at 
mondrian.rolap.RolapCube.pushAggregateModificationsToGlobalCache(RolapCube.java:1127)

INFO   | jvm 1    | 2007/02/24 12:56:43 |      [java] "TP-Processor2" 
daemon prio=1 tid=0x6bab7638 nid=0x781 runnable [0x69433000..0x69435228]
INFO   | jvm 1    | 2007/02/24 12:56:43 |      [java]   at 
java.util.AbstractList$Itr.remove(AbstractList.java:436)
INFO   | jvm 1    | 2007/02/24 12:56:43 |      [java]   at 
java.util.AbstractCollection.remove(AbstractCollection.java:255)
INFO   | jvm 1    | 2007/02/24 12:56:43 |      [java]   at 
mondrian.rolap.RolapStar.clearAggregationRequests(RolapStar.java:578)
INFO   | jvm 1    | 2007/02/24 12:56:43 |      [java]   - locked 
<0x841ec4c8> (a java.util.ArrayList)
INFO   | jvm 1    | 2007/02/24 12:56:43 |      [java]   at 
mondrian.rolap.RolapStar.pushAggregateModificationsToGlobalCache(RolapStar.java:506)
INFO   | jvm 1    | 2007/02/24 12:56:43 |      [java]   - locked 
<0x841ec398> (a mondrian.rolap.RolapStar)
INFO   | jvm 1    | 2007/02/24 12:56:43 |      [java]   at 
mondrian.rolap.RolapCube.pushAggregateModificationsToGlobalCache(RolapCube.java:1127)
INFO   | jvm 1    | 2007/02/24 12:56:43 |      [java]   at 
mondrian.rolap.RolapResult.<init>(RolapResult.java:230)

Based upon the code, this looks like an impossible situation.
Note the times of the thread dumps. At no time was
this thread above the RolapStar.clearAggregationRequests method.
My current explanation it a hotspot compilation error.
Anyone have other thoughts or have seen this before?
Thanks.

Richard

-- 
Quis custodiet ipsos custodes:
This email message is for the sole use of the intended recipient(s) and
may contain confidential information.  Any unauthorized review, use,
disclosure or distribution is prohibited.  If you are not the intended
recipient, please contact the sender by reply email and destroy all
copies of the original message.



More information about the Mondrian mailing list