Eager fetch leads to out of bounds error

[Daryl Stultz]

Hello,

I'm wondering if anyone has seen something like this:

<openjpa-1.2.1-r752877:753278 nonfatal general error>
org.apache.openjpa.persistence.PersistenceException: 5
at org.apache.openjpa.kernel.BrokerImpl.find(BrokerImpl.java:875)
at org.apache.openjpa.kernel.BrokerImpl.find(BrokerImpl.java:774)
at
org.apache.openjpa.jdbc.kernel.JDBCStoreManager.load(JDBCStoreManager.java:982)
at org.apache.openjpa.jdbc.sql.AbstractResult.load(AbstractResult.java:278)
at
org.apache.openjpa.jdbc.sql.SelectImpl$SelectResult.load(SelectImpl.java:2400)
at
org.apache.openjpa.jdbc.meta.strats.RelationFieldStrategy.loadEagerJoin(RelationFieldStrategy.java:541)
at
org.apache.openjpa.jdbc.meta.FieldMapping.loadEagerJoin(FieldMapping.java:807)
...snip
Caused by: java.lang.ArrayIndexOutOfBoundsException: 5
at
org.apache.openjpa.meta.ClassMetaData.getExtraFieldDataIndex(ClassMetaData.java:859)
at
org.apache.openjpa.jdbc.kernel.JDBCStoreManager.initializeState(JDBCStoreManager.java:358)
at
org.apache.openjpa.jdbc.kernel.JDBCStoreManager.initialize(JDBCStoreManager.java:278)
at
org.apache.openjpa.kernel.DelegatingStoreManager.initialize(DelegatingStoreManager.java:111)
at
org.apache.openjpa.kernel.ROPStoreManager.initialize(ROPStoreManager.java:57)
at org.apache.openjpa.kernel.BrokerImpl.initialize(BrokerImpl.java:894)
at org.apache.openjpa.kernel.BrokerImpl.find(BrokerImpl.java:852)
... 78 more

I'm running a query thathas a particular field added to the fetch group for
the query, something like this:

((QueryImpl) query).getFetchPlan().addField(A.class, "funkyField");

There error doesn't happen with all data sets and it doesn't happen with
other fields on the same query/class. Anyone have any clue? What is the
significance of the "5"? Can I use that as the index of field names into a
class or something?

Is there a better way to prefetch fields? (In this case a collection). Is
there a JPA (i.e. non-OpenJPA-specific) way to prefetch collections? Perhaps
I just have a major misunderstanding of fetching again.

Thanks.

--
Daryl Stultz
_____________________________________
6 Degrees Software and Consulting, Inc.
http://www.6degrees.com
mailto:[hidden email]

[Daryl Stultz]

On Wed, Nov 4, 2009 at 4:07 PM, Daryl Stultz <[hidden email]> wrote:

> Hello,
>
> I'm wondering if anyone has seen something like this:
>
> <openjpa-1.2.1-r752877:753278 nonfatal general error>
> org.apache.openjpa.persistence.PersistenceException: 5
>

I've run into this on 2 occasions with different fields in one entity. I can
get around it in both cases by changing a fetch from EAGER to LAZY. So I
have the entity A which has a collection of B which is set to LAZY. B has a
field C that is set to EAGER. I retrieve like this:

query = em.createQuery("select o from A as o");
((QueryImpl) query).getFetchPlan().addField(A.class, "bCol"); // bCol is
collection of B

The field B.c is defined like so

@ManyToOne(fetch = FetchType.EAGER, optional = false)
@JoinColumn(name = "cid", nullable = false)
@ForeignKey
private C c;

This has also happened on another field of B that is nullable/optional.

Right now the only way for me to feel the system is secure is to change
everything to LAZY and put fetchPlan addFields all over. Please tell me I
don't have to do that!

Thanks.

--
Daryl Stultz
_____________________________________
6 Degrees Software and Consulting, Inc.
http://www.6degrees.com
mailto:[hidden email]

[Michael Dick]

Hi Daryl,

A JOIN FETCH will eagerly load the collection for you, but should return one
reference to the entity for each element in the collection. I'm working on
fixing this under OPENJPA-894 (proceeding slowly though).

Adding a distinct clause will filter out these duplicates (but there are
issues there as well : OPENJPA-1365).

When it's all working you could use a query like this :
Query q = em.createQuery("SELECT DISTINCT o from A as o JOIN FETCH o.bCol");

If your goal is to selectively eagerly load a collection I believe this is
the vendor agnostic syntax you want..

Hope this helps,

-mike




On Wed, Nov 4, 2009 at 7:48 PM, Daryl Stultz <[hidden email]> wrote:

> On Wed, Nov 4, 2009 at 4:07 PM, Daryl Stultz <[hidden email]> wrote:
>
> > Hello,
> >
> > I'm wondering if anyone has seen something like this:
> >
> > <openjpa-1.2.1-r752877:753278 nonfatal general error>
> > org.apache.openjpa.persistence.PersistenceException: 5
> >
>
> I've run into this on 2 occasions with different fields in one entity. I
> can
> get around it in both cases by changing a fetch from EAGER to LAZY. So I
> have the entity A which has a collection of B which is set to LAZY. B has a
> field C that is set to EAGER. I retrieve like this:
>
> query = em.createQuery("select o from A as o");
> ((QueryImpl) query).getFetchPlan().addField(A.class, "bCol"); // bCol is
> collection of B
>
> The field B.c is defined like so
>
> @ManyToOne(fetch = FetchType.EAGER, optional = false)
> @JoinColumn(name = "cid", nullable = false)
> @ForeignKey
> private C c;
>
> This has also happened on another field of B that is nullable/optional.
>
> Right now the only way for me to feel the system is secure is to change
> everything to LAZY and put fetchPlan addFields all over. Please tell me I
> don't have to do that!
>
> Thanks.
>
> --
> Daryl Stultz
> _____________________________________
> 6 Degrees Software and Consulting, Inc.
> http://www.6degrees.com
> mailto:[hidden email]
>

[Daryl Stultz]

On Thu, Nov 5, 2009 at 11:10 AM, Michael Dick <[hidden email]>wrote:

> A JOIN FETCH will eagerly load the collection for you, but should return
> one reference to the entity for each element in the collection. I'm
> working on fixing this under OPENJPA-894 (proceeding slowly though).


Yes, I discovered this bug a month or so ago which led me to change my
approach for using fetch joins to modifying the fetch plan on the query. Now
I've run into this other problem.


> When it's all working you could use a query like this :
> Query q = em.createQuery("SELECT DISTINCT o from A as o JOIN FETCH
> o.bCol");
>
>
So this doesn't work now? (Or is it simply that DISTINCT is not required on
first pass due to the bug?)
Should LEFT JOIN FETCH work the same way?

I've experienced 894 directly but I don't fully understand 1365. If I have
OpenJPA 1.2.1, what are my options?

Thanks.

--
Daryl Stultz
_____________________________________
6 Degrees Software and Consulting, Inc.
http://www.6degrees.com
mailto:[hidden email]

[Michael Dick]

On Thu, Nov 5, 2009 at 10:26 AM, Daryl Stultz <[hidden email]> wrote:

> On Thu, Nov 5, 2009 at 11:10 AM, Michael Dick <[hidden email]
> >wrote:
>
> > A JOIN FETCH will eagerly load the collection for you, but should return
> > one reference to the entity for each element in the collection. I'm
> > working on fixing this under OPENJPA-894 (proceeding slowly though).
>
>
> Yes, I discovered this bug a month or so ago which led me to change my
> approach for using fetch joins to modifying the fetch plan on the query.
> Now
> I've run into this other problem.
>

I was afraid of that. I haven't kept up with the mailing lists as well as
I'd like to. In any event you are hitting a bug in FetchPlans.. Just thought
I'd throw out the vendor agnostic solution in case you hadn't seen it.

Can you give the 1.2.x patch attached to OPENJPA-894 a try? There are some
limitations with the approach I took (ie if Large Result Sets are used), but
it'd be nice to know whether it works for you.

>
>
> > When it's all working you could use a query like this :
> > Query q = em.createQuery("SELECT DISTINCT o from A as o JOIN FETCH
> > o.bCol");
> >
> >
> So this doesn't work now? (Or is it simply that DISTINCT is not required on
> first pass due to the bug?)
>

The latter. DISTINCT is not required due to the bug.


> Should LEFT JOIN FETCH work the same way?
>

LEFT JOINS should behave the same way.. The only difference is that it will
return entities from the LEFT side which do not 'match' any entities on the
RIGHT side..

ie
    SELECT m from Manager m LEFT JOIN FETCH m.employees
includes Managers with no employee.

    SELECT m from Manager m JOIN FETCH m.employees
only returns Managers which have a an employee in the employees collection..



> I've experienced 894 directly but I don't fully understand 1365. If I have
> OpenJPA 1.2.1, what are my options?
>
>
OPENJPA-1365 is a change in the way we handle distinct results. Previously
we just added in the DISTINCT keyword to the SQL statement and relied on
that to return the correct results. This doesn't work if DISTINCT is applied
to multiple tables.

The best example is buried in a longish email I wrote to the dev mailing
list :
http://n2.nabble.com/How-should-we-handle-the-JPQL-DISTINCT-keyword-td3908400.html#a3908400

In OpenJPA 1.2.1 if you want the duplicate references in your result list
then you're mostly out of luck. One ugly way to get them is to run the query
twice (the second time we operate in memory and return the correct results).
If you don't want duplicate references then I _think_ it works out of the
box (until you run the query in memory . . .).

-mike



> Thanks.
>
> --
> Daryl Stultz
> _____________________________________
> 6 Degrees Software and Consulting, Inc.
> http://www.6degrees.com
> mailto:[hidden email]
>

[Daryl Stultz]

On Thu, Nov 5, 2009 at 12:52 PM, Michael Dick <[hidden email]>wrote:

> Can you give the 1.2.x patch attached to OPENJPA-894 a try? There are some
> limitations with the approach I took (ie if Large Result Sets are used),
> but
> it'd be nice to know whether it works for you.
>

I'm pretty strapped for time, but I'll try to squeeze it in.

LEFT JOINS should behave the same way..


I was originally under the impression that JOIN FETCH was somehow different
from a JOIN from an SQL/DB perspective, perhaps because of the bug. If the
duplicates were returned, exactly how is JOIN different than JOIN FETCH? The
latter returns the parent duplicated for each child along with each child
AND populates each parent with the set of children? What is the difference
between

select o from A as o
join o.bCol as bCol

and

select o from A as o
join fetch o.bCol

assuming duplicate bug is fixed?



> In OpenJPA 1.2.1 if you want the duplicate references in your result list
> then you're mostly out of luck.


Fortunately I don't. But I don't want them to show up later because I forgot
the "DISTINCT" keyword...


> One ugly way to get them is to run the query
> twice (the second time we operate in memory and return the correct
> results).
> If you don't want duplicate references then I _think_ it works out of the
> box (until you run the query in memory . . .).
>

Meaning without the DISTINCT keyword? I should not get the duplicates now or
in the future (after bugs fixed) if I include DISTINCT, right (assuming
simple "select distinct o from A as o left join fetch o.bCol")?

--
Daryl Stultz
_____________________________________
6 Degrees Software and Consulting, Inc.
http://www.6degrees.com
mailto:[hidden email]

[Michael Dick]

On Thu, Nov 5, 2009 at 12:17 PM, Daryl Stultz <[hidden email]> wrote:

> On Thu, Nov 5, 2009 at 12:52 PM, Michael Dick <[hidden email]
> >wrote:
>
> > Can you give the 1.2.x patch attached to OPENJPA-894 a try? There are
> some
> > limitations with the approach I took (ie if Large Result Sets are used),
> > but
> > it'd be nice to know whether it works for you.
> >
>
> I'm pretty strapped for time, but I'll try to squeeze it in.
>
> LEFT JOINS should behave the same way..
>
>
> I was originally under the impression that JOIN FETCH was somehow different
> from a JOIN from an SQL/DB perspective, perhaps because of the bug. If the
> duplicates were returned, exactly how is JOIN different than JOIN FETCH?
> The
> latter returns the parent duplicated for each child along with each child
> AND populates each parent with the set of children? What is the difference
> between
>
> select o from A as o
> join o.bCol as bCol
>
> and
>
> select o from A as o
> join fetch o.bCol
>
> assuming duplicate bug is fixed?
>

I think I misunderstood your point (I thought the FETCH was implied).

A LEFT JOIN FETCH and a JOIN FETCH differ in that the LEFT JOIN FETCH is
more inclusive (ie you don't have to have a relation to the right side).

The same applies to a LEFT JOIN and a JOIN (no fetch).

Adding in the fetch clause:

A JOIN FETCH differs from a JOIN in two ways :
  * You will get one reference to the entity on the left side of the JOIN
for every related entity on the right side of the JOIN.
  * The relationship on the right side of the JOIN will be eagerly loaded.

LEFT JOIN FETCH and LEFT JOIN follow the same rules (but again the LEFT part
makes the results more inclusive).

Sorry for the confusion, I thought you were asking about the difference
between a JOIN FETCH and a LEFT JOIN FETCH. Not between a JOIN and a JOIN
FETCH.


> > In OpenJPA 1.2.1 if you want the duplicate references in your result list
> > then you're mostly out of luck.
>
>
> Fortunately I don't. But I don't want them to show up later because I
> forgot
> the "DISTINCT" keyword...
>

> > One ugly way to get them is to run the query
> > twice (the second time we operate in memory and return the correct
> > results).
> > If you don't want duplicate references then I _think_ it works out of the
> > box (until you run the query in memory . . .).
> >
>
> Meaning without the DISTINCT keyword? I should not get the duplicates now
> or
> in the future (after bugs fixed) if I include DISTINCT, right (assuming
> simple "select distinct o from A as o left join fetch o.bCol")?
>
>

Correct. Distinct should prevent duplicates now and in the future.


>  --
> Daryl Stultz
> _____________________________________
> 6 Degrees Software and Consulting, Inc.
> http://www.6degrees.com
> mailto:[hidden email]
>

[Daryl Stultz]

On Thu, Nov 5, 2009 at 1:48 PM, Michael Dick <[hidden email]>wrote:

> On Thu, Nov 5, 2009 at 12:17 PM, Daryl Stultz <[hidden email]> wrote:
>
>
> I think I misunderstood your point (I thought the FETCH was implied).
>

I think I asked a new question, not necessarily based on your previous
input, so I don't think you misunderstood anything, I just changed the line
of questioning on you. :-)

>
> Adding in the fetch clause:
>
> A JOIN FETCH differs from a JOIN in two ways :
>  * You will get one reference to the entity on the left side of the JOIN
> for every related entity on the right side of the JOIN.
>

This I misunderstood long ago before our discussion - I didn't misunderstand
what you were saying about it. I don't think. Anyway.


>  * The relationship on the right side of the JOIN will be eagerly loaded.
>

This is what I'm having trouble with. If a JOIN would return the children,
isn't it implied that they are eagerly loaded? What exactly does eagerly
loaded mean in this case? Picture this:

select a, bCol.b from A as a
join a.bCol as bCol

compared to this:

select a from A as a
join fetch a.bCol

They both return all the child B entities (same number of "rows") but the
second puts the B's into the appropriate a.bCol collection. In both cases,
B's are being loaded. Is it the bCol collection you are referring to as
being eagerly loaded? Sorry if I'm putting too fine a point on this, just
trying to cement it all in my brain.

--
Daryl Stultz
_____________________________________
6 Degrees Software and Consulting, Inc.
http://www.6degrees.com
mailto:[hidden email]

[Michael Dick]

On Thu, Nov 5, 2009 at 1:27 PM, Daryl Stultz <[hidden email]> wrote:

> On Thu, Nov 5, 2009 at 1:48 PM, Michael Dick <[hidden email]
> >wrote:
>
> > On Thu, Nov 5, 2009 at 12:17 PM, Daryl Stultz <[hidden email]>
> wrote:
> >
> >
> > I think I misunderstood your point (I thought the FETCH was implied).
> >
>
> I think I asked a new question, not necessarily based on your previous
> input, so I don't think you misunderstood anything, I just changed the line
> of questioning on you. :-)
>
> >
> > Adding in the fetch clause:
> >
> > A JOIN FETCH differs from a JOIN in two ways :
> >  * You will get one reference to the entity on the left side of the JOIN
> > for every related entity on the right side of the JOIN.
> >
>
> This I misunderstood long ago before our discussion - I didn't
> misunderstand
> what you were saying about it. I don't think. Anyway.
>
>
> >  * The relationship on the right side of the JOIN will be eagerly loaded.
> >
>
> This is what I'm having trouble with. If a JOIN would return the children,
> isn't it implied that they are eagerly loaded? What exactly does eagerly
> loaded mean in this case? Picture this:
>
> select a, bCol.b from A as a
> join a.bCol as bCol
>
> compared to this:
>
> select a from A as a
> join fetch a.bCol
>
> They both return all the child B entities (same number of "rows") but the
> second puts the B's into the appropriate a.bCol collection. In both cases,
> B's are being loaded. Is it the bCol collection you are referring to as
> being eagerly loaded? Sorry if I'm putting too fine a point on this, just
> trying to cement it all in my brain.
>
>

In the join fetch case the eager loading is a side effect of the query, the
entities on the right hand side are not part of the query results.

Another difference is that with a join fetch you can't set an identification
variable for the entities on the right hand side.
For example :
"select a from A join fetch a.bCol b" would throw an exception.

Having multiple references to the same entity from the left side of the
query is a side effect of this loading as far as I can tell.

I can't claim to be an expert here, I'm learning as I go. A lot of how we
handle joins seems a bit odd to me. Sorry if this isn't terribly
enlightening. The best reference I can give you is section 4.4.5 of the JPA
1.0 specification. Mike Keith's Pro EJB3 book is also a good place to look.

-mike

--
> Daryl Stultz
> _____________________________________
> 6 Degrees Software and Consulting, Inc.
> http://www.6degrees.com
> mailto:[hidden email]
>