Fetch Group questions

[Michael Vorburger-4]

Hello,
 
May I ask some questions about Fetch Groups with OpenJPA (using v1.1.0):
 
1. Is the "default" fetch group ALWAYS active? I was trying to load e.g.
only one field from an Entity in a query, but noticed that even if I do
fp.resetFetchGroups(); and fp.clearFields(); and then a
fp.addField(MyEntity.class, "name"); STILL always includes all columns
which I assume are in the Default Fetch Group (all Basic fields, and all
direct relations). This took me a moment to figure out, because as I
read the Doc in chapter 7, I was assuming that I can entirely control
this through the API.. is this intended? To really control exactly what
column is in a SELECT fine granuarly under some circumstances, is it
fair to say (and may be add to the doc?) that you have to annotate EVERY
field as @Basic(fetch=FetchType.LAZY) or @ManyToOne(cascade =
CascadeType.PERSIST, fetch=FetchType.LAZY) ? If under other
circumstances one then wants the standard fetching behaviour, do you
need to create your own myDefaultFetchGroup which explicitly includes
all Basic fields + and all direct relations? Seems a bit cumbersome, am
I getting something wrong?
 
2.
http://openjpa.apache.org/docs/latest/manual/manual.html#ref_guide_fetch
_impl says that "Even when a direct relation is not eagerly fetched,
OpenJPA selects the foreign key columns and caches the values. This way
when you do traverse the relation, OpenJPA can often find the related
object in its cache, or at least avoid joins when loading the related
object from the database." - that's very kind, but may be we have an
Entity with lot's of foreign key columns and I KNOW I only want a list
showing a few attributes, and I KNOW that the direct relations won't be
traversed (say because the Entites are detached, may be). Is there any
way to configure OpenJPA to NOT eagerly fetch direct relations, unless
they are explicitly part of the Fetch Group? Actually what I'd want is
to be able to by default not include foreign key columns, but have a way
to explicitly include them - which is not the same as having the field
as part of the FetchGroup (that will lead to a JOIN, I may just want to
control whether or not to selects the foreign key column).
 
3. When I do FetchPlan.addField() for a @ManyToMany or a
@PersistentCollection(elementEmbedded = true) field, there is no eager
fetching (like when the same field has FetchType.EAGER)... Should this
work? Can somebody point to any examples or test cases for this? Do we
need to fiddle with the EagerFetchMode join/parallel??
 
4. Caching: How do FetchGroups and L2 (EMF not EM) Caching relate to
each other in OpenJPA? If you do a query with an active FetchGroup that
excludes some fields, with active Data Cache, and later another query on
the same Entity with another FetchGroup (some columns may be used in
both, others not), is the cache a) never used (in other words, query
caching uses the Fetch Group as cache key), b) only the "missing"
columns are fetched and "merged" ?
 
Thank you very much,
Michael

_____________________________
Michael Vorburger, Odyssey Financial Technologies


 

____________________________________________________________

• This email and any files transmitted with it are CONFIDENTIAL and intended
  solely for the use of the individual or entity to which they are addressed.
• Any unauthorized copying, disclosure, or distribution of the material within
  this email is strictly forbidden.
• Any views or opinions presented within this e-mail are solely those of the
  author and do not necessarily represent those of Odyssey Financial
Technologies SA unless otherwise specifically stated.
• An electronic message is not binding on its sender. Any message referring to
  a binding engagement must be confirmed in writing and duly signed.
• If you have received this email in error, please notify the sender immediately
  and delete the original.

[Andy Schlaikjer]

Hi there,

Michael Vorburger wrote:

> 1. Is the "default" fetch group ALWAYS active? I was trying to load e.g.
> only one field from an Entity in a query, but noticed that even if I do
> fp.resetFetchGroups(); and fp.clearFields(); and then a
> fp.addField(MyEntity.class, "name"); STILL always includes all columns
> which I assume are in the Default Fetch Group (all Basic fields, and all
> direct relations). This took me a moment to figure out, because as I
> read the Doc in chapter 7, I was assuming that I can entirely control
> this through the API.. is this intended? To really control exactly what
> column is in a SELECT fine granuarly under some circumstances, is it
> fair to say (and may be add to the doc?) that you have to annotate EVERY
> field as @Basic(fetch=FetchType.LAZY) or @ManyToOne(cascade =
> CascadeType.PERSIST, fetch=FetchType.LAZY) ? If under other
> circumstances one then wants the standard fetching behaviour, do you
> need to create your own myDefaultFetchGroup which explicitly includes
> all Basic fields + and all direct relations? Seems a bit cumbersome, am
> I getting something wrong?

To my knowledge, if you haven't activated a specific fetch group, then
the default always applies. The JPA spec itself, not OpenJPA, dictates
default fetch behavior for different relationships. For instance, all
@*ToOne relationships are by default EAGER and all @*ToMany
relationships are LAZY. Someone please correct me if I'm wrong.

> 2.
> http://openjpa.apache.org/docs/latest/manual/manual.html#ref_guide_fetch
> _impl says that "Even when a direct relation is not eagerly fetched,
> OpenJPA selects the foreign key columns and caches the values. This way
> when you do traverse the relation, OpenJPA can often find the related
> object in its cache, or at least avoid joins when loading the related
> object from the database." - that's very kind, but may be we have an
> Entity with lot's of foreign key columns and I KNOW I only want a list
> showing a few attributes, and I KNOW that the direct relations won't be
> traversed (say because the Entites are detached, may be). Is there any
> way to configure OpenJPA to NOT eagerly fetch direct relations, unless
> they are explicitly part of the Fetch Group? Actually what I'd want is
> to be able to by default not include foreign key columns, but have a way
> to explicitly include them - which is not the same as having the field
> as part of the FetchGroup (that will lead to a JOIN, I may just want to
> control whether or not to selects the foreign key column).

I believe there is a way to configure this default behavior in OpenJPA,
but not sure how.

> 3. When I do FetchPlan.addField() for a @ManyToMany or a
> @PersistentCollection(elementEmbedded = true) field, there is no eager
> fetching (like when the same field has FetchType.EAGER)... Should this
> work? Can somebody point to any examples or test cases for this? Do we
> need to fiddle with the EagerFetchMode join/parallel??

I assume that even with OpenJPA's elementEmbedded extension, the default
fetch behavior for @*ToMany is lazy.

> 4. Caching: How do FetchGroups and L2 (EMF not EM) Caching relate to
> each other in OpenJPA? If you do a query with an active FetchGroup that
> excludes some fields, with active Data Cache, and later another query on
> the same Entity with another FetchGroup (some columns may be used in
> both, others not), is the cache a) never used (in other words, query
> caching uses the Fetch Group as cache key), b) only the "missing"
> columns are fetched and "merged" ?

Great question.. I'd love to hear the answer :)

Andy

[Pinaki Poddar]

Hi,
> 1. Is the "default" fetch group ALWAYS active?
Almost ALWAYS. Unless you really, seriously do not want it, force removal by
   FetchPlan.removeFetchGroup(FetchGroup.NAME_DEFAULT);

after clearFetchGroup(), it will still load the fields of default fetch group
after resetFetchGroup(), it will load the default + any other configured fetch groups.

> 2. Is there any way to configure OpenJPA to NOT eagerly fetch direct relations

OpenJPA is not really 'fetching' the direct relation unless it is included in the fetch group. If a row for A is read from database and row(A) contains foreign keys to row(B) and row(C) (as the case may be for direct to-one mappings), then those FK values are read and stored in a way that is invisible to user application. The reference A.b or A.c is not populated. The intermediate storage of foreign keys optimizes any subsequent traversal of A.b and A.c without a join.

Once A has lots of to-one relations, it is a valid concern that A will store a lot of FK values in intermediate storage (perhaps without ever using them), but just wanted to underline that there is no significant performance penalty in such cases as the values are read from an existing row.


> 3. When I do FetchPlan.addField() for a @ManyToMany or a @PersistentCollection(elementEmbedded = true) field, there is no eager fetching (like when the same field has FetchType.EAGER)... Should this
work? Can somebody point to any examples or test cases for this? Do we need to fiddle with the EagerFetchMode join/parallel??

Not sure I understand the question -- can you please elaborate?
 
> a) in other words, query caching uses the Fetch Group as cache key)
No.

> b) only the "missing" columns are fetched and "merged" ?
No.
If fetch plan is the only thing that is different between query Q1 and Q2 and QueryCache is enabled,  executing Q1 followed by Q2 will produce an incorrect behavior. OpenJPA will consider Q2 as the same as cached version Q1 and will never apply fetch plan of Q2.

[Michael Vorburger-4]

1. Thanks, that works great.

2. OK; understand perfectly what your saying; thanks for detailing. (Now
our data architect WILL probably cry that "there are unncessary columns
fetched!".  In the "upper layer" that is creating the FetchPlan,
dynamically, we would actually have the information to know that we will
NOT access A.c or A.c, so if a future OpenJPA version would offer a way
to override this,per attribute not globally, we could leverage that; but
it's certainly not critical.  Do you want a JIRA for this?)

3. Ignore; I think correctly configured Eager Fetching is what I'm
looking for; need to play more with this. I'll ask more specific
questions if needed.

4. Hm... I see.  So to be safe, until this "bug" (?) gets addressed (can
it? will it? Would a JIRA help?), or actually rather for the combination
of Cache/FetchGroup to work at all, would it be a good idea to may be
split all Entities into two types: A. fully cached ones (all instances
with "ALL" FetchGroup pinned in memory) - that should work well for
"lookup table" kind of Entites. No FetchGroups used.  B. No caching of
all other Entites (most Business Entites, all with lot's of columns),
and use of FetchGroups - good idea?  Q: Is there a runtime API to the
@DataCache(enabled=false), some configuration API to set which classes
(globally, given a Class, not an instance) should be cached and which
not?

4 1/2 [NEW] Q: On a Query where an association to a "fully cached"
Entity IS in the FetchGroup, can OpenJPA know this and NOT do the JOIN?
(Otherwise I'll think about building this into a layer on top, although
it would be nicer if OpenJPA itself could handle this...)

5. [NEW] Q: Is there any significant runtime performance difference
between putting the @FetchGroup/s annotation on an @Entity and merely
"activating" it through FetchPlan.addFetchGroup(String groupName), as
opposed to "building" it and using lot's of invididual
FetchPlan.addField() ?  Probably not, but I thought I'd ask.

Thanks!



-----Original Message-----
From: Pinaki Poddar [mailto:ppoddar@...]
Sent: jeudi, 31. juillet 2008 02:13
To: users@...
Subject: Re: Fetch Group questions


Hi,
> 1. Is the "default" fetch group ALWAYS active?
Almost ALWAYS. Unless you really, seriously do not want it, force
removal by
   FetchPlan.removeFetchGroup(FetchGroup.NAME_DEFAULT);

after clearFetchGroup(), it will still load the fields of default fetch
group after resetFetchGroup(), it will load the default + any other
configured fetch groups.

> 2. Is there any way to configure OpenJPA to NOT eagerly fetch direct
> relations

OpenJPA is not really 'fetching' the direct relation unless it is
included in the fetch group. If a row for A is read from database and
row(A) contains foreign keys to row(B) and row(C) (as the case may be
for direct to-one mappings), then those FK values are read and stored in
a way that is invisible to user application. The reference A.b or A.c is
not populated.
The intermediate storage of foreign keys optimizes any subsequent
traversal of A.b and A.c without a join.

Once A has lots of to-one relations, it is a valid concern that A will
store a lot of FK values in intermediate storage (perhaps without ever
using them), but just wanted to underline that there is no significant
performance penalty in such cases as the values are read from an
existing row.


> 3. When I do FetchPlan.addField() for a @ManyToMany or a
> @PersistentCollection(elementEmbedded = true) field, there is no eager

> fetching (like when the same field has FetchType.EAGER)... Should this
work? Can somebody point to any examples or test cases for this? Do we
need to fiddle with the EagerFetchMode join/parallel??

Not sure I understand the question -- can you please elaborate?
 
> a) in other words, query caching uses the Fetch Group as cache key)
No.

> b) only the "missing" columns are fetched and "merged" ?
No.
If fetch plan is the only thing that is different between query Q1 and
Q2 and QueryCache is enabled,  executing Q1 followed by Q2 will produce
an incorrect behavior. OpenJPA will consider Q2 as the same as cached
version
Q1 and will never apply fetch plan of Q2.


--
View this message in context:
http://n2.nabble.com/Fetch-Group-questions-tp534861p661340.html
Sent from the OpenJPA Users mailing list archive at Nabble.com.


____________________________________________________________

• This email and any files transmitted with it are CONFIDENTIAL and intended
  solely for the use of the individual or entity to which they are addressed.
• Any unauthorized copying, disclosure, or distribution of the material within
  this email is strictly forbidden.
• Any views or opinions presented within this e-mail are solely those of the
  author and do not necessarily represent those of Odyssey Financial
Technologies SA unless otherwise specifically stated.
• An electronic message is not binding on its sender. Any message referring to
  a binding engagement must be confirmed in writing and duly signed.
• If you have received this email in error, please notify the sender immediately
  and delete the original.

[Pinaki Poddar]

hi,
  > 2. so if a future OpenJPA version would offer a way to override this,per attribute not globally, we could leverage that; but it's certainly not critical.  Do you want a JIRA for this?
 
   Yes. Why not?

  > 4. Would a JIRA help?
     A JIRA has been created
          https://issues.apache.org/jira/browse/OPENJPA-671

   > Q: Is there a runtime API to the @DataCache(enabled=false), some configuration API to set which classes (globally, given a Class, not an instance) should be cached and which not?

   Data partitioning strategy for data cache exists at class level. There is no obvious API that I am aware of to dynamically modify whether a class X will be L2 cached or not. One can attempt to tinker via
                   ClassMetaData.setDataCacheName()
   but side effects of doing so remains unexplored.



> 4 1/2 [NEW] Q: On a Query where an association to a "fully cached"
Entity IS in the FetchGroup, can OpenJPA know this and NOT do the JOIN?
   
   That is the purpose of 'intermediate' storage of Foreign Key fields being discussed in this thread. To avoid a join and fetch the related entity if it is already in cache. Or am i missing something?

> 5. [NEW] Q: Is there any significant runtime performance difference
between putting the @FetchGroup/s annotation on an @Entity and merely
"activating" it through FetchPlan.addFetchGroup(String groupName), as
opposed to "building" it and using lot's of invididual
FetchPlan.addField() ?  Probably not, but I thought I'd ask.

I will hazard a guess that direct addField() may be wee bit faster than searching for a field in registered fetch groups. But the performance differential is unlikely to be significant.

[Michael Vorburger-4]

Hello again Pinaki et al,

We've run into a problem with FetchGroups which to me appears to be due
to an API Design limitation.  Example, simplified Entity pseudo code:

abstract class Base { String code; }

@Inheritance(strategy = InheritanceType.TABLE_PER_CLASS)
class A extends Base { B b; }

@Inheritance(strategy = InheritanceType.TABLE_PER_CLASS)
class B extends Base { }

We would like to Fetch A's with their asssociate B's and the code in the
B, but not in the A... see where I'm going? ;-) -- So addField(A.class,
"b"); and addField(Base.class, "code");  (NOT addField(B.class,
"code")... see doc).  The resulting SQL will JOIN A and B, but include
the 'code' from both A and B!

Is there any other (internal?) API to configure the FetchPlan more
precisely?  Do you agree this is a limitation in the current API, and
would you be open to address this in a future version (createa a JIRA?)
?

Thanks a lot,
Michael

--
View this message in context:
http://n2.nabble.com/Fetch-Group-questions-tp534861p681237.html


____________________________________________________________

• This email and any files transmitted with it are CONFIDENTIAL and intended
  solely for the use of the individual or entity to which they are addressed.
• Any unauthorized copying, disclosure, or distribution of the material within
  this email is strictly forbidden.
• Any views or opinions presented within this e-mail are solely those of the
  author and do not necessarily represent those of Odyssey Financial
Technologies SA unless otherwise specifically stated.
• An electronic message is not binding on its sender. Any message referring to
  a binding engagement must be confirmed in writing and duly signed.
• If you have received this email in error, please notify the sender immediately
  and delete the original.

[Michael Vorburger-4]

LOW: fyi, JIRA about earlier discussion point 2 created: https://issues.apache.org/jira/browse/OPENJPA-701


-----Original Message-----
From: Pinaki Poddar [mailto:ppoddar@...]
Sent: vendredi, 8. août 2008 19:41
To: users@...
Subject: RE: Fetch Group questions


hi,
  > 2. so if a future OpenJPA version would offer a way to override this,per attribute not globally, we could leverage that; but it's certainly not critical.  Do you want a JIRA for this?
 
   Yes. Why not?

  > 4. Would a JIRA help?
     A JIRA has been created
          https://issues.apache.org/jira/browse/OPENJPA-671

--
View this message in context: http://n2.nabble.com/Fetch-Group-questions-tp534861p681237.html


____________________________________________________________

• This email and any files transmitted with it are CONFIDENTIAL and intended
  solely for the use of the individual or entity to which they are addressed.
• Any unauthorized copying, disclosure, or distribution of the material within
  this email is strictly forbidden.
• Any views or opinions presented within this e-mail are solely those of the
  author and do not necessarily represent those of Odyssey Financial
Technologies SA unless otherwise specifically stated.
• An electronic message is not binding on its sender. Any message referring to
  a binding engagement must be confirmed in writing and duly signed.
• If you have received this email in error, please notify the sender immediately
  and delete the original.

[Pinaki Poddar]

Hi Michael,
  Yes, the field inclusion in fetch group is determined by the field's name with the declaring class and not by the defining class.

 The recent commit 688777, adds the behavior you want. So
   FetchPlan.addField(A.class, "b");
   FetchPlan.addField(B.class, "code");
 should fetch {A.b, A.b.code} but not {A.code}.

 I have added a JIRA issue to track [1].

 Let me know if you can try the recent changes and whether it addresses your use case.

 Regards --

[1] https://issues.apache.org/jira/browse/OPENJPA-704
[2] http://n2.nabble.com/svn-commit%3A-r688777---in--openjpa-trunk-openjpa-kernel-src-main-java-org-apache-openjpa%3A-kernel-FetchConfigurationImpl.java-meta-FieldMetaData.java-tc781976.html

[Michael Vorburger-4]

Pinaki,

Finally extending slightly beyond my previous "Wow" only response (thanks, amazing turn around!) :

>  Let me know if you can try the recent changes and whether it addresses your use case.

I built a 1.3.0-SNAPSHOT from SVN and hereby confirm it addresses the use case initially posted; and https://issues.apache.org/jira/browse/OPENJPA-704.  Once again, thanks a lot for your follow-up and this improvement.

Working further on our stuff using Fetch Groups (basically a 'model-driven' layer on top - one day I should make some slides about this...), we have come up with another fun use case though, a similar scenario as before, something where the current OpenJPA FetchGroup API appears too limited / not fine-grained enough.  Here is an example:

class B { String name; String address; }
class A { B mother; B child; }

Now suppose one would like to build a fetch group with only "child"'s name and "mother"'s address. If you use a fetch plan with the following fields:

addField(A.class, child)
addField(B.class, name)
addField(A.class, mother)
addField(B.class, address)

the result would be {A.child, A.child.name, A.child.address, B.mother, B.mother.name, B.mother.address}... unless I'm missing something, it doesn't currently seem possible to avoid to fetch A.child.address and B.mother.name, agreed?  

I guess from an API point of view, you would need to be able to specify addField(A.class, child.name) and addField(A.class, mother.address)?  There are probably other ways one could build an API allowing configuring this, but allowing "relationship traversal" in FetchGroups would seem most natural, to me.  However I'm not sure what this would imply OpenJPA internally for you? Is a solution feasible?

BTW: Does anybody know if the JPA 2.0 JSR includes an OpenJPA FetchGroup-like API into the standard JPA API?  If yes, hopefully it will cover the kind of stuff we're discussing here!

Regards,
Michael


-----Original Message-----
From: Pinaki Poddar [mailto:ppoddar@...]
Sent: lundi, 25. août 2008 19:21
To: users@...
Subject: RE: Fetch Group questions


Hi Michael,
  Yes, the field inclusion in fetch group is determined by the field's name with the declaring class and not by the defining class.

 The recent commit 688777, adds the behavior you want. So
   FetchPlan.addField(A.class, "b");
   FetchPlan.addField(B.class, "code");
 should fetch {A.b, A.b.code} but not {A.code}.

 I have added a JIRA issue to track [1].

 Let me know if you can try the recent changes and whether it addresses your use case.

 Regards --

[1] https://issues.apache.org/jira/browse/OPENJPA-704
[2]
http://n2.nabble.com/svn-commit%3A-r688777---in--openjpa-trunk-openjpa-kernel-src-main-java-org-apache-openjpa%3A-kernel-FetchConfigurationImpl.java-meta-FieldMetaData.java-tc781976.html



--
View this message in context: http://n2.nabble.com/Fetch-Group-questions-tp534861p782029.html
Sent from the OpenJPA Users mailing list archive at Nabble.com.

____________________________________________________________

• This email and any files transmitted with it are CONFIDENTIAL and intended
  solely for the use of the individual or entity to which they are addressed.
• Any unauthorized copying, disclosure, or distribution of the material within
  this email is strictly forbidden.
• Any views or opinions presented within this e-mail are solely those of the
  author and do not necessarily represent those of Odyssey Financial
Technologies SA unless otherwise specifically stated.
• An electronic message is not binding on its sender. Any message referring to
  a binding engagement must be confirmed in writing and duly signed.
• If you have received this email in error, please notify the sender immediately
  and delete the original.

[Pinaki Poddar]

Hi,
  > we have come up with another fun use case though, a similar scenario as before, something where  
  > the current OpenJPA FetchGroup API appears too limited / not fine-grained enough.  

  OpenJPA has certain capability to specify paths on a object graph, but it does not cover all cases. we are trying ;)

  With revision 696590, we tried to address the 'fun case' you mentioned with following caveats

package x.y.z;
class B { String name; String address; }
class A { B mother; B child; }

 FetchPlan fp = ...
 
 fp.addField("x.y.z.A.mother");
 fp.addField("x.y.z.A.mother.name");
 fp.addField("x.y.z.A.child");
 fp.addField("x.y.z.A.child.address");

will have the intended fetch. The caveats are
a) "x.y.z.A.mother" must be included to include "x.y.z.A.mother.name".
b) This new style is only tested with depth of 1. My hunch is it is non-trivial to make this work for multiple levels (the similar reasons that prevents path-like navigated field in FecthGroup annotation).

Pinaki,