Fetchgroups recursion problem

[calin014]

I have the following entities:

@Entity
public class Flow implements Serializable {
        @Id
        @GeneratedValue(strategy = GenerationType.IDENTITY)
        private Long id;

        @Column(length = 64)
        private String name;

        @OneToOne(cascade = { CascadeType.PERSIST, CascadeType.MERGE, CascadeType.REFRESH })
        @JoinColumn(name = "INITIAL_STATE")
        private State initialState;
}

@Entity
@FetchGroups( {
                @FetchGroup(name = "State_OutgoingTransitions", attributes = { @FetchAttribute(name = "outgoingTransitions") }),
                @FetchGroup(name = "State_IncomingTransitions", attributes = { @FetchAttribute(name = "incomingTransitions") }) })
public class State implements Serializable {
        @Id
        @GeneratedValue(strategy = GenerationType.IDENTITY)
        private Long id;

        @Column(length = 64)
        private String name;

        @OneToMany(mappedBy = "fromState", cascade = { CascadeType.PERSIST, CascadeType.MERGE, CascadeType.REFRESH })
        private List<Transition> outgoingTransitions;

        @OneToMany(mappedBy = "toState", cascade = { CascadeType.PERSIST, CascadeType.MERGE, CascadeType.REFRESH })
        private List<Transition> incomingTransitions;
}

@Entity
public class Transition implements Serializable {
        @Id
        @GeneratedValue(strategy = GenerationType.IDENTITY)
        private Long id;

        @Column(length = 64)
        private String name;

        @ManyToOne(cascade = { CascadeType.PERSIST, CascadeType.MERGE, CascadeType.REFRESH })
        @JoinColumn(name = "FROM_STATE_ID", referencedColumnName = "ID")
        private State fromState;

        @ManyToOne(cascade = { CascadeType.PERSIST, CascadeType.MERGE, CascadeType.REFRESH })
        @JoinColumn(name = "TO_STATE_ID", referencedColumnName = "ID")
        private State toState;
}

This is supposed to be the persistence model for a graph-like data structure.

I have an issue with fetch groups when using them to load the whole data structure from the database for cases involving recursion like the following:

This is what is persisted to the db:


This is what gets loaded into memory:

Transition tr24 and tr34 get loaded with a null reference on 'toState' field.

I can load the whole thing using lazy loading but i have to keep a user transaction the whole request(web app).Not to mention that i have to call some getters just to load the properties from db to use them on next requests.

I tried setting the recursion depth to a greater value although i read it is supposed to be infinite by default. Seems like a openjpa bug. Am i missing something?

[Pinaki Poddar]

Hi,
There are two attributes that control the closure of a graph as fetched by a FetchPlan/FetchConfiguration, namely
   Recursion Depth
   Max Fetch Depth

According to the cited use case, Max Fetch Depth is the relevant attribute that will control depth of traversal from a root entity (s1). By default, the max fetch depth is set to 1 and hence the immediate neighbors of s1 are fetched and not s4 or s5 which is at depth 2 from s1.

Recursion depth, on the other hand, controls the depth of traversal for recursive relation on types. If s1 had a recursive relation then recursion depth would have controlled traversal of that relation path.

calin014 wrote:

I have the following entities:

@Entity
public class Flow implements Serializable {
        @Id
        @GeneratedValue(strategy = GenerationType.IDENTITY)
        private Long id;

        @Column(length = 64)
        private String name;

        @OneToOne(cascade = { CascadeType.PERSIST, CascadeType.MERGE, CascadeType.REFRESH })
        @JoinColumn(name = "INITIAL_STATE")
        private State initialState;
}

@Entity
@FetchGroups( {
                @FetchGroup(name = "State_OutgoingTransitions", attributes = { @FetchAttribute(name = "outgoingTransitions") }),
                @FetchGroup(name = "State_IncomingTransitions", attributes = { @FetchAttribute(name = "incomingTransitions") }) })
public class State implements Serializable {
        @Id
        @GeneratedValue(strategy = GenerationType.IDENTITY)
        private Long id;

        @Column(length = 64)
        private String name;

        @OneToMany(mappedBy = "fromState", cascade = { CascadeType.PERSIST, CascadeType.MERGE, CascadeType.REFRESH })
        private List<Transition> outgoingTransitions;

        @OneToMany(mappedBy = "toState", cascade = { CascadeType.PERSIST, CascadeType.MERGE, CascadeType.REFRESH })
        private List<Transition> incomingTransitions;
}

@Entity
public class Transition implements Serializable {
        @Id
        @GeneratedValue(strategy = GenerationType.IDENTITY)
        private Long id;

        @Column(length = 64)
        private String name;

        @ManyToOne(cascade = { CascadeType.PERSIST, CascadeType.MERGE, CascadeType.REFRESH })
        @JoinColumn(name = "FROM_STATE_ID", referencedColumnName = "ID")
        private State fromState;

        @ManyToOne(cascade = { CascadeType.PERSIST, CascadeType.MERGE, CascadeType.REFRESH })
        @JoinColumn(name = "TO_STATE_ID", referencedColumnName = "ID")
        private State toState;
}

This is supposed to be the persistence model for a graph-like data structure.

I have an issue with fetch groups when using them to load the whole data structure from the database for cases involving recursion like the following:

This is what is persisted to the db:


This is what gets loaded into memory:

Transition tr24 and tr34 get loaded with a null reference on 'toState' field.

I can load the whole thing using lazy loading but i have to keep a user transaction the whole request(web app).Not to mention that i have to call some getters just to load the properties from db to use them on next requests.

I tried setting the recursion depth to a greater value although i read it is supposed to be infinite by default. Seems like a openjpa bug. Am i missing something?

[calin014]

Thanks for the quick reply.

I tried that but with no luck.
I did something like this before executing the query:

OpenJPAEntityManager ojem = OpenJPAPersistence.cast(entityManager);
System.out.println("getMaxFetchDepth() before: " + ojem.getFetchPlan().getMaxFetchDepth());
ojem.getFetchPlan().setMaxFetchDepth(15);
System.out.println("getMaxFetchDepth() after: " + ojem.getFetchPlan().getMaxFetchDepth());
ojem.getFetchPlan().addFetchGroups("State_OutgoingTransitions", "IncomingTransitions");

The output is:

getMaxFetchDepth() before: -1
getMaxFetchDepth() after: 15

The result is the same as described in the first post.

Am i doing it wrong?

Pinaki Poddar wrote:

Hi,
There are two attributes that control the closure of a graph as fetched by a FetchPlan/FetchConfiguration, namely
   Recursion Depth
   Max Fetch Depth

According to the cited use case, Max Fetch Depth is the relevant attribute that will control depth of traversal from a root entity (s1). By default, the max fetch depth is set to 1 and hence the immediate neighbors of s1 are fetched and not s4 or s5 which is at depth 2 from s1.

Recursion depth, on the other hand, controls the depth of traversal for recursive relation on types. If s1 had a recursive relation then recursion depth would have controlled traversal of that relation path.

[Fay Wang]

By default, the max fetch depth is set to -1 for no limit. However, this graph is an "indirect recursion", i.e., from State -> Transition -> State.  I think this makes Openjpa to stop prematurely...

Fay




----- Original Message ----
From: calin014 <[hidden email]>
To: [hidden email]
Sent: Fri, October 23, 2009 2:13:56 AM
Subject: Re: Fetchgroups recursion problem


Thanks for the quick reply.

I tried that but with no luck.
I did something like this before executing the query:

OpenJPAEntityManager ojem = OpenJPAPersistence.cast(entityManager);
System.out.println("getMaxFetchDepth() before: " +
ojem.getFetchPlan().getMaxFetchDepth());
ojem.getFetchPlan().setMaxFetchDepth(15);
System.out.println("getMaxFetchDepth() after: " +
ojem.getFetchPlan().getMaxFetchDepth());
ojem.getFetchPlan().addFetchGroups("State_OutgoingTransitions",
"IncomingTransitions");

The output is:

getMaxFetchDepth() before: -1
getMaxFetchDepth() after: 15

The result is the same as described in the first post.

Am i doing it wrong?


Pinaki Poddar wrote:

>
> Hi,
> There are two attributes that control the closure of a graph as fetched by
> a FetchPlan/FetchConfiguration, namely
>    Recursion Depth
>    Max Fetch Depth
>
> According to the cited use case, Max Fetch Depth is the relevant attribute
> that will control depth of traversal from a root entity (s1). By default,
> the max fetch depth is set to 1 and hence the immediate neighbors of s1
> are fetched and not s4 or s5 which is at depth 2 from s1.
>
> Recursion depth, on the other hand, controls the depth of traversal for
> recursive relation on types. If s1 had a recursive relation then recursion
> depth would have controlled traversal of that relation path.
>
>

--
View this message in context: http://n2.nabble.com/Fetchgroups-recursion-problem-tp3874382p3877617.html
Sent from the OpenJPA Users mailing list archive at Nabble.com.



     

[Craig L Russell]

One other thing to consider: there is a fetch plan for each query that  
may be different from the fetch plan for other operations. If you're  
modifying the fetch plan after the query is created, the modifications  
won't affect the query.

Craig

On Oct 23, 2009, at 8:21 AM, Fay Wang wrote:

> By default, the max fetch depth is set to -1 for no limit. However,  
> this graph is an "indirect recursion", i.e., from State ->  
> Transition -> State.  I think this makes Openjpa to stop  
> prematurely...
>
> Fay
>
>
>
>
> ----- Original Message ----
> From: calin014 <[hidden email]>
> To: [hidden email]
> Sent: Fri, October 23, 2009 2:13:56 AM
> Subject: Re: Fetchgroups recursion problem
>
>
> Thanks for the quick reply.
>
> I tried that but with no luck.
> I did something like this before executing the query:
>
> OpenJPAEntityManager ojem = OpenJPAPersistence.cast(entityManager);
> System.out.println("getMaxFetchDepth() before: " +
> ojem.getFetchPlan().getMaxFetchDepth());
> ojem.getFetchPlan().setMaxFetchDepth(15);
> System.out.println("getMaxFetchDepth() after: " +
> ojem.getFetchPlan().getMaxFetchDepth());
> ojem.getFetchPlan().addFetchGroups("State_OutgoingTransitions",
> "IncomingTransitions");
>
> The output is:
>
> getMaxFetchDepth() before: -1
> getMaxFetchDepth() after: 15
>
> The result is the same as described in the first post.
>
> Am i doing it wrong?
>
>
> Pinaki Poddar wrote:
>>
>> Hi,
>> There are two attributes that control the closure of a graph as  
>> fetched by
>> a FetchPlan/FetchConfiguration, namely
>>   Recursion Depth
>>   Max Fetch Depth
>>
>> According to the cited use case, Max Fetch Depth is the relevant  
>> attribute
>> that will control depth of traversal from a root entity (s1). By  
>> default,
>> the max fetch depth is set to 1 and hence the immediate neighbors  
>> of s1
>> are fetched and not s4 or s5 which is at depth 2 from s1.
>>
>> Recursion depth, on the other hand, controls the depth of traversal  
>> for
>> recursive relation on types. If s1 had a recursive relation then  
>> recursion
>> depth would have controlled traversal of that relation path.
>>
>>
>
> --
> View this message in context: http://n2.nabble.com/Fetchgroups-recursion-problem-tp3874382p3877617.html
> Sent from the OpenJPA Users mailing list archive at Nabble.com.
>
>
>
>

Craig L Russell
Architect, Sun Java Enterprise System http://db.apache.org/jdo
408 276-5638 mailto:[hidden email]
P.S. A good JDO? O, Gasp!

[calin014]

Craig L Russell wrote:

One other thing to consider: there is a fetch plan for each query that  
may be different from the fetch plan for other operations. If you're  
modifying the fetch plan after the query is created, the modifications  
won't affect the query.

Well, the order is:

OpenJPAEntityManager ojem = OpenJPAPersistence.cast(entityManager);
ojem.getFetchPlan().setMaxFetchDepth(15);
ojem.getFetchPlan().addFetchGroups("State_OutgoingTransitions", "State_IncomingTransitions");

Flow flow = entityManager.find(Flow.class, id);

And the query is affected because, if i don't add the fetch groups, only the first state gets loaded into memory.

Fay Wang wrote:

By default, the max fetch depth is set to -1 for no limit. However, this graph is an "indirect recursion", i.e., from State -> Transition -> State.  I think this makes Openjpa to stop prematurely...

If this is the case, is there a workaround?

[Fay Wang]

This apparently has something to do with the _availableRecursion in the FetchConfigurationImpl. I hardcoded a big number and the whole thing is retrieved. There seem a missing link between the recursionDepth in the FetchGroup and this field. More investigation is underway...

Fay




----- Original Message ----
From: calin014 <[hidden email]>
To: [hidden email]
Sent: Fri, October 23, 2009 10:37:46 AM
Subject: Re: Fetchgroups recursion problem



Craig L Russell wrote:
>
> One other thing to consider: there is a fetch plan for each query that  
> may be different from the fetch plan for other operations. If you're  
> modifying the fetch plan after the query is created, the modifications  
> won't affect the query.
>

Well, the order is:

OpenJPAEntityManager ojem = OpenJPAPersistence.cast(entityManager);
ojem.getFetchPlan().setMaxFetchDepth(15);
ojem.getFetchPlan().addFetchGroups("State_OutgoingTransitions",
"State_IncomingTransitions");

Flow flow = entityManager.find(Flow.class, id);

And the query is affected because, if i don't add the fetch groups, only the
first state gets loaded into memory.



Fay Wang wrote:
>
> By default, the max fetch depth is set to -1 for no limit. However, this
> graph is an "indirect recursion", i.e., from State -> Transition -> State.
> I think this makes Openjpa to stop prematurely...
>

If this is the case, is there a workaround?

--
View this message in context: http://n2.nabble.com/Fetchgroups-recursion-problem-tp3874382p3880166.html
Sent from the OpenJPA Users mailing list archive at Nabble.com.



     

[Fay Wang]

Hi,
        Since this is an indirect recursion, i.e., from State -> Transition -> State -> Transition -> ...
you need to add the recursionDepth in both State and Transition as shown below:

(1) State.java
@Entity
@FetchGroups( {
        @FetchGroup(name = "State_OutgoingTransitions",
                attributes = {@FetchAttribute(name = "outgoingTransitions", recursionDepth = 10) }),
        @FetchGroup(name = "State_IncomingTransitions",
                attributes = {@FetchAttribute(name = "incomingTransitions", recursionDepth = 10) }) })
               
public class State implements Serializable {
    @Id
    @GeneratedValue(strategy = GenerationType.IDENTITY)
    private Long id;

    @Column(length = 64)
    private String name;

    @OneToMany(mappedBy = "fromState",
             cascade = {CascadeType.PERSIST, CascadeType.MERGE, CascadeType.REFRESH})
    private List<Transition> outgoingTransitions;

    @OneToMany(mappedBy = "toState",
            cascade = { CascadeType.PERSIST, CascadeType.MERGE, CascadeType.REFRESH })
    private List<Transition> incomingTransitions;
   
...
}

(2) Transition.java
@Entity
@FetchGroups( {
    @FetchGroup(name = "Transition_ToState",
            attributes = {@FetchAttribute(name = "toState", recursionDepth = 10) }),
    @FetchGroup(name = "Transition_FromState",
            attributes = {@FetchAttribute(name = "fromState", recursionDepth = 10) }) })
           

public class Transition implements Serializable {
    @Id
    @GeneratedValue(strategy = GenerationType.IDENTITY)
    private Long id;

    @Column(length = 64)
    private String name;

    @ManyToOne(cascade = { CascadeType.PERSIST, CascadeType.MERGE,
            CascadeType.REFRESH })
    @JoinColumn(name = "FROM_STATE_ID", referencedColumnName = "ID")
    private State fromState;

    @ManyToOne(cascade = { CascadeType.PERSIST, CascadeType.MERGE,
            CascadeType.REFRESH })
    @JoinColumn(name = "TO_STATE_ID", referencedColumnName = "ID")
    private State toState;
...
}


(3) When doing the findBy operation, add the fetch groups to the fetch plan:

        EntityManager em = emf.createEntityManager();
        OpenJPAEntityManager oem = OpenJPAPersistence.cast (em);
        JDBCFetchPlan jfp = (JDBCFetchPlan)oem.getFetchPlan();
        jfp.addFetchGroup("State_OutgoingTransitions");
        jfp.addFetchGroup("State_IncomingTransitions");
        jfp.addFetchGroup("Transition_ToState");
        jfp.addFetchGroup("Transition_FromState");
       
        State s1 = em.find(State.class, 2);

The resulting s1 should contain the complete graph.

Regards,
Fay



:    





----- Original Message ----
From: Fay Wang <[hidden email]>
To: [hidden email]
Sent: Fri, October 23, 2009 2:44:13 PM
Subject: Re: Fetchgroups recursion problem

This apparently has something to do with the _availableRecursion in the FetchConfigurationImpl. I hardcoded a big number and the whole thing is retrieved. There seem a missing link between the recursionDepth in the FetchGroup and this field. More investigation is underway...

Fay




----- Original Message ----
From: calin014 <[hidden email]>
To: [hidden email]
Sent: Fri, October 23, 2009 10:37:46 AM
Subject: Re: Fetchgroups recursion problem



Craig L Russell wrote:
>
> One other thing to consider: there is a fetch plan for each query that  
> may be different from the fetch plan for other operations. If you're  
> modifying the fetch plan after the query is created, the modifications  
> won't affect the query.
>

Well, the order is:

OpenJPAEntityManager ojem = OpenJPAPersistence.cast(entityManager);
ojem.getFetchPlan().setMaxFetchDepth(15);
ojem.getFetchPlan().addFetchGroups("State_OutgoingTransitions",
"State_IncomingTransitions");

Flow flow = entityManager.find(Flow.class, id);

And the query is affected because, if i don't add the fetch groups, only the
first state gets loaded into memory.



Fay Wang wrote:
>
> By default, the max fetch depth is set to -1 for no limit. However, this
> graph is an "indirect recursion", i.e., from State -> Transition -> State.
> I think this makes Openjpa to stop prematurely...
>

If this is the case, is there a workaround?

--
View this message in context: http://n2.nabble.com/Fetchgroups-recursion-problem-tp3874382p3880166.html
Sent from the OpenJPA Users mailing list archive at Nabble.com.


     

[calin014]

Hello,

Thank you for your response. It worked. However, it works just when loading the graph from the first state. If i try to load the flow directly it doesn't work.

This is what i did:

@Entity
@FetchGroups( {
                @FetchGroup(name = "State_outgoingTransitions", attributes = { @FetchAttribute(name = "outgoingTransitions", recursionDepth = -1) }),
                @FetchGroup(name = "State_incomingTransitions", attributes = { @FetchAttribute(name = "incomingTransitions", recursionDepth = -1) }) })
public class State implements Serializable {
...

@Entity
@FetchGroups( {
                @FetchGroup(name = "State_outgoingTransitions", attributes = { @FetchAttribute(name = "fromState", recursionDepth = -1) }),
                @FetchGroup(name = "State_incomingTransitions", attributes = { @FetchAttribute(name = "toState", recursionDepth = -1) }) })
public class Transition implements Serializable {
...


...
System.out.println("FLOW: LAZY LOADING: ");
EntityManager em = emf.createEntityManager();
Flow f = em.find(Flow.class, 2L);
System.out.println(f);
em.close();
                       

System.out.println("FLOW: FETCH GROUPS: ");
em = emf.createEntityManager();
OpenJPAEntityManager oem = OpenJPAPersistence.cast(em);
oem.getFetchPlan().addFetchGroups("State_outgoingTransitions", "State_incomingTransitions");
f = em.find(Flow.class, 2L);
               
//close persistence context
em.close();
System.out.println(f);
               
               
System.out.println("STATE: FETCH GROUPS");
em = emf.createEntityManager();
oem = OpenJPAPersistence.cast(em);
oem.getFetchPlan().addFetchGroups("State_outgoingTransitions", "State_incomingTransitions");
State s = em.find(State.class, 4L);

//close persistence context
em.close();
       
//build the flow object
f = new Flow();
f.setInitialState(s);
System.out.println(f);
...

And this is the output:

FLOW: LAZY LOADING:
[s1, s2, s4, s5, s3]
[t12, t24, t45, t23, t34, t31, t32, t21]
FLOW: FETCH GROUPS:
[s1, s2, s3]
[t12, t24, t23, t34, t31, t32, t21]
STATE: FETCH GROUPS
[s1, s2, s4, s5, s3]
[t12, t24, t45, t23, t34, t31, t32, t21]


As you can see, when trying to load the flow directly with f = em.find(Flow.class, 2L) the result is the same as before.

Anyway, i think i can handle it from here. Thanks a lot for your help.

Fay Wang wrote:

Hi,
        Since this is an indirect recursion, i.e., from State -> Transition -> State -> Transition -> ...
you need to add the recursionDepth in both State and Transition as shown below:

(1) State.java
@Entity
@FetchGroups( {
        @FetchGroup(name = "State_OutgoingTransitions",
                attributes = {@FetchAttribute(name = "outgoingTransitions", recursionDepth = 10) }),
        @FetchGroup(name = "State_IncomingTransitions",
                attributes = {@FetchAttribute(name = "incomingTransitions", recursionDepth = 10) }) })
               
public class State implements Serializable {
    @Id
    @GeneratedValue(strategy = GenerationType.IDENTITY)
    private Long id;

    @Column(length = 64)
    private String name;

    @OneToMany(mappedBy = "fromState",
             cascade = {CascadeType.PERSIST, CascadeType.MERGE, CascadeType.REFRESH})
    private List<Transition> outgoingTransitions;

    @OneToMany(mappedBy = "toState",
            cascade = { CascadeType.PERSIST, CascadeType.MERGE, CascadeType.REFRESH })
    private List<Transition> incomingTransitions;
   
...
}

(2) Transition.java
@Entity
@FetchGroups( {
    @FetchGroup(name = "Transition_ToState",
            attributes = {@FetchAttribute(name = "toState", recursionDepth = 10) }),
    @FetchGroup(name = "Transition_FromState",
            attributes = {@FetchAttribute(name = "fromState", recursionDepth = 10) }) })
           

public class Transition implements Serializable {
    @Id
    @GeneratedValue(strategy = GenerationType.IDENTITY)
    private Long id;

    @Column(length = 64)
    private String name;

    @ManyToOne(cascade = { CascadeType.PERSIST, CascadeType.MERGE,
            CascadeType.REFRESH })
    @JoinColumn(name = "FROM_STATE_ID", referencedColumnName = "ID")
    private State fromState;

    @ManyToOne(cascade = { CascadeType.PERSIST, CascadeType.MERGE,
            CascadeType.REFRESH })
    @JoinColumn(name = "TO_STATE_ID", referencedColumnName = "ID")
    private State toState;
...
}


(3) When doing the findBy operation, add the fetch groups to the fetch plan:

        EntityManager em = emf.createEntityManager();
        OpenJPAEntityManager oem = OpenJPAPersistence.cast (em);
        JDBCFetchPlan jfp = (JDBCFetchPlan)oem.getFetchPlan();
        jfp.addFetchGroup("State_OutgoingTransitions");
        jfp.addFetchGroup("State_IncomingTransitions");
        jfp.addFetchGroup("Transition_ToState");
        jfp.addFetchGroup("Transition_FromState");
       
        State s1 = em.find(State.class, 2);

The resulting s1 should contain the complete graph.

Regards,
Fay

[Pinaki Poddar]

Hi,
  Committed change 830431. OPENJPA-944. Tested against your use case as appears in
openjpa-persistence-jdbc/src/test/java/org/apache/openjpa/persistence/kernel/TestIndirectRecursion.java

  Can you please confirm?

  Regards --

Pinaki