Filtering with Many-to-Many relationship

10 messages Options
Embed this post
Permalink
Shawn Marincas

Filtering with Many-to-Many relationship

Reply Threaded More More options
Print post
Permalink
Alright, it seemed to me that what I wanted to do was straightforward, but it doesn't seem like I can find any resources on how to do it... so maybe I'm just going about it the wrong way.

I have a table Products and a table Tags with a many_to_many relationship setup, the purpose of which would be to easily generate a resultset of products with the given tag(s).  However, I'm running in to problems when I want to generate a resultset of product records filtered with multiple tags.  I was thinking that one of the following would work:

my $tag1 = 'A';
my $tag2 = 'B';

$c->model('MyApp:Tag')->single({ tag_id => $tag1 })->products->search({ tags => { tag_id => $tag2 }});

OR

$c->model('MyApp:Tag')->single({ tag_id => $tag1 })->products->search_related('product_tags', { tag_id  => $tag2 });

But apparently the many_to_many relationship accessor only returns a list of row_objects rather than a resultset itself, so I can't perform resultset searches on that.  So I tried the following to chain together two searches:

$c->model('MyApp:Product')->search_related('product_tags',{ tag_id => $tag1 })->search_related('product_tags', { tag_id => $tag2 });

I tried the same chaining using search and joins to link to the ProductTagsLink table, they combined the searches so that it was joining on the related table searching for tag_id = $tag1 AND tag_id = $tag2 so that it wasn't returning anything.  I was looking in to creating a subquery for this, but my attempts have failed and the documentation mentions that subqueries are experimental.  I'm thinking if I had an accessor to access the many_to_many resultset as an actual resultset rather than a list of row_objects I could probably get it working, but I didn't see any documentation on actually doing that.  I managed to get the actual concept working using this code here, but would prefer a straight SQL/DBIX solution since this doesn't seem very efficient:

my $tag1 = 'A';
my $tag2 = 'B';
my $tag_rs = $c->model('ReaneyDesignDB::Tag')->single({ tag_id => $tag });

my @products = $tag_rs->products;

$c->stash->{products} = [grep { ($_->search_related('product_tags', { tag_id => $tag2 })->single) } @products];

Anyone have experience trying to accomplish what I'm doing here?  Let me know if I'm missing something here.  Thanks.

- Shawn Marincas

_______________________________________________
List: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/dbix-class
IRC: irc.perl.org#dbix-class
SVN: http://dev.catalyst.perl.org/repos/bast/DBIx-Class/
Searchable Archive: http://www.grokbase.com/group/dbix-class@...
Wallace Reis

Re: Filtering with Many-to-Many relationship

Reply Threaded More More options
Print post
Permalink
On Fri, Sep 18, 2009 at 3:06 PM, Shawn Marincas
<[hidden email]> wrote:
> I have a table Products and a table Tags with a many_to_many relationship
> setup, the purpose of which would be to easily generate a resultset of
> products with the given tag(s).  However, I'm running in to problems when I
> want to generate a resultset of product records filtered with multiple
> tags.  I was thinking that one of the following would work:
>
> my $tag1 = 'A';
> my $tag2 = 'B';

my $product_rs = $schema->resultset('Product')->search_rs({
    'tag.tag_id' => { -in => [qw/A B/] }
}, {
    join => { 'product_tags' => 'tag' }
});

> But apparently the many_to_many relationship accessor only returns a list of
> row_objects rather than a resultset itself, so I can't perform resultset
> searches on that.  So I tried the following to chain together two searches:

You've used the ->search method in your code in list context, so it
calls ->all over the resultset and returns a list of row objects.

--
     wallace reis/wreis         Catalyst and DBIx::Class consultancy with a clue
     Software Engineer and a commit bit: http://shadowcat.co.uk/catalyst/
 Shadowcat Systems Limited
 http://www.shadowcat.co.uk     http://www.linkedin.com/in/wallacereis

_______________________________________________
List: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/dbix-class
IRC: irc.perl.org#dbix-class
SVN: http://dev.catalyst.perl.org/repos/bast/DBIx-Class/
Searchable Archive: http://www.grokbase.com/group/dbix-class@...
Shawn Marincas

Re: Filtering with Many-to-Many relationship

Reply Threaded More More options
Print post
Permalink
It seems that the above query won't work in any case since searching for tag_id IN a set will still returned the results OR'd together, and I need the set of products which contain BOTH tags.  I've been toying with it some more and came up with a solution which uses a subquery, still not sure if this is the optimum solution yet or not.

$filtered_product_rs = $schema->resultset('Product')->search({
    'product_tags.tag_id' => $tag2_rs->tag_id,
    'me.product_id' => {
        -in => [
            $schema->resultset('Product')->search_rs({
                'product_tags.tag_id' => $tag_rs->tag_id
            },{
                join => 'product_tags'
            })->get_column('product_id')->all
        ]
    }
}, {
    join=> 'product_tags'
});


On Fri, Sep 18, 2009 at 7:35 PM, Wallace Reis <[hidden email]> wrote:
On Fri, Sep 18, 2009 at 3:06 PM, Shawn Marincas
<[hidden email]> wrote:
> I have a table Products and a table Tags with a many_to_many relationship
> setup, the purpose of which would be to easily generate a resultset of
> products with the given tag(s).  However, I'm running in to problems when I
> want to generate a resultset of product records filtered with multiple
> tags.  I was thinking that one of the following would work:
>
> my $tag1 = 'A';
> my $tag2 = 'B';

my $product_rs = $schema->resultset('Product')->search_rs({
   'tag.tag_id' => { -in => [qw/A B/] }
}, {
   join => { 'product_tags' => 'tag' }
});

> But apparently the many_to_many relationship accessor only returns a list of
> row_objects rather than a resultset itself, so I can't perform resultset
> searches on that.  So I tried the following to chain together two searches:

You've used the ->search method in your code in list context, so it
calls ->all over the resultset and returns a list of row objects.

--
    wallace reis/wreis         Catalyst and DBIx::Class consultancy with a clue
    Software Engineer          and a commit bit: http://shadowcat.co.uk/catalyst/
 Shadowcat Systems Limited
 http://www.shadowcat.co.uk     http://www.linkedin.com/in/wallacereis

_______________________________________________
List: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/dbix-class
IRC: irc.perl.org#dbix-class
SVN: http://dev.catalyst.perl.org/repos/bast/DBIx-Class/
Searchable Archive: http://www.grokbase.com/group/dbix-class@...


_______________________________________________
List: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/dbix-class
IRC: irc.perl.org#dbix-class
SVN: http://dev.catalyst.perl.org/repos/bast/DBIx-Class/
Searchable Archive: http://www.grokbase.com/group/dbix-class@...
Wallace Reis

Re: Filtering with Many-to-Many relationship

Reply Threaded More More options
Print post
Permalink
On Sat, Sep 19, 2009 at 2:16 PM, Shawn Marincas
<[hidden email]> wrote:
> It seems that the above query won't work in any case since searching for
> tag_id IN a set will still returned the results OR'd together, and I need
> the set of products which contain BOTH tags.  I've been toying with it some
> more and came up with a solution which uses a subquery, still not sure if
> this is the optimum solution yet or not.

Right. You need of a query using XOR op in WHERE clause plus DISTINCT clause.

--
     wallace reis/wreis         Catalyst and DBIx::Class consultancy with a clue
     Software Engineer and a commit bit: http://shadowcat.co.uk/catalyst/
 Shadowcat Systems Limited
 http://www.shadowcat.co.uk     http://www.linkedin.com/in/wallacereis

_______________________________________________
List: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/dbix-class
IRC: irc.perl.org#dbix-class
SVN: http://dev.catalyst.perl.org/repos/bast/DBIx-Class/
Searchable Archive: http://www.grokbase.com/group/dbix-class@...
Oleg Kostyuk-2

Re: Filtering with Many-to-Many relationship

Reply Threaded More More options
Print post
Permalink
In reply to this post by Shawn Marincas
I think, you just need to join twice with tags table, and this will
give you what you want:

my $product_rs = $schema->resultset('Product')->search_rs(
  { # conditions will be AND'ed by default
   'product_tags.tag_id' => 'tagA',
   'product_tags_2.tag_id' => 'tagB',
  }, {
   join => [ 'product_tags', 'product_tags' ],
  }
);

For manuals, see here:
http://search.cpan.org/~ribasushi/DBIx-Class-0.08112/lib/DBIx/Class/Manual/Joining.pod#Joining_to_the_same_table_twice

PS: I will not be surprised, if this variant will be faster than all other ;)

--
Sincerely yours,
Oleg Kostyuk (CUB-UANIC)

_______________________________________________
List: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/dbix-class
IRC: irc.perl.org#dbix-class
SVN: http://dev.catalyst.perl.org/repos/bast/DBIx-Class/
Searchable Archive: http://www.grokbase.com/group/dbix-class@...
Wallace Reis

Re: Filtering with Many-to-Many relationship

Reply Threaded More More options
Print post
Permalink
In reply to this post by Wallace Reis
On Sat, Sep 19, 2009 at 3:23 PM, Wallace Reis <[hidden email]> wrote:
> On Sat, Sep 19, 2009 at 2:16 PM, Shawn Marincas
> <[hidden email]> wrote:
>> It seems that the above query won't work in any case since searching for
>> tag_id IN a set will still returned the results OR'd together, and I need
>> the set of products which contain BOTH tags.  I've been toying with it some
>> more and came up with a solution which uses a subquery, still not sure if
>> this is the optimum solution yet or not.
>
> Right. You need of a query using XOR op in WHERE clause plus DISTINCT clause.

Hmm. Sorry, that won't work for your case.

--
     wallace reis/wreis         Catalyst and DBIx::Class consultancy with a clue
     Software Engineer and a commit bit: http://shadowcat.co.uk/catalyst/
 Shadowcat Systems Limited
 http://www.shadowcat.co.uk     http://www.linkedin.com/in/wallacereis

_______________________________________________
List: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/dbix-class
IRC: irc.perl.org#dbix-class
SVN: http://dev.catalyst.perl.org/repos/bast/DBIx-Class/
Searchable Archive: http://www.grokbase.com/group/dbix-class@...
Wallace Reis

Re: Filtering with Many-to-Many relationship

Reply Threaded More More options
Print post
Permalink
In reply to this post by Oleg Kostyuk-2
On Tue, Sep 22, 2009 at 7:23 AM, Oleg Kostyuk <[hidden email]> wrote:
> I think, you just need to join twice with tags table, and this will
> give you what you want:

And if you need products which have 3 or more tags? One join per tag?
It is not ideal.

--
     wallace reis/wreis         Catalyst and DBIx::Class consultancy with a clue
     Software Engineer and a commit bit: http://shadowcat.co.uk/catalyst/
 Shadowcat Systems Limited
 http://www.shadowcat.co.uk     http://www.linkedin.com/in/wallacereis

_______________________________________________
List: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/dbix-class
IRC: irc.perl.org#dbix-class
SVN: http://dev.catalyst.perl.org/repos/bast/DBIx-Class/
Searchable Archive: http://www.grokbase.com/group/dbix-class@...
Oleg Kostyuk-2

Re: Filtering with Many-to-Many relationship

Reply Threaded More More options
Print post
Permalink
We have several tags, so we need make several lookups into tags table,
for translate tag to tag_id - for me this is obvious. I don't said
that my solution is best, but it's working, and this it's main
advantage. My solution will be fast, because it make lookups only
through primary keys. Of course, my solution is not ideal, but I don't
have such intentions - make it ideal. It's just response to concrete
question from topic starter.

>From other hand, your solution use something unknown operation XOR.
Where is manual for XOR in SQL standart?. Seems that XOR is
implemented in MS-SQL, but this is DBIx::Class mailing list, not
MS-SQL. Your solution use DISTINCT for unknown reasons. Even if it
will be working - it will never be fast.

I hope that topic starter will be glad to see some useful response. If
personally you want to see something better, then you can hire me :)

--
Sincerely yours,
Oleg Kostyuk (CUB-UANIC)

_______________________________________________
List: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/dbix-class
IRC: irc.perl.org#dbix-class
SVN: http://dev.catalyst.perl.org/repos/bast/DBIx-Class/
Searchable Archive: http://www.grokbase.com/group/dbix-class@...
Wallace Reis

Re: Filtering with Many-to-Many relationship

Reply Threaded More More options
Print post
Permalink
On Tue, Sep 22, 2009 at 10:25 AM, Oleg Kostyuk <[hidden email]> wrote:
> >From other hand, your solution use something unknown operation XOR.
> Where is manual for XOR in SQL standart?. Seems that XOR is
> implemented in MS-SQL, but this is DBIx::Class mailing list, not
> MS-SQL.

Yes, not SQL standard, but MySQL implements it as well. That's one
reason why i came back and sait it wouldn't work.

> I hope that topic starter will be glad to see some useful response. If
> personally you want to see something better, then you can hire me :)

He asked for -the optimum solution-, so if people need to hire you in
order to get your best answer, so don't loose your time helping them
into community MLs.

--
     wallace reis/wreis         Catalyst and DBIx::Class consultancy with a clue
     Software Engineer and a commit bit: http://shadowcat.co.uk/catalyst/
 Shadowcat Systems Limited
 http://www.shadowcat.co.uk     http://www.linkedin.com/in/wallacereis

_______________________________________________
List: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/dbix-class
IRC: irc.perl.org#dbix-class
SVN: http://dev.catalyst.perl.org/repos/bast/DBIx-Class/
Searchable Archive: http://www.grokbase.com/group/dbix-class@...
Matt Whipple

Re: Filtering with Many-to-Many relationship

Reply Threaded More More options
Print post
Permalink
In reply to this post by Shawn Marincas
Shawn Marincas wrote:
> It seems that the above query won't work in any case since searching
> for tag_id IN a set will still returned the results OR'd together, and
> I need the set of products which contain BOTH tags.  I've been toying
> with it some more and came up with a solution which uses a subquery,
> still not sure if this is the optimum solution yet or not.
If you're worried about optimization, the one thing I would watch for is
to ensure that the DBMS is still able to optimize the subquery by
determining the most restrictive initial set of data, or otherwise
implementing that optimization manually.  With something like tags I
would assume that some are far more general than others so this could
have a significant impact.  This is something that would normally occur
with a self-join, but a subquery is likely a more fitting solution in
your case.

I may also suggest coupling this with a digest form of the tags (similar
to a SET data type) which can then be run through a bitwise AND.  This
would require organizing the tags (possibly into word-sized categories)
but should allow for very quick checks that wouldn't add overhead as
searches grew more complex.

>
> $filtered_product_rs = $schema->resultset('Product')->search({
>     'product_tags.tag_id' => $tag2_rs->tag_id,
>     'me.product_id' => {
>         -in => [
>             $schema->resultset('Product')->search_rs({
>                 'product_tags.tag_id' => $tag_rs->tag_id
>             },{
>                 join => 'product_tags'
>             })->get_column('product_id')->all
>         ]
>     }
> }, {
>     join=> 'product_tags'
> });
>
>
> On Fri, Sep 18, 2009 at 7:35 PM, Wallace Reis <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     On Fri, Sep 18, 2009 at 3:06 PM, Shawn Marincas
>     <[hidden email] <mailto:[hidden email]>> wrote:
>     > I have a table Products and a table Tags with a many_to_many
>     relationship
>     > setup, the purpose of which would be to easily generate a
>     resultset of
>     > products with the given tag(s).  However, I'm running in to
>     problems when I
>     > want to generate a resultset of product records filtered with
>     multiple
>     > tags.  I was thinking that one of the following would work:
>     >
>     > my $tag1 = 'A';
>     > my $tag2 = 'B';
>
>     my $product_rs = $schema->resultset('Product')->search_rs({
>        'tag.tag_id' => { -in => [qw/A B/] }
>     }, {
>        join => { 'product_tags' => 'tag' }
>     });
>
>     > But apparently the many_to_many relationship accessor only
>     returns a list of
>     > row_objects rather than a resultset itself, so I can't perform
>     resultset
>     > searches on that.  So I tried the following to chain together
>     two searches:
>
>     You've used the ->search method in your code in list context, so it
>     calls ->all over the resultset and returns a list of row objects.
>
>     --
>         wallace reis/wreis         Catalyst and DBIx::Class
>     consultancy with a clue
>         Software Engineer          and a commit bit:
>     http://shadowcat.co.uk/catalyst/
>      Shadowcat Systems Limited
>      http://www.shadowcat.co.uk     http://www.linkedin.com/in/wallacereis
>
>     _______________________________________________
>     List: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/dbix-class
>     IRC: irc.perl.org#dbix-class <http://irc.perl.org#dbix-class>
>     SVN: http://dev.catalyst.perl.org/repos/bast/DBIx-Class/
>     Searchable Archive:
>     http://www.grokbase.com/group/dbix-class@...
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> List: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/dbix-class
> IRC: irc.perl.org#dbix-class
> SVN: http://dev.catalyst.perl.org/repos/bast/DBIx-Class/
> Searchable Archive: http://www.grokbase.com/group/dbix-class@...


_______________________________________________
List: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/dbix-class
IRC: irc.perl.org#dbix-class
SVN: http://dev.catalyst.perl.org/repos/bast/DBIx-Class/
Searchable Archive: http://www.grokbase.com/group/dbix-class@...