Searching on a field in a custom content type

9 messages Options
Embed this post
Permalink
audleman () Searching on a field in a custom content type
Reply Threaded More More options
Print post
Permalink
I am struggling with searching on a custom content type. I could use some help understanding how to search on, let's say, first name. My SpeakerSchema has that field defined as:

atapi.StringField('first_name',
        schemata='default',
        searchable=True,
        required=True,
        languageIndependent=True,
        widget = atapi.StringWidget(
            label=_(u'First Name'),
            description = _(u''),
            ),
        storage = atapi.AnnotationStorage(),
        ),

I have tried several things, including searching on the getFirst_name function:

    self.catalog = getToolByName(context, 'portal_catalog')
    self.catalog.searchResults({'portal_type' :'Speaker', 'getFirst_name' : 'Kevin'})

...searching on the field name itself

    self.catalog = getToolByName(context, 'portal_catalog')
    self.catalog.searchResults({'portal_type':'Speaker', 'first_name':'Kevin'})

In both cases the query returns all Speaker objects, totally ignoring the extra search parameter.

Any suggestions?

Thanks,
Kevin Audleman
Vincent Fretin () Re: Searching on a field in a custom content type
Reply Threaded More More options
Print post
Permalink
On Fri, Feb 27, 2009 at 12:10 AM, audleman <[hidden email]> wrote:

>
> I am struggling with searching on a custom content type. I could use some
> help understanding how to search on, let's say, first name. My SpeakerSchema
> has that field defined as:
>
> atapi.StringField('first_name',
>        schemata='default',
>        searchable=True,
>        required=True,
>        languageIndependent=True,
>        widget = atapi.StringWidget(
>            label=_(u'First Name'),
>            description = _(u''),
>            ),
>        storage = atapi.AnnotationStorage(),
>        ),
>
> I have tried several things, including searching on the getFirst_name
> function:
>
>    self.catalog = getToolByName(context, 'portal_catalog')
>    self.catalog.searchResults({'portal_type' :'Speaker', 'getFirst_name' :
> 'Kevin'})
>
> ...searching on the field name itself
>
>    self.catalog = getToolByName(context, 'portal_catalog')
>    self.catalog.searchResults({'portal_type':'Speaker',
> 'first_name':'Kevin'})
>
> In both cases the query returns all Speaker objects, totally ignoring the
> extra search parameter.
>
> Any suggestions?

You need a getFirst_name index too.
A getFirst_name should exist in portal_catalog.

In profiles/default/catalog.xml, you should have something like this:
<?xml version="1.0"?>
<object name="portal_catalog" meta_type="Plone Catalog Tool">
  <index name="getFirst_name"
         meta_type="FieldIndex">
    <indexed_attr value="getFirst_name"/>
  </index>
</object>

ArchGenXML takes care of generating all this code for you.

--
Vincent Fretin

------------------------------------------------------------------------------
Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco, CA
-OSBC tackles the biggest issue in open source: Open Sourcing the Enterprise
-Strategies to boost innovation and cut costs with open source participation
-Receive a $600 discount off the registration fee with the source code: SFAD
http://p.sf.net/sfu/XcvMzF8H
_______________________________________________
Archetypes-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/archetypes-users
audleman () Re: Searching on a field in a custom content type
Reply Threaded More More options
Print post
Permalink
That did the trick. Thank you both for your help. The Plone documentation may be sparse, but the help provided by the community is top notch, and much appreciated!

Kevin
audleman () Re: Searching on a field in a custom content type
Reply Threaded More More options
Print post
Permalink
Ok, I can search my getFirst_name field now, but I can't seem to perform a wildcard search. If I have the speaker 'Kevin' then according to the Zope book (http://www.zope.org/Documentation/Books/ZopeBook/2_6Edition/SearchingZCatalog.stx) I should be able to perform the following:

> self.catalog.searchResults(portal_type='Speaker', getFirst_name='K*')

However it returns an empty set. Searching on the full first name works fine:

> self.catalog.searchResults(portal_type='Speaker', getFirst_name='Kevin')

Help!

Kevin
audleman () Re: Searching on a field in a custom content type
Reply Threaded More More options
Print post
Permalink
While I'm at it, I also have to search a field called Topics. It is defined as a StringField with a MultiSelectionWidget (I'm really not sure if this is the right way to do it) like so:

atapi.StringField('topics',
        required=False,
        searchable=True,
        storage = atapi.AnnotationStorage(),
        validators = ('isTidyHtmlWithCleanup',),
        default_output_type = 'text/x-html-safe',
        widget = atapi.MultiSelectionWidget(
            label = _(u'Topics'),
            description = u'',
            allow_file_upload = False
            ),
        vocabulary=SPEAKERS_BUREAU_SPEAKER_TOPICS,
        ),

The following command shows what the storage for this field looks like:

>>> [s.getObject().getTopics() for s in self.catalog.searchResults(portal_type='Speaker', getFirst_name='Kevin')]
>>> [['Arts / Media', 'Economies / Work / Globalization', 'Race / Culture', '']]

Of course when I try to search on it, the search returns as if I hadn't specified that field at all

>>>[s.getObject().getTopics() for s in self.catalog.searchResults(portal_type='Speaker', getTopic=['Arts / Media'])]
>>> returns all speakers

>>>[s.getObject().getTopics() for s in self.catalog.searchResults(portal_type='Speaker', getTopic='Arts / Media')]
>>> returns all speakers

No clue on how to search on this. Any help greatly appreciated.

Kevin


audleman wrote:
Ok, I can search my getFirst_name field now, but I can't seem to perform a wildcard search. If I have the speaker 'Kevin' then according to the Zope book (http://www.zope.org/Documentation/Books/ZopeBook/2_6Edition/SearchingZCatalog.stx) I should be able to perform the following:

> self.catalog.searchResults(portal_type='Speaker', getFirst_name='K*')

However it returns an empty set. Searching on the full first name works fine:

> self.catalog.searchResults(portal_type='Speaker', getFirst_name='Kevin')

Help!

Kevin
Vincent Fretin () Re: Searching on a field in a custom content type
Reply Threaded More More options
Print post
Permalink
In reply to this post by audleman
On Fri, Feb 27, 2009 at 8:35 PM, audleman <[hidden email]> wrote:
>
> Ok, I can search my getFirst_name field now, but I can't seem to perform a
> wildcard search. If I have the speaker 'Kevin' then according to the Zope
> book
> (http://www.zope.org/Documentation/Books/ZopeBook/2_6Edition/SearchingZCatalog.stx)
Please use the new version of the book, it have been migrated recently to
http://docs.zope.org/
http://docs.zope.org/zope2/zope2book/source/SearchingZCatalog.html


> I should be able to perform the following:
>
>> self.catalog.searchResults(portal_type='Speaker', getFirst_name='K*')
>
> However it returns an empty set. Searching on the full first name works
> fine:
>
>> self.catalog.searchResults(portal_type='Speaker', getFirst_name='Kevin')
>
> Help!
>
> Kevin

I don't know for wildcard search. I never used it.
It's not possible to use a wildcard with a FieldIndex.
But the doc say you need a TextIndexNG index.
http://pypi.python.org/pypi/Products.TextIndexNG3
To change the type of your getFirst_name index, I guess you have to
remove your index in portal_catalog/Indexes
Change your catalog.xml with something like that:
  <index name="getMInformations"
         meta_type="TextIndexNG">
    <indexed_attr value="getMInformations"/>
    <extra name="index_type" value="Cosine Measure"/>
    <extra name="lexicon_id" value="plaintext_lexicon"/>
  </index>
reinstall your product and do a recatalog of your objects.

--
Vincent Fretin

------------------------------------------------------------------------------
Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco, CA
-OSBC tackles the biggest issue in open source: Open Sourcing the Enterprise
-Strategies to boost innovation and cut costs with open source participation
-Receive a $600 discount off the registration fee with the source code: SFAD
http://p.sf.net/sfu/XcvMzF8H
_______________________________________________
Archetypes-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/archetypes-users
Vincent Fretin () Re: Searching on a field in a custom content type
Reply Threaded More More options
Print post
Permalink
In reply to this post by audleman
On Fri, Feb 27, 2009 at 9:00 PM, audleman <[hidden email]> wrote:

>
> While I'm at it, I also have to search a field called Topics. It is defined
> as a StringField with a MultiSelectionWidget (I'm really not sure if this is
> the right way to do it) like so:
>
> atapi.StringField('topics',
>        required=False,
>        searchable=True,
>        storage = atapi.AnnotationStorage(),
>        validators = ('isTidyHtmlWithCleanup',),
>        default_output_type = 'text/x-html-safe',
>        widget = atapi.MultiSelectionWidget(
>            label = _(u'Topics'),
>            description = u'',
>            allow_file_upload = False
>            ),
>        vocabulary=SPEAKERS_BUREAU_SPEAKER_TOPICS,
>        ),
>
> The following command shows what the storage for this field looks like:
>
>>>> [s.getObject().getTopics() for s in
>>>> self.catalog.searchResults(portal_type='Speaker',
>>>> getFirst_name='Kevin')]
>>>> [['Arts / Media', 'Economies / Work / Globalization', 'Race / Culture',
>>>> '']]
>
> Of course when I try to search on it, the search returns as if I hadn't
> specified that field at all
>
>>>>[s.getObject().getTopics() for s in
> self.catalog.searchResults(portal_type='Speaker', getTopic=['Arts /
> Media'])]
>>>> returns all speakers
>
>>>>[s.getObject().getTopics() for s in
> self.catalog.searchResults(portal_type='Speaker', getTopic='Arts / Media')]
>>>> returns all speakers
>
> No clue on how to search on this. Any help greatly appreciated.
>
> Kevin

getTopics, not getTopic!
You need an index for this field too.
--
Vincent Fretin

------------------------------------------------------------------------------
Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco, CA
-OSBC tackles the biggest issue in open source: Open Sourcing the Enterprise
-Strategies to boost innovation and cut costs with open source participation
-Receive a $600 discount off the registration fee with the source code: SFAD
http://p.sf.net/sfu/XcvMzF8H
_______________________________________________
Archetypes-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/archetypes-users
Raphael Ritz () Re: Searching on a field in a custom content type
Reply Threaded More More options
Print post
Permalink
In reply to this post by audleman
audleman wrote:

Hi,

in addition to the help and advice that you've gotten
already I'd like to provide some further guidance:


> I am struggling with searching on a custom content type. I could use some
> help understanding how to search on, let's say, first name. My SpeakerSchema
> has that field defined as:
>
> atapi.StringField('first_name',
>         schemata='default',
>         searchable=True,

This means you make it available via the full text search
(it's added to the 'SearchableText').

In order to create a dedicated index for that field you need
to provide the appropriate snippet in your products installation
profile as others have pointed out.

Note that Zope's catalog provides different index types
(Field-, Keyword-, TextIndex, etc.) that provide different
functionality. Read the chapter in the Zope book that you have
been pointed to to understand the differences. Then you can
make an informed decision.

Sometimes, it is necessary to index the same information
several times to support different use cases (like globbing
- aka wild card support - versus sorting). For that reason
you can usually define the name of the index to be different
from the attribute/method to call.

Doing a search for 'K*' on a (ZC)TextIndex(NG) and expecting
everything that starts with 'K' in return can be a bit too naive
depending on how you configured the index. Some index types
interpret non-ascii characters as stop characters for instance
so you might get false positives.

Finally, using a StringField with a MultiSelectWidget is usually
a bad idea as multiple selections should usually result in a
list-like data type (or a collection). While techically the
'string' type is a sequence (of characters) you might want
to consider using a lines field instead.
Also, when indexing such a field specifically, what you most
often want is a KeywordIndex. But only thinking
about your intended use cases will tell.

HTH

        Raphael



>         required=True,
>         languageIndependent=True,
>         widget = atapi.StringWidget(
>             label=_(u'First Name'),
>             description = _(u''),
>             ),
>         storage = atapi.AnnotationStorage(),
>         ),
>
> I have tried several things, including searching on the getFirst_name
> function:
>
>     self.catalog = getToolByName(context, 'portal_catalog')
>     self.catalog.searchResults({'portal_type' :'Speaker', 'getFirst_name' :
> 'Kevin'})
>
> ...searching on the field name itself
>
>     self.catalog = getToolByName(context, 'portal_catalog')
>     self.catalog.searchResults({'portal_type':'Speaker',
> 'first_name':'Kevin'})
>
> In both cases the query returns all Speaker objects, totally ignoring the
> extra search parameter.
>
> Any suggestions?
>
> Thanks,
> Kevin Audleman


------------------------------------------------------------------------------
Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco, CA
-OSBC tackles the biggest issue in open source: Open Sourcing the Enterprise
-Strategies to boost innovation and cut costs with open source participation
-Receive a $600 discount off the registration fee with the source code: SFAD
http://p.sf.net/sfu/XcvMzF8H
_______________________________________________
Archetypes-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/archetypes-users
audleman () Re: [Archetypes-users] Searching on a field in a custom content type
Reply Threaded More More options
Print post
Permalink
Some javascript/style in this post has been disabled (why?)
Raphael,

Hopefully I'm responding to only you, not the mailing list. Thanks a lot, this is very helpful information. I will read the chapter in more detail to get a better sense for how to configure indexes.

Cheers,
Kevin

Raphael Ritz (via Nabble) wrote:
audleman wrote:

Hi,

in addition to the help and advice that you've gotten
already I'd like to provide some further guidance:


> I am struggling with searching on a custom content type. I could use some
> help understanding how to search on, let's say, first name. My SpeakerSchema
> has that field defined as:
>
> atapi.StringField('first_name',
>         schemata='default',
>         searchable=True,

This means you make it available via the full text search
(it's added to the 'SearchableText').

In order to create a dedicated index for that field you need
to provide the appropriate snippet in your products installation
profile as others have pointed out.

Note that Zope's catalog provides different index types
(Field-, Keyword-, TextIndex, etc.) that provide different
functionality. Read the chapter in the Zope book that you have
been pointed to to understand the differences. Then you can
make an informed decision.

Sometimes, it is necessary to index the same information
several times to support different use cases (like globbing
- aka wild card support - versus sorting). For that reason
you can usually define the name of the index to be different
from the attribute/method to call.

Doing a search for 'K*' on a (ZC)TextIndex(NG) and expecting
everything that starts with 'K' in return can be a bit too naive
depending on how you configured the index. Some index types
interpret non-ascii characters as stop characters for instance
so you might get false positives.

Finally, using a StringField with a MultiSelectWidget is usually
a bad idea as multiple selections should usually result in a
list-like data type (or a collection). While techically the
'string' type is a sequence (of characters) you might want
to consider using a lines field instead.
Also, when indexing such a field specifically, what you most
often want is a KeywordIndex. But only thinking
about your intended use cases will tell.

HTH

        Raphael



>         required=True,
>         languageIndependent=True,
>         widget = atapi.StringWidget(
>             label=_(u'First Name'),
>             description = _(u''),
>             ),
>         storage = atapi.AnnotationStorage(),
>         ),
>
> I have tried several things, including searching on the getFirst_name
> function:
>
>     self.catalog = getToolByName(context, 'portal_catalog')
>     self.catalog.searchResults({'portal_type' :'Speaker', 'getFirst_name' :
> 'Kevin'})
>
> ...searching on the field name itself
>
>     self.catalog = getToolByName(context, 'portal_catalog')
>     self.catalog.searchResults({'portal_type':'Speaker',
> 'first_name':'Kevin'})
>
> In both cases the query returns all Speaker objects, totally ignoring the
> extra search parameter.
>
> Any suggestions?
>
> Thanks,
> Kevin Audleman


------------------------------------------------------------------------------
Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco, CA
-OSBC tackles the biggest issue in open source: Open Sourcing the Enterprise
-Strategies to boost innovation and cut costs with open source participation
-Receive a $600 discount off the registration fee with the source code: SFAD
http://p.sf.net/sfu/XcvMzF8H
_______________________________________________
Archetypes-users mailing list
Archetypes-users@...
https://lists.sourceforge.net/lists/listinfo/archetypes-users