Folder navigation (GSoC) -- project updates

[Sean Upton]

All:

I have put together some detailed updates on my GSoC project for folder navigation user-interface enhancements to support folders with many items contained.  The primary focus of this project is faceted navigation for folder listings.

http://www.mostscript.com/sean/2009/6/27/folder-ui:-gsoc-project-update-6-27

I have been working bottom-up on some backend pieces for this project, and I am now shifting gears to writing views, viewlets, and configuration persistence.  My goal is to have a simple cohesive (demonstratable) user-interface prototype in the next few weeks.  The next week's work is focused on integration testing of a yet-untested "query runner" component I wrote using AdvancedQuery, as well as views and viewlets for each facet, and some simple configuration mechanisms to provide for what is displayed in the faceted UI.

Early code is in svn as plone.app.folderui -- this is not yet very cohesive, but I aim for it to be very soon. 

Any feedback, questions, or comments: they are very much appreciated please feel free to leave them here on the list or on my blog.

Thanks,
Sean

------------------------------------------------------------------------------

_______________________________________________
Plone-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-developers

[Laurence Rowe]

Hi Sean,

Please take a look at collective.solr [1] - we recently implemented
facted search for a customer of ours. I believe it will be very
difficult to efficiently retrieve facets from ZCatalog, performing a
separate search for each facet will give very bad performance, so
depending on solr for this functionality seems reasonable to me.

Regards,

Laurence

[1] http://pypi.python.org/pypi/collective.solr

Sean Upton wrote:

> All:
>
> I have put together some detailed updates on my GSoC project for folder
> navigation user-interface enhancements to support folders with many
> items contained.  The primary focus of this project is faceted
> navigation for folder listings.
>
> http://www.mostscript.com/sean/2009/6/27/folder-ui:-gsoc-project-update-6-27
>
> I have been working bottom-up on some backend pieces for this project,
> and I am now shifting gears to writing views, viewlets, and
> configuration persistence.  My goal is to have a simple cohesive
> (demonstratable) user-interface prototype in the next few weeks.  The
> next week's work is focused on integration testing of a yet-untested
> "query runner" component I wrote using AdvancedQuery, as well as views
> and viewlets for each facet, and some simple configuration mechanisms to
> provide for what is displayed in the faceted UI.
>
> Early code is in svn as plone.app.folderui -- this is not yet very
> cohesive, but I aim for it to be very soon.
>
> Any feedback, questions, or comments: they are very much appreciated
> please feel free to leave them here on the list or on my blog.
>
> Thanks,
> Sean

------------------------------------------------------------------------------
_______________________________________________
Plone-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-developers

[Laurence Rowe]

Hi Sean,

Please take a look at collective.solr [1] - we recently implemented facted search for a customer of ours. I believe it will be very difficult to efficiently retrieve facets from ZCatalog, performing a separate search for each facet will give very bad performance, so depending on solr for this functionality seems reasonable to me.

Regards,

Laurence

[1] http://pypi.python.org/pypi/collective.solr 

Sean Upton wrote:

All:

I have put together some detailed updates on my GSoC project for folder
navigation user-interface enhancements to support folders with many items
contained.  The primary focus of this project is faceted navigation for
folder listings.

http://www.mostscript.com/sean/2009/6/27/folder-ui:-gsoc-project-update-6-27

I have been working bottom-up on some backend pieces for this project, and I
am now shifting gears to writing views, viewlets, and configuration
persistence.  My goal is to have a simple cohesive (demonstratable)
user-interface prototype in the next few weeks.  The next week's work is
focused on integration testing of a yet-untested "query runner" component I
wrote using AdvancedQuery, as well as views and viewlets for each facet, and
some simple configuration mechanisms to provide for what is displayed in the
faceted UI.

Early code is in svn as plone.app.folderui -- this is not yet very cohesive,
but I aim for it to be very soon.

Any feedback, questions, or comments: they are very much appreciated please
feel free to leave them here on the list or on my blog.

Thanks,
Sean

[Martin Aspeli]

Laurence Rowe wrote:

> Hi Sean,
>
> Please take a look at collective.solr [1] - we recently implemented
> facted search for a customer of ours. I believe it will be very
> difficult to efficiently retrieve facets from ZCatalog, performing a
> separate search for each facet will give very bad performance, so
> depending on solr for this functionality seems reasonable to me.
>
> Regards,
>
> Laurence
>
> [1] http://pypi.python.org/pypi/collective.solr

I think we need to be careful here. Introducing a solr/lucene dependency
is probably not going to be feasible for most sites. As I remember it,
Sean's mock-ups had lots of useful UI things that ought to be achievable
with ZCatalog. We probably want those to be deliverable independently on
any such radical architectural shift.

Martin

--
Author of `Professional Plone Development`, a book for developers who
want to work with Plone. See http://martinaspeli.net/plone-book


------------------------------------------------------------------------------
_______________________________________________
Plone-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-developers

[Laurence Rowe]

Martin Aspeli wrote:

Laurence Rowe wrote:
> Hi Sean,
>
> Please take a look at collective.solr [1] - we recently implemented
> facted search for a customer of ours. I believe it will be very
> difficult to efficiently retrieve facets from ZCatalog, performing a
> separate search for each facet will give very bad performance, so
> depending on solr for this functionality seems reasonable to me.

I think we need to be careful here. Introducing a solr/lucene dependency
is probably not going to be feasible for most sites. As I remember it,
Sean's mock-ups had lots of useful UI things that ought to be achievable
with ZCatalog. We probably want those to be deliverable independently on
any such radical architectural shift.

I'm not advocating we make Solr a dependency of Plone, just pointing out that we will not be able to ship UI that requires lots of ZCatalog requests while over at http://plone.uservoice.com "Improve overall speed" is the most requested feature.

Solr returns facet information in a single request. It might be possible to implement an efficient way of querying this from the ZCatalog data structures. Maybe http://pypi.python.org/pypi/experimental.catalogqueryplan can offer inspiration.

Laurence

[Sean Upton]

On Sat, Jun 27, 2009 at 5:42 AM, Laurence Rowe <[hidden email]> wrote:

Hi Sean,

Please take a look at collective.solr [1] - we recently implemented facted
search for a customer of ours. I believe it will be very difficult to
efficiently retrieve facets from ZCatalog, performing a separate search for
each facet will give very bad performance, so depending on solr for this
functionality seems reasonable to me.

An eventual goal should be multiple backends for running queries.  My immediate goal is small number of dependencies and interfaces that are implementation-neutral, so that writing a backend later for collective.solr would be possible.

The challenges for ZCatalog in this area:

(1) most indexing operations run in-band inside the application, which leaves few opportunities to offload the memory pressure this creates.  There is not much that can be done for this problem that is reasonably simple.

(2) there is not yet any means to cache individual filter sets (of catalog record ids) for each filter/facet choice (like filterCache does in Solr), which leaves few ways to compute set intersections other than re-running the queries.  This is not an impossible problem, just something that needs some clever work (I hope to do some of that work later this summer).

I plan to try and investigate what might be possible to improve #2, but my default assumption is that really big sites with lots of content would choose an out-of-band indexing system like Solr.  With this project, though, we need the code to walk before it can run.

Sean 

------------------------------------------------------------------------------

_______________________________________________
Plone-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-developers

[Martin Aspeli]

Sean Upton wrote:

> An eventual goal should be multiple backends for running queries.  My
> immediate goal is small number of dependencies and interfaces that are
> implementation-neutral, so that writing a backend later for
> collective.solr would be possible.

If indeed we need some kind of catalog abstraction, it probably belongs
in another package (like collective.indexing), if indeed it doesn't
exist already. ZCatalog.searchResults() is pretty generic already, of
course. :)

> The challenges for ZCatalog in this area:
>
> (1) most indexing operations run in-band inside the application, which
> leaves few opportunities to offload the memory pressure this creates.  
> There is not much that can be done for this problem that is reasonably
> simple.

Well, collective.indexing tries to address this somewhat from what I
understand.

> (2) there is not yet any means to cache individual filter sets (of
> catalog record ids) for each filter/facet choice (like filterCache does
> in Solr), which leaves few ways to compute set intersections other than
> re-running the queries.  This is not an impossible problem, just
> something that needs some clever work (I hope to do some of that work
> later this summer).

Cool. :)

> I plan to try and investigate what might be possible to improve #2, but
> my default assumption is that really big sites with lots of content
> would choose an out-of-band indexing system like Solr.  With this
> project, though, we need the code to walk before it can run.

That sounds like the right approach to take. I'd say there's a lot that
can be done to make the user experience of large folders and search
results way better than what we have now, without any major
infrastructure changes. Those wins should probably come before anything
more ambitious.

Cheers,
Martin

--
Author of `Professional Plone Development`, a book for developers who
want to work with Plone. See http://martinaspeli.net/plone-book


------------------------------------------------------------------------------
_______________________________________________
Plone-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-developers

[Raphael Ritz]

Sean Upton wrote:
[..]

Hi Sean,

>
> (2) there is not yet any means to cache individual filter sets (of
> catalog record ids) for each filter/facet choice (like filterCache does
> in Solr), which leaves few ways to compute set intersections other than
> re-running the queries.  This is not an impossible problem, just
> something that needs some clever work (I hope to do some of that work
> later this summer).

Just in case you have not seen this yet but maybe

   http://pypi.python.org/pypi/unimr.compositeindex

can also provide some inspiration.

Raphael

>
> I plan to try and investigate what might be possible to improve #2, but
> my default assumption is that really big sites with lots of content
> would choose an out-of-band indexing system like Solr.  With this
> project, though, we need the code to walk before it can run.
>
> Sean
>
>
> ------------------------------------------------------------------------
>
> ------------------------------------------------------------------------------
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Plone-developers mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/plone-developers

------------------------------------------------------------------------------
_______________________________________________
Plone-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-developers

[Sean Upton]

Update for 7/4 for the folder navigation work going on in
plone.app.folderui (my GSoC project):

Most of the work related to basic configuration for facets/filters was
completed this week; the current (throw-away) implementation is a
bunch of transient components that are loaded from hard-coded
configuration of facets in Python code (this is eventually replaced
with persistent configuration in a local utility, along with a GS
profile loading sensible defaults -- sometime in the next few weeks).

Facets (the term I am using for the box/group of related filters) and
filters (links you click on to filter results) are configured in such
a way that filter specifications are enhanced zope.schema vocabulary
terms IFacetSpecification components inherit
zope.schema.interfaces.IVocabularyFactory (facet spec is callable with
context object, you then get back a list of filters as a
SimpleVocabulary iterable object that contains IFilterSpecification
term objects).

Today, I wrote a listings view around this configuration, it does
really very little now other than display the filter links for two
facets I have hard-coded (creator, and modified date), but it
validates that the design of interfaces for the facet configuration
mechanism are generally sound.  This view should evolve rapidly.  One
near-term goal is to implement viewlets for each IFacetSpecification
object so that the listings template does not have responsibility for
rendering each facet.

Over the next week or two, the listings view will be made basically
functional so that listings are displayed and correctly filtered for a
limited number of facets.  I think we should be at a point where I
will be soliciting community feedback on UI and configuration in about
mid-July.  Most work in the next two weeks are in getting the view
working with hard-coded facet configuration, writing tests, polishing
user-interface before moving on to supporting a control panel for
persistent configuration of facets, and dealing with under-the-hood
improvements (such as integrating with plone.folder, scalability work,
any query optimization).

Please feel free to let me know if you have any ideas/questions here.

Thanks,
Sean

On Fri, Jun 26, 2009 at 9:52 PM, Sean Upton<[hidden email]> wrote:

> All:
>
> I have put together some detailed updates on my GSoC project for folder
> navigation user-interface enhancements to support folders with many items
> contained.  The primary focus of this project is faceted navigation for
> folder listings.
>
> http://www.mostscript.com/sean/2009/6/27/folder-ui:-gsoc-project-update-6-27
>
> I have been working bottom-up on some backend pieces for this project, and I
> am now shifting gears to writing views, viewlets, and configuration
> persistence.  My goal is to have a simple cohesive (demonstratable)
> user-interface prototype in the next few weeks.  The next week's work is
> focused on integration testing of a yet-untested "query runner" component I
> wrote using AdvancedQuery, as well as views and viewlets for each facet, and
> some simple configuration mechanisms to provide for what is displayed in the
> faceted UI.
>
> Early code is in svn as plone.app.folderui -- this is not yet very cohesive,
> but I aim for it to be very soon.
>
> Any feedback, questions, or comments: they are very much appreciated please
> feel free to leave them here on the list or on my blog.
>
> Thanks,
> Sean
>

------------------------------------------------------------------------------
_______________________________________________
Plone-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-developers

[Sean Upton]

All:

I have tagged a "YMMV, but you might want to take a look" community
feedback early release (0.01) of my plone.app.folderui (faceted folder
listings) work in Plone svn.

More information is available from my weblog:
http://www.mostscript.com/sean/2009/7/13/plone-app-folderui-update

I will be creating weekly iteration release tags in svn for the next
5-6 weeks, hopefully with marked improvements in capabilities of this
package.

There are definitely rough edges, but please feel free to take a look
if you are so inclined/interested.  Any feedback is appreciated.

Thanks,
Sean

------------------------------------------------------------------------------
Enter the BlackBerry Developer Challenge  
This is your chance to win up to $100,000 in prizes! For a limited time,
vendors submitting new applications to BlackBerry App World(TM) will have
the opportunity to enter the BlackBerry Developer Challenge. See full prize  
details at: http://p.sf.net/sfu/Challenge
_______________________________________________
Plone-developers mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/plone-developers