[Proposal] [GSOC] Wiki Importer Enhancement

[Arun Reddy]

Hi Devs,

As the existing Wiki Importer ( GSOC PROJECT - SANDBOX
<http://svn.xwiki.org/svnroot/xwiki/sandbox/gsoc/importexport/>) have
few set backs in  architecture, i would like to make a proposal to
modify and enhance to make it better.

Here is my proposal :

http://gsoc.myxwiki.org/xwiki/bin/view/Import+Export+other+wikis/Wiki+Importer+Enhancement

- A new Wiki model api ( Wiki, Wiki Page, Wiki Page Revision, Wiki
Attachment ) which adheres to XWiki Standards.

- Separation of Import process, Parsing the import data into a abstract wiki
model will be a component. Clients can use it to make their own custom
implementation. This abstract model is used by the other XWiki component /
client code (Groovy & velocity ) to load the data into XWiki with standard
features and little customization (like Preserving history, default names,
attachments etc.. ).


- Using the existing XWiki model api or xwiki-bridge rather than coming up
with a new one.
 But the issues here is, i dont have all the things which are needed and it
has lot more methods which are of no use in import process. Can i proceed
with a custom wiki model like this. ?


- As wiki importer process large streams of data, would it be wise to design
the API more stream oriented and include few stream oriented methods.?


I would like to have your suggestions on this.


Thanks
-Arun Reddy
_______________________________________________
devs mailing list
[hidden email]
http://lists.xwiki.org/mailman/listinfo/devs

[tmortagne]

On Mon, Oct 12, 2009 at 08:48, Arun Reddy <[hidden email]> wrote:
> Hi Devs,
>
> As the existing Wiki Importer ( GSOC PROJECT - SANDBOX
> <http://svn.xwiki.org/svnroot/xwiki/sandbox/gsoc/importexport/>) have
> few set backs in  architecture, i would like to make a proposal to
> modify and enhance to make it better.

Great !

>
> Here is my proposal :
>
> http://gsoc.myxwiki.org/xwiki/bin/view/Import+Export+other+wikis/Wiki+Importer+Enhancement
>
> - A new Wiki model api ( Wiki, Wiki Page, Wiki Page Revision, Wiki
> Attachment ) which adheres to XWiki Standards.
>
> - Separation of Import process, Parsing the import data into a abstract wiki
> model will be a component. Clients can use it to make their own custom
> implementation. This abstract model is used by the other XWiki component /
> client code (Groovy & velocity ) to load the data into XWiki with standard
> features and little customization (like Preserving history, default names,
> attachments etc.. ).

+1

Always good to separate as much as possible every aspect of the
process in different components.

>
>
> - Using the existing XWiki model api or xwiki-bridge rather than coming up
> with a new one.
>  But the issues here is, i dont have all the things which are needed and it
> has lot more methods which are of no use in import process. Can i proceed
> with a custom wiki model like this. ?

Main problem is that you need a XDOM based model instead of a String
content based model right ? The current model will definitely don't
fit this, too bad the new model is not advanced enough. Also the
current model is too linked to the storage so it will not be easy to
do a clean stream based process.

I would say do you own clean custom model and we will see latter for
refactor with the new model.

>
>
> - As wiki importer process large streams of data, would it be wise to design
> the API more stream oriented and include few stream oriented methods.?

Big +1 for stream oriented, the "normal" importer should be stream
based also IMO. Denis Gervalle is currently working on making the
whole export process and its dependencies stream oriented (See
http://markmail.org/message/vzfdkteguzselouy).

It's definitely the way to go. We can always to a wrapper api which is
using the stream based api if we need a not stream based tools but the
other way around is not possible so better to do now the more generic
one.

>
>
> I would like to have your suggestions on this.
>
>
> Thanks
> -Arun Reddy
> _______________________________________________
> devs mailing list
> [hidden email]
> http://lists.xwiki.org/mailman/listinfo/devs
>

--
Thomas Mortagne
_______________________________________________
devs mailing list
[hidden email]
http://lists.xwiki.org/mailman/listinfo/devs

[tmortagne]

On Mon, Oct 12, 2009 at 11:56, Thomas Mortagne
<[hidden email]> wrote:

> On Mon, Oct 12, 2009 at 08:48, Arun Reddy <[hidden email]> wrote:
>> Hi Devs,
>>
>> As the existing Wiki Importer ( GSOC PROJECT - SANDBOX
>> <http://svn.xwiki.org/svnroot/xwiki/sandbox/gsoc/importexport/>) have
>> few set backs in  architecture, i would like to make a proposal to
>> modify and enhance to make it better.
>
> Great !
>
>>
>> Here is my proposal :
>>
>> http://gsoc.myxwiki.org/xwiki/bin/view/Import+Export+other+wikis/Wiki+Importer+Enhancement
>>
>> - A new Wiki model api ( Wiki, Wiki Page, Wiki Page Revision, Wiki
>> Attachment ) which adheres to XWiki Standards.

Here is some quick review comments:
General:
* this API is not streaming oriented, WikiImporter.import should take
listener and not send a full Wiki for example. This api is nice but as
a wrapper for the streaming api (like SAX and DOM or XWiki rednering
events and XDOM).

* Wiki Importer
** you should add some more informationnal methods to use in the UI or
to make easier to do a generic tool like:
*** String getName() : the pretty name of the importer
*** String getDescriptions(): a description of the importer
*** [NICE TO HAVE] List<Class<?>> getParameters(): the list of
parameters type to provide to the importer, that way you can generate
a UI based on it since parameters totally depends on the importer, you
could have an importer able to import the wiki from an URL for example
using REST/XMLRPC... for the same reason List<InputStream> is not
generic enough IMO. You can look at MacroDescriptor for inspiration,
we could make it more generic as a list of parameters instead of a
list of macro parameters to use it in wiki importer, authenticators,
etc. I see this tool as a generic way to get wiki content from any
kind of external source, we could even use it to copy a XWiki wiki
without having to do an export/import by hand for example.

>>
>> - Separation of Import process, Parsing the import data into a abstract wiki
>> model will be a component. Clients can use it to make their own custom
>> implementation. This abstract model is used by the other XWiki component /
>> client code (Groovy & velocity ) to load the data into XWiki with standard
>> features and little customization (like Preserving history, default names,
>> attachments etc.. ).
>
> +1
>
> Always good to separate as much as possible every aspect of the
> process in different components.
>
>>
>>
>> - Using the existing XWiki model api or xwiki-bridge rather than coming up
>> with a new one.
>>  But the issues here is, i dont have all the things which are needed and it
>> has lot more methods which are of no use in import process. Can i proceed
>> with a custom wiki model like this. ?
>
> Main problem is that you need a XDOM based model instead of a String
> content based model right ? The current model will definitely don't
> fit this, too bad the new model is not advanced enough. Also the
> current model is too linked to the storage so it will not be easy to
> do a clean stream based process.
>
> I would say do you own clean custom model and we will see latter for
> refactor with the new model.
>
>>
>>
>> - As wiki importer process large streams of data, would it be wise to design
>> the API more stream oriented and include few stream oriented methods.?
>
> Big +1 for stream oriented, the "normal" importer should be stream
> based also IMO. Denis Gervalle is currently working on making the
> whole export process and its dependencies stream oriented (See
> http://markmail.org/message/vzfdkteguzselouy).
>
> It's definitely the way to go. We can always to a wrapper api which is
> using the stream based api if we need a not stream based tools but the
> other way around is not possible so better to do now the more generic
> one.
>
>>
>>
>> I would like to have your suggestions on this.
>>
>>
>> Thanks
>> -Arun Reddy
>> _______________________________________________
>> devs mailing list
>> [hidden email]
>> http://lists.xwiki.org/mailman/listinfo/devs
>>
>
>
>
> --
> Thomas Mortagne
>

--
Thomas Mortagne
_______________________________________________
devs mailing list
[hidden email]
http://lists.xwiki.org/mailman/listinfo/devs

[Asiri Rathnayake]

Hi,


Here is some quick review comments:
> General:
> * this API is not streaming oriented, WikiImporter.import should take
> listener and not send a full Wiki for example. This api is nice but as
> a wrapper for the streaming api (like SAX and DOM or XWiki rednering
> events and XDOM).
>

I kind of had a different idea about the streaming api. I don't know why I
thought this but what I had in mind was something like:

Wiki::hasNextPage():boolean

Wiki::getNextPage():WikiPage

This is kind of like reading a result set (pull-parsing???)

I thought this kind of a api would be more natural but I have no idea if
this is even possible to implement for all the wiki export formats out
there.

If I understood you correct, what you have suggested is something like:

WikiImporterListener::onWikiPage(WikiPage)

WikiImporter::import(......, WikiImporterListener)

This might be viewed as a "push-parsing" mechanism?? Anyway, I'll be glad to
know what are the pros & cons of these two approaches.

> *** [NICE TO HAVE] List<Class<?>> getParameters(): the list of
> parameters type to provide to the importer, that way you can generate
> a UI based on it since parameters totally depends on the importer, you
> could have an importer able to import the wiki from an URL for example
> using REST/XMLRPC... for the same reason List<InputStream> is not
> generic enough IMO. You can look at MacroDescriptor for inspiration,
> we could make it more generic as a list of parameters instead of a
> list of macro parameters to use it in wiki importer, authenticators,
> etc. I see this tool as a generic way to get wiki content from any
> kind of external source, we could even use it to copy a XWiki wiki
> without having to do an export/import by hand for example.
>

+1

This will make it possible to define different types of importers with
different requirements (rather than restricting them to List<InputStream>)

Thanks.

- Asiri
_______________________________________________
devs mailing list
[hidden email]
http://lists.xwiki.org/mailman/listinfo/devs

[tmortagne]

On Mon, Oct 12, 2009 at 17:57, Asiri Rathnayake
<[hidden email]> wrote:

> Hi,
>
>
> Here is some quick review comments:
>> General:
>> * this API is not streaming oriented, WikiImporter.import should take
>> listener and not send a full Wiki for example. This api is nice but as
>> a wrapper for the streaming api (like SAX and DOM or XWiki rednering
>> events and XDOM).
>>
>
> I kind of had a different idea about the streaming api. I don't know why I
> thought this but what I had in mind was something like:
>
> Wiki::hasNextPage():boolean
>
> Wiki::getNextPage():WikiPage
>
> This is kind of like reading a result set (pull-parsing???)
>
> I thought this kind of a api would be more natural but I have no idea if
> this is even possible to implement for all the wiki export formats out
> there.
>
> If I understood you correct, what you have suggested is something like:
>
> WikiImporterListener::onWikiPage(WikiPage)

Not sure yet for the details but that's the idea yes.

>
> WikiImporter::import(......, WikiImporterListener)
>
> This might be viewed as a "push-parsing" mechanism?? Anyway, I'll be glad to
> know what are the pros & cons of these two approaches.

Actually in my mind there was more than just this method, i see at least also

WikiImporterListener::onWikiPageAttachment(AttachmentName, InputStream)

since having the whole WikiPage with all its attachments in memory
make the import not strong enough if you have too big attachments.
Making a streaming API is about making every memory consuming part
streamed (Actually to be sure maybe also have
WikiImporterListener::onWikiPageObject(WikiPageObject) but since XWiki
does not support this well internally currently it's less important
than attachments).

The getNextPage() API is good when you have only one simple kind of
data in a plane list but the events based way is more generic and
easier to implements for the importer writer IMO since it send the
events when he wants.

>
>
>> *** [NICE TO HAVE] List<Class<?>> getParameters(): the list of
>> parameters type to provide to the importer, that way you can generate
>> a UI based on it since parameters totally depends on the importer, you
>> could have an importer able to import the wiki from an URL for example
>> using REST/XMLRPC... for the same reason List<InputStream> is not
>> generic enough IMO. You can look at MacroDescriptor for inspiration,
>> we could make it more generic as a list of parameters instead of a
>> list of macro parameters to use it in wiki importer, authenticators,
>> etc. I see this tool as a generic way to get wiki content from any
>> kind of external source, we could even use it to copy a XWiki wiki
>> without having to do an export/import by hand for example.
>>
>
> +1
>
> This will make it possible to define different types of importers with
> different requirements (rather than restricting them to List<InputStream>)
>
> Thanks.
>
> - Asiri
> _______________________________________________
> devs mailing list
> [hidden email]
> http://lists.xwiki.org/mailman/listinfo/devs
>

--
Thomas Mortagne
_______________________________________________
devs mailing list
[hidden email]
http://lists.xwiki.org/mailman/listinfo/devs

[Asiri Rathnayake]

Hi,

Actually in my mind there was more than just this method, i see at least

> also
>
> WikiImporterListener::onWikiPageAttachment(AttachmentName, InputStream)
>
> since having the whole WikiPage with all its attachments in memory
> make the import not strong enough if you have too big attachments.
> Making a streaming API is about making every memory consuming part
> streamed (Actually to be sure maybe also have
> WikiImporterListener::onWikiPageObject(WikiPageObject) but since XWiki
> does not support this well internally currently it's less important
> than attachments).
>
> The getNextPage() API is good when you have only one simple kind of
> data in a plane list but the events based way is more generic and
> easier to implements for the importer writer IMO since it send the
> events when he wants.
>

Ok, Now I understand that importer writers work would be easy if we go with
an event based API.

So for Arun this means,

1. The model you proposed is ok

2. You need to refine the WikiImporter API so that it takes into account
what we discussed here. This means:

i) Have a mechanism for defining WikiImporter components with different
requirements (look at the getParameters() approach suggested by Thomas)

ii) Make the WikiImporter api mode stream oriented. I think you can start
with onWikiPage() & onAttachment() events for now.

I think you should start working on a revised proposal with this information
taken into account. In the mean time other developers will comment on this
proposal if they have further suggestions.

- Asiri

> >
> >
> >> *** [NICE TO HAVE] List<Class<?>> getParameters(): the list of
> >> parameters type to provide to the importer, that way you can generate
> >> a UI based on it since parameters totally depends on the importer, you
> >> could have an importer able to import the wiki from an URL for example
> >> using REST/XMLRPC... for the same reason List<InputStream> is not
> >> generic enough IMO. You can look at MacroDescriptor for inspiration,
> >> we could make it more generic as a list of parameters instead of a
> >> list of macro parameters to use it in wiki importer, authenticators,
> >> etc. I see this tool as a generic way to get wiki content from any
> >> kind of external source, we could even use it to copy a XWiki wiki
> >> without having to do an export/import by hand for example.
> >>
> >
> > +1
> >
> > This will make it possible to define different types of importers with
> > different requirements (rather than restricting them to
> List<InputStream>)
> >
> > Thanks.
> >
> > - Asiri
> > _______________________________________________
> > devs mailing list
> > [hidden email]
> > http://lists.xwiki.org/mailman/listinfo/devs
> >
>
>
>
> --
> Thomas Mortagne
> _______________________________________________
> devs mailing list
> [hidden email]
> http://lists.xwiki.org/mailman/listinfo/devs
>

_______________________________________________
devs mailing list
[hidden email]
http://lists.xwiki.org/mailman/listinfo/devs

[Arun Reddy]

Hi,

On Mon, Oct 12, 2009 at 10:27 PM, Asiri Rathnayake <
[hidden email]> wrote:

> Hi,
>
> Actually in my mind there was more than just this method, i see at least
> > also
> >
> > WikiImporterListener::onWikiPageAttachment(AttachmentName, InputStream)
> >
> > since having the whole WikiPage with all its attachments in memory
> > make the import not strong enough if you have too big attachments.
> > Making a streaming API is about making every memory consuming part
> > streamed (Actually to be sure maybe also have
> > WikiImporterListener::onWikiPageObject(WikiPageObject) but since XWiki
> > does not support this well internally currently it's less important
> > than attachments).
> >
> > The getNextPage() API is good when you have only one simple kind of
> > data in a plane list but the events based way is more generic and
> > easier to implements for the importer writer IMO since it send the
> > events when he wants.
> >
>
> Ok, Now I understand that importer writers work would be easy if we go with
> an event based API.
>
> So for Arun this means,
>
> 1. The model you proposed is ok
>
> 2. You need to refine the WikiImporter API so that it takes into account
> what we discussed here. This means:
>
> i) Have a mechanism for defining WikiImporter components with different
> requirements (look at the getParameters() approach suggested by Thomas)
>
> ii) Make the WikiImporter api mode stream oriented. I think you can start
> with onWikiPage() & onAttachment() events for now.
>
>

- Can we have beginWikiPage() followed by endWikiPage()
beginAttachment() and endAttachment() rather than having a    single
onWikiPage(),onAttachment() in the WikiImporterListener.

-How to handle revisions - Can we have methods to read those tags too. one
for WikiPageRevision and other for AttachmentRevision.

> I think you should start working on a revised proposal with this
> information
> taken into account. In the mean time other developers will comment on this
> proposal if they have further suggestions.
>
> - Asiri
>
>
> > >
> > >
> > >> *** [NICE TO HAVE] List<Class<?>> getParameters(): the list of
> > >> parameters type to provide to the importer, that way you can generate
> > >> a UI based on it since parameters totally depends on the importer, you
> > >> could have an importer able to import the wiki from an URL for example
> > >> using REST/XMLRPC... for the same reason List<InputStream> is not
> > >> generic enough IMO. You can look at MacroDescriptor for inspiration,
> > >> we could make it more generic as a list of parameters instead of a
> > >> list of macro parameters to use it in wiki importer, authenticators,
> > >> etc. I see this tool as a generic way to get wiki content from any
> > >> kind of external source, we could even use it to copy a XWiki wiki
> > >> without having to do an export/import by hand for example.
> > >>
> > >
> > > +1
> > >
> > > This will make it possible to define different types of importers with
> > > different requirements (rather than restricting them to
> > List<InputStream>)
> > >
> > > Thanks.
> > >
> > > - Asiri
> > > _______________________________________________
> > > devs mailing list
> > > [hidden email]
> > > http://lists.xwiki.org/mailman/listinfo/devs
> > >
> >
> >
> >
> > --
> > Thomas Mortagne
> > _______________________________________________
> > devs mailing list
> > [hidden email]
> > http://lists.xwiki.org/mailman/listinfo/devs
> >
> _______________________________________________
> devs mailing list
> [hidden email]
> http://lists.xwiki.org/mailman/listinfo/devs
>

--
Best Regards,
Arun Reddy
_______________________________________________
devs mailing list
[hidden email]
http://lists.xwiki.org/mailman/listinfo/devs

[tmortagne]

On Wed, Oct 14, 2009 at 17:03, Arun Reddy <[hidden email]> wrote:

> Hi,
>
> On Mon, Oct 12, 2009 at 10:27 PM, Asiri Rathnayake <
> [hidden email]> wrote:
>
>> Hi,
>>
>> Actually in my mind there was more than just this method, i see at least
>> > also
>> >
>> > WikiImporterListener::onWikiPageAttachment(AttachmentName, InputStream)
>> >
>> > since having the whole WikiPage with all its attachments in memory
>> > make the import not strong enough if you have too big attachments.
>> > Making a streaming API is about making every memory consuming part
>> > streamed (Actually to be sure maybe also have
>> > WikiImporterListener::onWikiPageObject(WikiPageObject) but since XWiki
>> > does not support this well internally currently it's less important
>> > than attachments).
>> >
>> > The getNextPage() API is good when you have only one simple kind of
>> > data in a plane list but the events based way is more generic and
>> > easier to implements for the importer writer IMO since it send the
>> > events when he wants.
>> >
>>
>> Ok, Now I understand that importer writers work would be easy if we go with
>> an event based API.
>>
>> So for Arun this means,
>>
>> 1. The model you proposed is ok
>>
>> 2. You need to refine the WikiImporter API so that it takes into account
>> what we discussed here. This means:
>>
>> i) Have a mechanism for defining WikiImporter components with different
>> requirements (look at the getParameters() approach suggested by Thomas)
>>
>> ii) Make the WikiImporter api mode stream oriented. I think you can start
>> with onWikiPage() & onAttachment() events for now.
>>
>>
> - Can we have beginWikiPage() followed by endWikiPage()
> beginAttachment() and endAttachment() rather than having a    single
> onWikiPage(),onAttachment() in the WikiImporterListener.

If you plan to have something like

beginWikiPage(name)
  beginObject(type)
    onProperty(name, value, ...)
  endObject()
  beginObject()
    onProperty()
  endObject()
  beginAttachment(name)
  endAttachment()
endWikiPage()

+1 it's a good idea, now i'm not sure why you need begin and end for
an attachment, what other events would you have inside it ?

>
> -How to handle revisions - Can we have methods to read those tags too. one
> for WikiPageRevision and other for AttachmentRevision.
>
>
>> I think you should start working on a revised proposal with this
>> information
>> taken into account. In the mean time other developers will comment on this
>> proposal if they have further suggestions.
>>
>> - Asiri
>>
>>
>> > >
>> > >
>> > >> *** [NICE TO HAVE] List<Class<?>> getParameters(): the list of
>> > >> parameters type to provide to the importer, that way you can generate
>> > >> a UI based on it since parameters totally depends on the importer, you
>> > >> could have an importer able to import the wiki from an URL for example
>> > >> using REST/XMLRPC... for the same reason List<InputStream> is not
>> > >> generic enough IMO. You can look at MacroDescriptor for inspiration,
>> > >> we could make it more generic as a list of parameters instead of a
>> > >> list of macro parameters to use it in wiki importer, authenticators,
>> > >> etc. I see this tool as a generic way to get wiki content from any
>> > >> kind of external source, we could even use it to copy a XWiki wiki
>> > >> without having to do an export/import by hand for example.
>> > >>
>> > >
>> > > +1
>> > >
>> > > This will make it possible to define different types of importers with
>> > > different requirements (rather than restricting them to
>> > List<InputStream>)
>> > >
>> > > Thanks.
>> > >
>> > > - Asiri
>> > > _______________________________________________
>> > > devs mailing list
>> > > [hidden email]
>> > > http://lists.xwiki.org/mailman/listinfo/devs
>> > >
>> >
>> >
>> >
>> > --
>> > Thomas Mortagne
>> > _______________________________________________
>> > devs mailing list
>> > [hidden email]
>> > http://lists.xwiki.org/mailman/listinfo/devs
>> >
>> _______________________________________________
>> devs mailing list
>> [hidden email]
>> http://lists.xwiki.org/mailman/listinfo/devs
>>
>
>
>
> --
> Best Regards,
> Arun Reddy
> _______________________________________________
> devs mailing list
> [hidden email]
> http://lists.xwiki.org/mailman/listinfo/devs
>

--
Thomas Mortagne
_______________________________________________
devs mailing list
[hidden email]
http://lists.xwiki.org/mailman/listinfo/devs

[Arun Reddy]

On Wed, Oct 14, 2009 at 8:41 PM, Thomas Mortagne
<[hidden email]>wrote:

> On Wed, Oct 14, 2009 at 17:03, Arun Reddy <[hidden email]> wrote:
> > Hi,
> >
> > On Mon, Oct 12, 2009 at 10:27 PM, Asiri Rathnayake <
> > [hidden email]> wrote:
> >
> >> Hi,
> >>
> >> Actually in my mind there was more than just this method, i see at least
> >> > also
> >> >
> >> > WikiImporterListener::onWikiPageAttachment(AttachmentName,
> InputStream)
> >> >
> >> > since having the whole WikiPage with all its attachments in memory
> >> > make the import not strong enough if you have too big attachments.
> >> > Making a streaming API is about making every memory consuming part
> >> > streamed (Actually to be sure maybe also have
> >> > WikiImporterListener::onWikiPageObject(WikiPageObject) but since XWiki
> >> > does not support this well internally currently it's less important
> >> > than attachments).
> >> >
> >> > The getNextPage() API is good when you have only one simple kind of
> >> > data in a plane list but the events based way is more generic and
> >> > easier to implements for the importer writer IMO since it send the
> >> > events when he wants.
> >> >
> >>
> >> Ok, Now I understand that importer writers work would be easy if we go
> with
> >> an event based API.
> >>
> >> So for Arun this means,
> >>
> >> 1. The model you proposed is ok
> >>
> >> 2. You need to refine the WikiImporter API so that it takes into account
> >> what we discussed here. This means:
> >>
> >> i) Have a mechanism for defining WikiImporter components with different
> >> requirements (look at the getParameters() approach suggested by Thomas)
> >>
> >> ii) Make the WikiImporter api mode stream oriented. I think you can
> start
> >> with onWikiPage() & onAttachment() events for now.
> >>
> >>
> > - Can we have beginWikiPage() followed by endWikiPage()
> > beginAttachment() and endAttachment() rather than having a    single
> > onWikiPage(),onAttachment() in the WikiImporterListener.
>
> If you plan to have something like
>
> beginWikiPage(name)
>  beginObject(type)
>    onProperty(name, value, ...)
>  endObject()
>  beginObject()
>    onProperty()
>  endObject()
>  beginAttachment(name)
>  endAttachment()
> endWikiPage()
>
> +1 it's a good idea, now i'm not sure why you need begin and end for
> an attachment, what other events would you have inside it ?
>
>

Because even it has few properties and revisions.
Kindly correct me, if am wrong.

>  >
> > -How to handle revisions - Can we have methods to read those tags too.
> one
> > for WikiPageRevision and other for AttachmentRevision.
> >
> >
> >> I think you should start working on a revised proposal with this
> >> information
> >> taken into account. In the mean time other developers will comment on
> this
> >> proposal if they have further suggestions.
> >>
> >> - Asiri
> >>
> >>
> >> > >
> >> > >
> >> > >> *** [NICE TO HAVE] List<Class<?>> getParameters(): the list of
> >> > >> parameters type to provide to the importer, that way you can
> generate
> >> > >> a UI based on it since parameters totally depends on the importer,
> you
> >> > >> could have an importer able to import the wiki from an URL for
> example
> >> > >> using REST/XMLRPC... for the same reason List<InputStream> is not
> >> > >> generic enough IMO. You can look at MacroDescriptor for
> inspiration,
> >> > >> we could make it more generic as a list of parameters instead of a
> >> > >> list of macro parameters to use it in wiki importer,
> authenticators,
> >> > >> etc. I see this tool as a generic way to get wiki content from any
> >> > >> kind of external source, we could even use it to copy a XWiki wiki
> >> > >> without having to do an export/import by hand for example.
> >> > >>
> >> > >
> >> > > +1
> >> > >
> >> > > This will make it possible to define different types of importers
> with
> >> > > different requirements (rather than restricting them to
> >> > List<InputStream>)
> >> > >
> >> > > Thanks.
> >> > >
> >> > > - Asiri
> >> > > _______________________________________________
> >> > > devs mailing list
> >> > > [hidden email]
> >> > > http://lists.xwiki.org/mailman/listinfo/devs
> >> > >
> >> >
> >> >
> >> >
> >> > --
> >> > Thomas Mortagne
> >> > _______________________________________________
> >> > devs mailing list
> >> > [hidden email]
> >> > http://lists.xwiki.org/mailman/listinfo/devs
> >> >
> >> _______________________________________________
> >> devs mailing list
> >> [hidden email]
> >> http://lists.xwiki.org/mailman/listinfo/devs
> >>
> >
> >
> >
> > --
> > Best Regards,
> > Arun Reddy
> > _______________________________________________
> > devs mailing list
> > [hidden email]
> > http://lists.xwiki.org/mailman/listinfo/devs
> >
>
>
>
> --
> Thomas Mortagne
> _______________________________________________
> devs mailing list
> [hidden email]
> http://lists.xwiki.org/mailman/listinfo/devs
>

--
Best Regards,
Arun Reddy
_______________________________________________
devs mailing list
[hidden email]
http://lists.xwiki.org/mailman/listinfo/devs

[tmortagne]

On Wed, Oct 14, 2009 at 17:11, Thomas Mortagne
<[hidden email]> wrote:

> On Wed, Oct 14, 2009 at 17:03, Arun Reddy <[hidden email]> wrote:
>> Hi,
>>
>> On Mon, Oct 12, 2009 at 10:27 PM, Asiri Rathnayake <
>> [hidden email]> wrote:
>>
>>> Hi,
>>>
>>> Actually in my mind there was more than just this method, i see at least
>>> > also
>>> >
>>> > WikiImporterListener::onWikiPageAttachment(AttachmentName, InputStream)
>>> >
>>> > since having the whole WikiPage with all its attachments in memory
>>> > make the import not strong enough if you have too big attachments.
>>> > Making a streaming API is about making every memory consuming part
>>> > streamed (Actually to be sure maybe also have
>>> > WikiImporterListener::onWikiPageObject(WikiPageObject) but since XWiki
>>> > does not support this well internally currently it's less important
>>> > than attachments).
>>> >
>>> > The getNextPage() API is good when you have only one simple kind of
>>> > data in a plane list but the events based way is more generic and
>>> > easier to implements for the importer writer IMO since it send the
>>> > events when he wants.
>>> >
>>>
>>> Ok, Now I understand that importer writers work would be easy if we go with
>>> an event based API.
>>>
>>> So for Arun this means,
>>>
>>> 1. The model you proposed is ok
>>>
>>> 2. You need to refine the WikiImporter API so that it takes into account
>>> what we discussed here. This means:
>>>
>>> i) Have a mechanism for defining WikiImporter components with different
>>> requirements (look at the getParameters() approach suggested by Thomas)
>>>
>>> ii) Make the WikiImporter api mode stream oriented. I think you can start
>>> with onWikiPage() & onAttachment() events for now.
>>>
>>>
>> - Can we have beginWikiPage() followed by endWikiPage()
>> beginAttachment() and endAttachment() rather than having a    single
>> onWikiPage(),onAttachment() in the WikiImporterListener.
>
> If you plan to have something like
>
> beginWikiPage(name)
>  beginObject(type)
>    onProperty(name, value, ...)
>  endObject()
>  beginObject()
>    onProperty()
>  endObject()
>  beginAttachment(name)
>  endAttachment()
> endWikiPage()
>
> +1 it's a good idea, now i'm not sure why you need begin and end for
> an attachment, what other events would you have inside it ?
>
>>
>> -How to handle revisions - Can we have methods to read those tags too. one
>> for WikiPageRevision and other for AttachmentRevision.

Maybe something like:

beginWikiPage()
 beginWikiPageRevision()
  beginObject(type)
   onProperty()
  endObject()
  beginObject()
   onProperty()
  endObject()
  beginClass()
   onProperty()
  endClass()
  beginAttachment()
    onAttachmentRevision()
   endAttachment()
 endWikiPage()
 beginWikiPageRevision()
  beginObject(type)
   onProperty()
  endObject()
  beginObject()
   onProperty()
  endObject()
  onAttachmentRevision() (since i think you can't have several
attachement revision for the same wiki page revision)
 endWikiPage()
endWikiPage()

basically you can look at XWiki xml ecport format and map something
similar for events I think (not the exact same thing the export format
need improvements).

>>
>>
>>> I think you should start working on a revised proposal with this
>>> information
>>> taken into account. In the mean time other developers will comment on this
>>> proposal if they have further suggestions.
>>>
>>> - Asiri
>>>
>>>
>>> > >
>>> > >
>>> > >> *** [NICE TO HAVE] List<Class<?>> getParameters(): the list of
>>> > >> parameters type to provide to the importer, that way you can generate
>>> > >> a UI based on it since parameters totally depends on the importer, you
>>> > >> could have an importer able to import the wiki from an URL for example
>>> > >> using REST/XMLRPC... for the same reason List<InputStream> is not
>>> > >> generic enough IMO. You can look at MacroDescriptor for inspiration,
>>> > >> we could make it more generic as a list of parameters instead of a
>>> > >> list of macro parameters to use it in wiki importer, authenticators,
>>> > >> etc. I see this tool as a generic way to get wiki content from any
>>> > >> kind of external source, we could even use it to copy a XWiki wiki
>>> > >> without having to do an export/import by hand for example.
>>> > >>
>>> > >
>>> > > +1
>>> > >
>>> > > This will make it possible to define different types of importers with
>>> > > different requirements (rather than restricting them to
>>> > List<InputStream>)
>>> > >
>>> > > Thanks.
>>> > >
>>> > > - Asiri
>>> > > _______________________________________________
>>> > > devs mailing list
>>> > > [hidden email]
>>> > > http://lists.xwiki.org/mailman/listinfo/devs
>>> > >
>>> >
>>> >
>>> >
>>> > --
>>> > Thomas Mortagne
>>> > _______________________________________________
>>> > devs mailing list
>>> > [hidden email]
>>> > http://lists.xwiki.org/mailman/listinfo/devs
>>> >
>>> _______________________________________________
>>> devs mailing list
>>> [hidden email]
>>> http://lists.xwiki.org/mailman/listinfo/devs
>>>
>>
>>
>>
>> --
>> Best Regards,
>> Arun Reddy
>> _______________________________________________
>> devs mailing list
>> [hidden email]
>> http://lists.xwiki.org/mailman/listinfo/devs
>>
>
>
>
> --
> Thomas Mortagne
>

--
Thomas Mortagne
_______________________________________________
devs mailing list
[hidden email]
http://lists.xwiki.org/mailman/listinfo/devs

[tmortagne]

On Wed, Oct 14, 2009 at 17:22, Arun Reddy <[hidden email]> wrote:

> On Wed, Oct 14, 2009 at 8:41 PM, Thomas Mortagne
> <[hidden email]>wrote:
>
>> On Wed, Oct 14, 2009 at 17:03, Arun Reddy <[hidden email]> wrote:
>> > Hi,
>> >
>> > On Mon, Oct 12, 2009 at 10:27 PM, Asiri Rathnayake <
>> > [hidden email]> wrote:
>> >
>> >> Hi,
>> >>
>> >> Actually in my mind there was more than just this method, i see at least
>> >> > also
>> >> >
>> >> > WikiImporterListener::onWikiPageAttachment(AttachmentName,
>> InputStream)
>> >> >
>> >> > since having the whole WikiPage with all its attachments in memory
>> >> > make the import not strong enough if you have too big attachments.
>> >> > Making a streaming API is about making every memory consuming part
>> >> > streamed (Actually to be sure maybe also have
>> >> > WikiImporterListener::onWikiPageObject(WikiPageObject) but since XWiki
>> >> > does not support this well internally currently it's less important
>> >> > than attachments).
>> >> >
>> >> > The getNextPage() API is good when you have only one simple kind of
>> >> > data in a plane list but the events based way is more generic and
>> >> > easier to implements for the importer writer IMO since it send the
>> >> > events when he wants.
>> >> >
>> >>
>> >> Ok, Now I understand that importer writers work would be easy if we go
>> with
>> >> an event based API.
>> >>
>> >> So for Arun this means,
>> >>
>> >> 1. The model you proposed is ok
>> >>
>> >> 2. You need to refine the WikiImporter API so that it takes into account
>> >> what we discussed here. This means:
>> >>
>> >> i) Have a mechanism for defining WikiImporter components with different
>> >> requirements (look at the getParameters() approach suggested by Thomas)
>> >>
>> >> ii) Make the WikiImporter api mode stream oriented. I think you can
>> start
>> >> with onWikiPage() & onAttachment() events for now.
>> >>
>> >>
>> > - Can we have beginWikiPage() followed by endWikiPage()
>> > beginAttachment() and endAttachment() rather than having a    single
>> > onWikiPage(),onAttachment() in the WikiImporterListener.
>>
>> If you plan to have something like
>>
>> beginWikiPage(name)
>>  beginObject(type)
>>    onProperty(name, value, ...)
>>  endObject()
>>  beginObject()
>>    onProperty()
>>  endObject()
>>  beginAttachment(name)
>>  endAttachment()
>> endWikiPage()
>>
>> +1 it's a good idea, now i'm not sure why you need begin and end for
>> an attachment, what other events would you have inside it ?
>>
>>
> Because even it has few properties and revisions.
> Kindly correct me, if am wrong.

You don't need an event for each property, event are more for
variables things, you can have onAttachment(name, version, content,
etc.). For revisions i already anwser in the previous mail, if you put
the onAttachement inside a begin/endWikiPageRevision you will only
have one version of the attachement for a version of wiki page.

>
>>  >
>> > -How to handle revisions - Can we have methods to read those tags too.
>> one
>> > for WikiPageRevision and other for AttachmentRevision.
>> >
>> >
>> >> I think you should start working on a revised proposal with this
>> >> information
>> >> taken into account. In the mean time other developers will comment on
>> this
>> >> proposal if they have further suggestions.
>> >>
>> >> - Asiri
>> >>
>> >>
>> >> > >
>> >> > >
>> >> > >> *** [NICE TO HAVE] List<Class<?>> getParameters(): the list of
>> >> > >> parameters type to provide to the importer, that way you can
>> generate
>> >> > >> a UI based on it since parameters totally depends on the importer,
>> you
>> >> > >> could have an importer able to import the wiki from an URL for
>> example
>> >> > >> using REST/XMLRPC... for the same reason List<InputStream> is not
>> >> > >> generic enough IMO. You can look at MacroDescriptor for
>> inspiration,
>> >> > >> we could make it more generic as a list of parameters instead of a
>> >> > >> list of macro parameters to use it in wiki importer,
>> authenticators,
>> >> > >> etc. I see this tool as a generic way to get wiki content from any
>> >> > >> kind of external source, we could even use it to copy a XWiki wiki
>> >> > >> without having to do an export/import by hand for example.
>> >> > >>
>> >> > >
>> >> > > +1
>> >> > >
>> >> > > This will make it possible to define different types of importers
>> with
>> >> > > different requirements (rather than restricting them to
>> >> > List<InputStream>)
>> >> > >
>> >> > > Thanks.
>> >> > >
>> >> > > - Asiri
>> >> > > _______________________________________________
>> >> > > devs mailing list
>> >> > > [hidden email]
>> >> > > http://lists.xwiki.org/mailman/listinfo/devs
>> >> > >
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> > Thomas Mortagne
>> >> > _______________________________________________
>> >> > devs mailing list
>> >> > [hidden email]
>> >> > http://lists.xwiki.org/mailman/listinfo/devs
>> >> >
>> >> _______________________________________________
>> >> devs mailing list
>> >> [hidden email]
>> >> http://lists.xwiki.org/mailman/listinfo/devs
>> >>
>> >
>> >
>> >
>> > --
>> > Best Regards,
>> > Arun Reddy
>> > _______________________________________________
>> > devs mailing list
>> > [hidden email]
>> > http://lists.xwiki.org/mailman/listinfo/devs
>> >
>>
>>
>>
>> --
>> Thomas Mortagne
>> _______________________________________________
>> devs mailing list
>> [hidden email]
>> http://lists.xwiki.org/mailman/listinfo/devs
>>
>
>
>
> --
> Best Regards,
> Arun Reddy
> _______________________________________________
> devs mailing list
> [hidden email]
> http://lists.xwiki.org/mailman/listinfo/devs
>

--
Thomas Mortagne
_______________________________________________
devs mailing list
[hidden email]
http://lists.xwiki.org/mailman/listinfo/devs

[Asiri Rathnayake]

Hi,

>> -How to handle revisions - Can we have methods to read those tags too.
> one
> >> for WikiPageRevision and other for AttachmentRevision.
>
> Maybe something like:
>
> beginWikiPage()
>  beginWikiPageRevision()
>  beginObject(type)
>   onProperty()
>  endObject()
>  beginObject()
>   onProperty()
>  endObject()
>  beginClass()
>   onProperty()
>  endClass()
>  beginAttachment()
>    onAttachmentRevision()
>   endAttachment()
>  endWikiPage()
>  beginWikiPageRevision()
>  beginObject(type)
>   onProperty()
>  endObject()
>  beginObject()
>   onProperty()
>  endObject()
>  onAttachmentRevision() (since i think you can't have several
> attachement revision for the same wiki page revision)
>  endWikiPage()
> endWikiPage()
>

+1 for something like this.

Now I don't think that other wiki formats (mediawiki, confluence) will
normally fire events like beginObject() / endObject() but it's important to
have the full span of XWiki's capabilities embodied into this API. Because
then we can,

1) Use this generic importer even when importing xwiki xml exports,

2) We'll be able to map different concepts of other wiki exports into xwiki
concepts. An example is mapping [category] links of mediawiki into xwiki
tags.

Thanks.

- Asiri

> basically you can look at XWiki xml ecport format and map something
> similar for events I think (not the exact same thing the export format
> need improvements).
>
> >>
> >>
> >>> I think you should start working on a revised proposal with this
> >>> information
> >>> taken into account. In the mean time other developers will comment on
> this
> >>> proposal if they have further suggestions.
> >>>
> >>> - Asiri
> >>>
> >>>
> >>> > >
> >>> > >
> >>> > >> *** [NICE TO HAVE] List<Class<?>> getParameters(): the list of
> >>> > >> parameters type to provide to the importer, that way you can
> generate
> >>> > >> a UI based on it since parameters totally depends on the importer,
> you
> >>> > >> could have an importer able to import the wiki from an URL for
> example
> >>> > >> using REST/XMLRPC... for the same reason List<InputStream> is not
> >>> > >> generic enough IMO. You can look at MacroDescriptor for
> inspiration,
> >>> > >> we could make it more generic as a list of parameters instead of a
> >>> > >> list of macro parameters to use it in wiki importer,
> authenticators,
> >>> > >> etc. I see this tool as a generic way to get wiki content from any
> >>> > >> kind of external source, we could even use it to copy a XWiki wiki
> >>> > >> without having to do an export/import by hand for example.
> >>> > >>
> >>> > >
> >>> > > +1
> >>> > >
> >>> > > This will make it possible to define different types of importers
> with
> >>> > > different requirements (rather than restricting them to
> >>> > List<InputStream>)
> >>> > >
> >>> > > Thanks.
> >>> > >
> >>> > > - Asiri
> >>> > > _______________________________________________
> >>> > > devs mailing list
> >>> > > [hidden email]
> >>> > > http://lists.xwiki.org/mailman/listinfo/devs
> >>> > >
> >>> >
> >>> >
> >>> >
> >>> > --
> >>> > Thomas Mortagne
> >>> > _______________________________________________
> >>> > devs mailing list
> >>> > [hidden email]
> >>> > http://lists.xwiki.org/mailman/listinfo/devs
> >>> >
> >>> _______________________________________________
> >>> devs mailing list
> >>> [hidden email]
> >>> http://lists.xwiki.org/mailman/listinfo/devs
> >>>
> >>
> >>
> >>
> >> --
> >> Best Regards,
> >> Arun Reddy
> >> _______________________________________________
> >> devs mailing list
> >> [hidden email]
> >> http://lists.xwiki.org/mailman/listinfo/devs
> >>
> >
> >
> >
> > --
> > Thomas Mortagne
> >
>
>
>
> --
> Thomas Mortagne
> _______________________________________________
> devs mailing list
> [hidden email]
> http://lists.xwiki.org/mailman/listinfo/devs
>

_______________________________________________
devs mailing list
[hidden email]
http://lists.xwiki.org/mailman/listinfo/devs

[tmortagne]

On Wed, Oct 14, 2009 at 20:49, Asiri Rathnayake
<[hidden email]> wrote:

> Hi,
>
>
>>> -How to handle revisions - Can we have methods to read those tags too.
>> one
>> >> for WikiPageRevision and other for AttachmentRevision.
>>
>> Maybe something like:
>>
>> beginWikiPage()
>>  beginWikiPageRevision()
>>  beginObject(type)
>>   onProperty()
>>  endObject()
>>  beginObject()
>>   onProperty()
>>  endObject()
>>  beginClass()
>>   onProperty()
>>  endClass()
>>  beginAttachment()
>>    onAttachmentRevision()
>>   endAttachment()
>>  endWikiPage()
>>  beginWikiPageRevision()
>>  beginObject(type)
>>   onProperty()
>>  endObject()
>>  beginObject()
>>   onProperty()
>>  endObject()
>>  onAttachmentRevision() (since i think you can't have several
>> attachement revision for the same wiki page revision)
>>  endWikiPage()
>> endWikiPage()
>>
>
> +1 for something like this.
>
> Now I don't think that other wiki formats (mediawiki, confluence) will
> normally fire events like beginObject() / endObject() but it's important to
> have the full span of XWiki's capabilities embodied into this API. Because
> then we can,
>
> 1) Use this generic importer even when importing xwiki xml exports,
>
> 2) We'll be able to map different concepts of other wiki exports into xwiki
> concepts. An example is mapping [category] links of mediawiki into xwiki
> tags.

3) actually the use case I prefer is the possibility to copy a XWiki
wiki/space on the fly using a XMLRPC or REST based importer :)

>
> Thanks.
>
> - Asiri
>
>
>> basically you can look at XWiki xml ecport format and map something
>> similar for events I think (not the exact same thing the export format
>> need improvements).
>>
>> >>
>> >>
>> >>> I think you should start working on a revised proposal with this
>> >>> information
>> >>> taken into account. In the mean time other developers will comment on
>> this
>> >>> proposal if they have further suggestions.
>> >>>
>> >>> - Asiri
>> >>>
>> >>>
>> >>> > >
>> >>> > >
>> >>> > >> *** [NICE TO HAVE] List<Class<?>> getParameters(): the list of
>> >>> > >> parameters type to provide to the importer, that way you can
>> generate
>> >>> > >> a UI based on it since parameters totally depends on the importer,
>> you
>> >>> > >> could have an importer able to import the wiki from an URL for
>> example
>> >>> > >> using REST/XMLRPC... for the same reason List<InputStream> is not
>> >>> > >> generic enough IMO. You can look at MacroDescriptor for
>> inspiration,
>> >>> > >> we could make it more generic as a list of parameters instead of a
>> >>> > >> list of macro parameters to use it in wiki importer,
>> authenticators,
>> >>> > >> etc. I see this tool as a generic way to get wiki content from any
>> >>> > >> kind of external source, we could even use it to copy a XWiki wiki
>> >>> > >> without having to do an export/import by hand for example.
>> >>> > >>
>> >>> > >
>> >>> > > +1
>> >>> > >
>> >>> > > This will make it possible to define different types of importers
>> with
>> >>> > > different requirements (rather than restricting them to
>> >>> > List<InputStream>)
>> >>> > >
>> >>> > > Thanks.
>> >>> > >
>> >>> > > - Asiri
>> >>> > > _______________________________________________
>> >>> > > devs mailing list
>> >>> > > [hidden email]
>> >>> > > http://lists.xwiki.org/mailman/listinfo/devs
>> >>> > >
>> >>> >
>> >>> >
>> >>> >
>> >>> > --
>> >>> > Thomas Mortagne
>> >>> > _______________________________________________
>> >>> > devs mailing list
>> >>> > [hidden email]
>> >>> > http://lists.xwiki.org/mailman/listinfo/devs
>> >>> >
>> >>> _______________________________________________
>> >>> devs mailing list
>> >>> [hidden email]
>> >>> http://lists.xwiki.org/mailman/listinfo/devs
>> >>>
>> >>
>> >>
>> >>
>> >> --
>> >> Best Regards,
>> >> Arun Reddy
>> >> _______________________________________________
>> >> devs mailing list
>> >> [hidden email]
>> >> http://lists.xwiki.org/mailman/listinfo/devs
>> >>
>> >
>> >
>> >
>> > --
>> > Thomas Mortagne
>> >
>>
>>
>>
>> --
>> Thomas Mortagne
>> _______________________________________________
>> devs mailing list
>> [hidden email]
>> http://lists.xwiki.org/mailman/listinfo/devs
>>
> _______________________________________________
> devs mailing list
> [hidden email]
> http://lists.xwiki.org/mailman/listinfo/devs
>

--
Thomas Mortagne
_______________________________________________
devs mailing list
[hidden email]
http://lists.xwiki.org/mailman/listinfo/devs

[Arun Reddy]

Hi Thomas,

On Mon, Oct 12, 2009 at 8:50 PM, Thomas Mortagne
<[hidden email]>wrote:

> On Mon, Oct 12, 2009 at 11:56, Thomas Mortagne
> <[hidden email]> wrote:
> > On Mon, Oct 12, 2009 at 08:48, Arun Reddy <[hidden email]>
> wrote:
> >> Hi Devs,
> >>
> >> As the existing Wiki Importer ( GSOC PROJECT - SANDBOX
> >> <http://svn.xwiki.org/svnroot/xwiki/sandbox/gsoc/importexport/>) have
> >> few set backs in  architecture, i would like to make a proposal to
> >> modify and enhance to make it better.
> >
> > Great !
> >
> >>
> >> Here is my proposal :
> >>
> >>
> http://gsoc.myxwiki.org/xwiki/bin/view/Import+Export+other+wikis/Wiki+Importer+Enhancement
> >>
> >> - A new Wiki model api ( Wiki, Wiki Page, Wiki Page Revision, Wiki
> >> Attachment ) which adheres to XWiki Standards.
>
> Here is some quick review comments:
> General:
> * this API is not streaming oriented, WikiImporter.import should take
> listener and not send a full Wiki for example. This api is nice but as
> a wrapper for the streaming api (like SAX and DOM or XWiki rednering
> events and XDOM).
>
> * Wiki Importer
> ** you should add some more informationnal methods to use in the UI or
> to make easier to do a generic tool like:
> *** String getName() : the pretty name of the importer
> *** String getDescriptions(): a description of the importer
> *** [NICE TO HAVE] List<Class<?>> getParameters(): the

 Can we have a Descriptor class for these instead of having these methods
inside WikiImporter. Something similiar to MacroDescriptor we can have
WikiDescriptor.

WikiDescriptor::getName():String
WikiDescriptor::getDescription():String
WikiDescriptor::getParametersBeanClass():Class<?>

list of

> parameters type to provide to the importer, that way you can generate
> a UI based on it since parameters totally depends on the importer, you
> could have an importer able to import the wiki from an URL for example
> using REST/XMLRPC... for the same reason List<InputStream> is not
> generic enough IMO. You can look at MacroDescriptor for inspiration,
> we could make it more generic as a list of parameters instead of a
> list of macro parameters to use it in wiki importer, authenticators,
> etc. I see this tool as a generic way to get wiki content from any
> kind of external source, we could even use it to copy a XWiki wiki
> without having to do an export/import by hand for example.
>
>
>>
> >> - Separation of Import process, Parsing the import data into a abstract
> wiki
> >> model will be a component. Clients can use it to make their own custom
> >> implementation. This abstract model is used by the other XWiki component
> /
> >> client code (Groovy & velocity ) to load the data into XWiki with
> standard
> >> features and little customization (like Preserving history, default
> names,
> >> attachments etc.. ).
> >
> > +1
> >
> > Always good to separate as much as possible every aspect of the
> > process in different components.
> >
> >>
> >>
> >> - Using the existing XWiki model api or xwiki-bridge rather than coming
> up
> >> with a new one.
> >>  But the issues here is, i dont have all the things which are needed and
> it
> >> has lot more methods which are of no use in import process. Can i
> proceed
> >> with a custom wiki model like this. ?
> >
> > Main problem is that you need a XDOM based model instead of a String
> > content based model right ? The current model will definitely don't
> > fit this, too bad the new model is not advanced enough. Also the
> > current model is too linked to the storage so it will not be easy to
> > do a clean stream based process.
> >
> > I would say do you own clean custom model and we will see latter for
> > refactor with the new model.
> >
> >>
> >>
> >> - As wiki importer process large streams of data, would it be wise to
> design
> >> the API more stream oriented and include few stream oriented methods.?
> >
> > Big +1 for stream oriented, the "normal" importer should be stream
> > based also IMO. Denis Gervalle is currently working on making the
> > whole export process and its dependencies stream oriented (See
> > http://markmail.org/message/vzfdkteguzselouy).
> >
> > It's definitely the way to go. We can always to a wrapper api which is
> > using the stream based api if we need a not stream based tools but the
> > other way around is not possible so better to do now the more generic
> > one.
> >
> >>
> >>
> >> I would like to have your suggestions on this.
> >>
> >>
> >> Thanks
> >> -Arun Reddy
> >> _______________________________________________
> >> devs mailing list
> >> [hidden email]
> >> http://lists.xwiki.org/mailman/listinfo/devs
> >>
> >
> >
> >
> > --
> > Thomas Mortagne
> >
>
>
>
> --
> Thomas Mortagne
> _______________________________________________
> devs mailing list
> [hidden email]
> http://lists.xwiki.org/mailman/listinfo/devs
>

--
Best Regards,
Arun Reddy
_______________________________________________
devs mailing list
[hidden email]
http://lists.xwiki.org/mailman/listinfo/devs

[tmortagne]

On Thu, Nov 5, 2009 at 06:43, Arun Reddy <[hidden email]> wrote:

> Hi Thomas,
>
> On Mon, Oct 12, 2009 at 8:50 PM, Thomas Mortagne
> <[hidden email]>wrote:
>
>> On Mon, Oct 12, 2009 at 11:56, Thomas Mortagne
>> <[hidden email]> wrote:
>> > On Mon, Oct 12, 2009 at 08:48, Arun Reddy <[hidden email]>
>> wrote:
>> >> Hi Devs,
>> >>
>> >> As the existing Wiki Importer ( GSOC PROJECT - SANDBOX
>> >> <http://svn.xwiki.org/svnroot/xwiki/sandbox/gsoc/importexport/>) have
>> >> few set backs in  architecture, i would like to make a proposal to
>> >> modify and enhance to make it better.
>> >
>> > Great !
>> >
>> >>
>> >> Here is my proposal :
>> >>
>> >>
>> http://gsoc.myxwiki.org/xwiki/bin/view/Import+Export+other+wikis/Wiki+Importer+Enhancement
>> >>
>> >> - A new Wiki model api ( Wiki, Wiki Page, Wiki Page Revision, Wiki
>> >> Attachment ) which adheres to XWiki Standards.
>>
>> Here is some quick review comments:
>> General:
>> * this API is not streaming oriented, WikiImporter.import should take
>> listener and not send a full Wiki for example. This api is nice but as
>> a wrapper for the streaming api (like SAX and DOM or XWiki rednering
>> events and XDOM).
>>
>> * Wiki Importer
>> ** you should add some more informationnal methods to use in the UI or
>> to make easier to do a generic tool like:
>> *** String getName() : the pretty name of the importer
>> *** String getDescriptions(): a description of the importer
>> *** [NICE TO HAVE] List<Class<?>> getParameters(): the
>
>
>  Can we have a Descriptor class for these instead of having these methods
> inside WikiImporter. Something similiar to MacroDescriptor we can have
> WikiDescriptor.
>
> WikiDescriptor::getName():String
> WikiDescriptor::getDescription():String
> WikiDescriptor::getParametersBeanClass():Class<?>

Sure do the same than macros, it's cleaner and more consistent that way.

But it should be WikiImporterDescriptor I think, it's not a descriptor
for a wiki ;)

>
> list of
>> parameters type to provide to the importer, that way you can generate
>> a UI based on it since parameters totally depends on the importer, you
>> could have an importer able to import the wiki from an URL for example
>> using REST/XMLRPC... for the same reason List<InputStream> is not
>> generic enough IMO. You can look at MacroDescriptor for inspiration,
>> we could make it more generic as a list of parameters instead of a
>> list of macro parameters to use it in wiki importer, authenticators,
>> etc. I see this tool as a generic way to get wiki content from any
>> kind of external source, we could even use it to copy a XWiki wiki
>> without having to do an export/import by hand for example.
>>
>>
>>>
>> >> - Separation of Import process, Parsing the import data into a abstract
>> wiki
>> >> model will be a component. Clients can use it to make their own custom
>> >> implementation. This abstract model is used by the other XWiki component
>> /
>> >> client code (Groovy & velocity ) to load the data into XWiki with
>> standard
>> >> features and little customization (like Preserving history, default
>> names,
>> >> attachments etc.. ).
>> >
>> > +1
>> >
>> > Always good to separate as much as possible every aspect of the
>> > process in different components.
>> >
>> >>
>> >>
>> >> - Using the existing XWiki model api or xwiki-bridge rather than coming
>> up
>> >> with a new one.
>> >>  But the issues here is, i dont have all the things which are needed and
>> it
>> >> has lot more methods which are of no use in import process. Can i
>> proceed
>> >> with a custom wiki model like this. ?
>> >
>> > Main problem is that you need a XDOM based model instead of a String
>> > content based model right ? The current model will definitely don't
>> > fit this, too bad the new model is not advanced enough. Also the
>> > current model is too linked to the storage so it will not be easy to
>> > do a clean stream based process.
>> >
>> > I would say do you own clean custom model and we will see latter for
>> > refactor with the new model.
>> >
>> >>
>> >>
>> >> - As wiki importer process large streams of data, would it be wise to
>> design
>> >> the API more stream oriented and include few stream oriented methods.?
>> >
>> > Big +1 for stream oriented, the "normal" importer should be stream
>> > based also IMO. Denis Gervalle is currently working on making the
>> > whole export process and its dependencies stream oriented (See
>> > http://markmail.org/message/vzfdkteguzselouy).
>> >
>> > It's definitely the way to go. We can always to a wrapper api which is
>> > using the stream based api if we need a not stream based tools but the
>> > other way around is not possible so better to do now the more generic
>> > one.
>> >
>> >>
>> >>
>> >> I would like to have your suggestions on this.
>> >>
>> >>
>> >> Thanks
>> >> -Arun Reddy
>> >> _______________________________________________
>> >> devs mailing list
>> >> [hidden email]
>> >> http://lists.xwiki.org/mailman/listinfo/devs
>> >>
>> >
>> >
>> >
>> > --
>> > Thomas Mortagne
>> >
>>
>>
>>
>> --
>> Thomas Mortagne
>> _______________________________________________
>> devs mailing list
>> [hidden email]
>> http://lists.xwiki.org/mailman/listinfo/devs
>>
>
>
>
> --
> Best Regards,
> Arun Reddy
> _______________________________________________
> devs mailing list
> [hidden email]
> http://lists.xwiki.org/mailman/listinfo/devs
>

--
Thomas Mortagne
_______________________________________________
devs mailing list
[hidden email]
http://lists.xwiki.org/mailman/listinfo/devs