Extension processes for WPS in deegree3

[Stefan Kurzbach]

Hello everybody,

the developers of the Kalypso software platform
(http://kalypso.sourceforge.net) would like to use deegree3 WPS for
providing Kalypso simulations as processes. I want to discuss how this
could be done in a dynamic way so that other projects can profit from
the solution as well.

Kalypso is an open source software system for numerical simulations with
user interface and server components. Kalypso offers server-side
simulations via a WPS interface (own implementation, version 0.4).
Because we don't want to re-invent the wheel (again) we would like to
use the existing deegree WPS implementation for version 1.x processes.

Our software system is based on the Eclipse/OSGI plug-in model which
makes it hard to be integrated with deegree in a consistent way. Deegree
WPS uses static configuration files while Kalypso simulations are
dynamic extensions to the platform. This model allows us to hook new
simulations into a system by deploying a new plug-in. We can also
provide deegree3 services by means of a deegree3 plug-in.

The main problem is that each plug-in has its own classloader and
libraries. This prohibits that the deegree3 plug-in can load the
required libraries and processes of our simulations. Calling
Class.forName() for loading a processlet from its name in the
configuration file simply does not work if the processlet resides in a
different plug-in. For this reason we need a different method to add our
processlet implementation to a deegree3 WPService the "OSGI-way".

One solution would be to use either OSGI dynamic services or an Eclipse
extension point in the deegree3 plug-in. As it is probably not feasible
to make deegree based on OSGI, this deegree3 plug-in could reside in
Kalypso. The WPService implementation would then need a method to
register processlets and process definitions from outside.

Does anybody else have the need to do something similar?

Bye,
Stefan

------------------------------------------------------------------------------
Come build with us! The BlackBerry® Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9-12, 2009. Register now!
http://p.sf.net/sfu/devconf
_______________________________________________
deegree-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/deegree-devel

[Markus Schneider-4]

Hi Stefan,

Stefan Kurzbach wrote:

> ...
>
> Kalypso is an open source software system for numerical simulations with
> user interface and server components. Kalypso offers server-side
> simulations via a WPS interface (own implementation, version 0.4).
> Because we don't want to re-invent the wheel (again) we would like to
> use the existing deegree WPS implementation for version 1.x processes.

Sounds great!

> Our software system is based on the Eclipse/OSGI plug-in model which
> makes it hard to be integrated with deegree in a consistent way. Deegree
> WPS uses static configuration files while Kalypso simulations are
> dynamic extensions to the platform. This model allows us to hook new
> simulations into a system by deploying a new plug-in. We can also
> provide deegree3 services by means of a deegree3 plug-in.

As already discussed in other threads, we would also like to enable dynamic configuration of our deegree 3 services at
runtime. Besides other possibilties, this would be especially useful for the implementation of a service management
console [1]. However, we currently don't have resources and funding for that, so it has rather low-priority for the core
team at lat/lon :-(

> The main problem is that each plug-in has its own classloader and
> libraries. This prohibits that the deegree3 plug-in can load the
> required libraries and processes of our simulations. Calling
> Class.forName() for loading a processlet from its name in the
> configuration file simply does not work if the processlet resides in a
> different plug-in. For this reason we need a different method to add our
> processlet implementation to a deegree3 WPService the "OSGI-way".
>
> One solution would be to use either OSGI dynamic services or an Eclipse
> extension point in the deegree3 plug-in. As it is probably not feasible
> to make deegree based on OSGI, this deegree3 plug-in could reside in
> Kalypso. The WPService implementation would then need a method to
> register processlets and process definitions from outside.

I can offer to add the following two methods to WPService:

    /**
     * Adds a new {@link Process} to the service.
     *
     * @param def
     *            process definition, may no be null
     * @param processletClass
     *            matching {@link Processlet} implementation, may be null (in this case the class name from
     *            <code>def</code> is instantiated using the default classloader)
     */
    public void addProcess( ProcessDefinition def, Class<Processlet> processletClass );

    /**
     * Removes the specified {@link Process} from the active processes.
     *
     * @param processId
     *            id of the process, must not be null
     */
    public void removeProcess( String processId );

Would this be sufficient for your use-case?

Best regards,
Markus

P.S.: Can you elaborate a bit about the extension-point mechanism of OSGI? How does it work and what are the benefits?
And what do you need for OSGI-enabling an application? I am aware of the additional JAR metainformation (dependencies),
but I wonder what else is needed...

[1] https://wiki.deegree.org/deegreeWiki/deegree3/ServiceManagementConsolePlans

--
Markus Schneider

l a t / l o n  GmbH
Aennchenstrasse 19           53177 Bonn, Germany
phone ++49 [phone]       fax ++49 [phone]
http://www.lat-lon.de        http://www.deegree.org



------------------------------------------------------------------------------
Come build with us! The BlackBerry® Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9-12, 2009. Register now!
http://p.sf.net/sfu/devconf
_______________________________________________
deegree-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/deegree-devel

[Stefan Kurzbach]

Hello Markus,

thank you for your reply.

> As already discussed in other threads, we would also like to enable dynamic configuration of our deegree 3 services at
> runtime. Besides other possibilties, this would be especially useful for the implementation of a service management
> console [1]. However, we currently don't have resources and funding for that, so it has rather low-priority for the core
> team at lat/lon :-(
>
>  
> I can offer to add the following two methods to WPService:
>
>     /**
>      * Adds a new {@link Process} to the service.
>      *
>      * @param def
>      *            process definition, may no be null
>      * @param processletClass
>      *            matching {@link Processlet} implementation, may be null (in this case the class name from
>      *            <code>def</code> is instantiated using the default classloader)
>      */
>     public void addProcess( ProcessDefinition def, Class<Processlet> processletClass );
>
>     /**
>      * Removes the specified {@link Process} from the active processes.
>      *
>      * @param processId
>      *            id of the process, must not be null
>      */
>     public void removeProcess( String processId );
>
> Would this be sufficient for your use-case

I think these methods do what I need. I would call the static method
OGCFrontController.getServiceController(WPSController.class).getService()
to add new processes to the WPService. However, I have a remark
regarding the current implementation of WPService. I noticed that only
one instance of a Processlet of a kind is created by the server. This
instance is reused for each call to the specific process. This practise
may lead to synchronization errors if the Processlet is not designed to
be thread-safe. I suggest to create an instance of a Processlet for each
execution request instead.
> P.S.: Can you elaborate a bit about the extension-point mechanism of OSGI? How does it work and what are the benefits?
> And what do you need for OSGI-enabling an application? I am aware of the additional JAR metainformation (dependencies),
> but I wonder what else is needed...
>  
Extensions and extension points are an Eclipse mechanism for plug-ins to
add functionality to another plug-in. If you want to use it, you have to
create a dependency on the core Eclipse platform plug-ins and are bound
to use Equinox as your OSGI implementation (it comes with Eclipse). An
extension can be lazily initialized. In the case of a Processlet this
might be when the process is executed for the first time. The advantage
is that the classes of a plug-in providing a process need not be loaded
before it is used. You have loose coupling between the process and the
WPService. The same can be achieved using OSGI declarative services.

All that is needed for OSGI-enabling deegree is a manifest.mf file
containing the meta information about the OSGI bundle (in a directory
called "META-INF"). This would include the libraries in the bundle
classpath, which of the classes are made visible to other bundles (e.g.
not all of the libraries and internal classes have to be exposed), and a
dependency on the "org.osgi.framework" bundle. Pack this into a jar file
and put it into an OSGI container.

About extension points see e.g.
http://wiki.eclipse.org/FAQ_What_are_extensions_and_extension_points%3F
About OSGI declarative services see e.g.
http://www.eclipsezone.com/eclipse/forums/t96740.html and
http://www.slideshare.net/njbartlett/component-oriented-development-in-osgi-with-declarative-services-spring-dynamic-modules-and-apache-ipojo
Note: Eclipse plug-ins are OSGI bundles, but the extension points are
not implemented as OSGI declarative services

Best Regards,
Stefan

------------------------------------------------------------------------------
Come build with us! The BlackBerry® Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9-12, 2009. Register now!
http://p.sf.net/sfu/devconf
_______________________________________________
deegree-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/deegree-devel

[Markus Schneider-4]

Hi Stefan,

Stefan Kurzbach wrote:

>> Would this be sufficient for your use-case
> I think these methods do what I need. I would call the static method
> OGCFrontController.getServiceController(WPSController.class).getService()
> to add new processes to the WPService.

Good. I will add them ASAP.

> However, I have a remark
> regarding the current implementation of WPService. I noticed that only
> one instance of a Processlet of a kind is created by the server. This
> instance is reused for each call to the specific process. This practise
> may lead to synchronization errors if the Processlet is not designed to
> be thread-safe. I suggest to create an instance of a Processlet for each
> execution request instead.

You are right, processlets *need* to be thread-safe (and we must make this very clear in the documentation). But this is
not necessarily bad (the same applies for servlets) and it also has a clear advantage. BTW, the processlet interface was
designed with the (well-tested) Servlet interface in mind:

a) A Servlet is a piece of code that can be accessed via HTTP methods.
b) A Processlet is a piece of code that can be accessed via the WPS-Execute protocol.

Besides being based on a proven concept, I see the main benefit of the current approach (one Processlet instance per
Process type) in its clarity with respect to the lifecycle of Processlet objects: init() and destroy() are just called
once. If you had multiple instances, would you want init() to be called multiple times?

>> P.S.: Can you elaborate a bit about the extension-point mechanism of OSGI? How does it work and what are the benefits?
>> And what do you need for OSGI-enabling an application? I am aware of the additional JAR metainformation (dependencies),
>> but I wonder what else is needed...
>>  
> Extensions and extension points are an Eclipse mechanism for plug-ins to
> add functionality to another plug-in. If you want to use it, you have to
> create a dependency on the core Eclipse platform plug-ins and are bound
> to use Equinox as your OSGI implementation (it comes with Eclipse). An
> extension can be lazily initialized. In the case of a Processlet this
> might be when the process is executed for the first time. The advantage
> is that the classes of a plug-in providing a process need not be loaded
> before it is used. You have loose coupling between the process and the
> WPService. The same can be achieved using OSGI declarative services.
>
> All that is needed for OSGI-enabling deegree is a manifest.mf file
> containing the meta information about the OSGI bundle (in a directory
> called "META-INF"). This would include the libraries in the bundle
> classpath, which of the classes are made visible to other bundles (e.g.
> not all of the libraries and internal classes have to be exposed), and a
> dependency on the "org.osgi.framework" bundle. Pack this into a jar file
> and put it into an OSGI container.
>
> About extension points see e.g.
> http://wiki.eclipse.org/FAQ_What_are_extensions_and_extension_points%3F
> About OSGI declarative services see e.g.
> http://www.eclipsezone.com/eclipse/forums/t96740.html and
> http://www.slideshare.net/njbartlett/component-oriented-development-in-osgi-with-declarative-services-spring-dynamic-modules-and-apache-ipojo
> Note: Eclipse plug-ins are OSGI bundles, but the extension points are
> not implemented as OSGI declarative services

Thanks for the explanation. We're still thinking about the best way to package process definitions and processlets
implementations for deploying dynamically, e.g. through a web management console or a hot-deployment directory.
Obviously, they could be packaged in a JAR. Of course, we would have to provide our own classloader for dynamically
instantiating the Processlet class. Do you think this works easier / better if we would just use OSGI bundles for that?

Best regards,
Markus
--
Markus Schneider

l a t / l o n  GmbH
Aennchenstrasse 19           53177 Bonn, Germany
phone ++49 [phone]       fax ++49 [phone]
http://www.lat-lon.de        http://www.deegree.org



------------------------------------------------------------------------------
Come build with us! The BlackBerry® Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9-12, 2009. Register now!
http://p.sf.net/sfu/devconf
_______________________________________________
deegree-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/deegree-devel

[Markus Schneider-4]

Another thought:

Markus Schneider wrote:

> You are right, processlets *need* to be thread-safe (and we must make this very clear in the documentation). But this is
> not necessarily bad (the same applies for servlets) and it also has a clear advantage. BTW, the processlet interface was
> designed with the (well-tested) Servlet interface in mind:
>
> a) A Servlet is a piece of code that can be accessed via HTTP methods.
> b) A Processlet is a piece of code that can be accessed via the WPS-Execute protocol.
>
> Besides being based on a proven concept, I see the main benefit of the current approach (one Processlet instance per
> Process type) in its clarity with respect to the lifecycle of Processlet objects: init() and destroy() are just called
> once. If you had multiple instances, would you want init() to be called multiple times?

Maybe we could just add an abstract base class (e.g. ThreadSafeProcesslet) on top of the current concept that takes care
of initializing a new instance for each execute request and just calls another method, e.g. #executeSafe(...).

What do you think?

Best regards,
Markus

--
Markus Schneider

l a t / l o n  GmbH
Aennchenstrasse 19           53177 Bonn, Germany
phone ++49 [phone]       fax ++49 [phone]
http://www.lat-lon.de        http://www.deegree.org



------------------------------------------------------------------------------
Come build with us! The BlackBerry® Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9-12, 2009. Register now!
http://p.sf.net/sfu/devconf
_______________________________________________
deegree-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/deegree-devel

[Stefan Kurzbach]

Hello again,

> You are right, processlets *need* to be thread-safe (and we must make
> this very clear in the documentation). But this is
> not necessarily bad (the same applies for servlets) and it also has a
> clear advantage. BTW, the processlet interface was
> designed with the (well-tested) Servlet interface in mind:
>
> a) A Servlet is a piece of code that can be accessed via HTTP methods.
> b) A Processlet is a piece of code that can be accessed via the
> WPS-Execute protocol.
>
> Besides being based on a proven concept, I see the main benefit of the
> current approach (one Processlet instance per
> Process type) in its clarity with respect to the lifecycle of
> Processlet objects: init() and destroy() are just called
> once. If you had multiple instances, would you want init() to be
> called multiple times?
> Maybe we could just add an abstract base class (e.g.
> ThreadSafeProcesslet) on top of the current concept that takes care
> of initializing a new instance for each execute request and just calls
> another method, e.g. #executeSafe(...).
>
> What do you think

Your point is clear. I guess there are just different views on that and
they all have advantages and disadvantages. I can live with the analogy
of a Servlet and a Processlet. If you want to provide a thread-safe
implementation I would not put that code into the Processlet class (or a
parent class), but provide a configuration option.
> Thanks for the explanation. We're still thinking about the best way to
> package process definitions and processlets
> implementations for deploying dynamically, e.g. through a web
> management console or a hot-deployment directory.
> Obviously, they could be packaged in a JAR. Of course, we would have
> to provide our own classloader for dynamically
> instantiating the Processlet class. Do you think this works easier /
> better if we would just use OSGI bundles for that?
Definately. You get the classloader with plug-in scope for free. The
overhead of OSGI is not that big, and it is possible to deploy an
OSGI-framework in an application server
(http://eclipse.dzone.com/articles/embedding-osgi-tomcat). A
hot-deployment directory also exists in some implementations (e.g.
http://felix.apache.org, "fileinstall" bundle).

When looking at the code I noticed that the process execution state is
only updated once after the process has started and once after the
process() method returns. This way the progress monitor is not really
useful. It is also not possible to report intermediate results of a
long-running simulation to the user.

Best regards,
Stefan

------------------------------------------------------------------------------
Come build with us! The BlackBerry® Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9-12, 2009. Register now!
http://p.sf.net/sfu/devconf
_______________________________________________
deegree-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/deegree-devel

[Markus Schneider-4]

Hi Stefan,

Stefan Kurzbach wrote:

> When looking at the code I noticed that the process execution state is
> only updated once after the process has started and once after the
> process() method returns. This way the progress monitor is not really
> useful. It is also not possible to report intermediate results of a
> long-running simulation to the user.

This is not true -- you just have to make sure that your process actually provides this information.

See for yourself: Simply start executing the parameter demo process and set LiteralInput to something high, e.g. 60
(this means that the execution will take 60 seconds). Afterwards, you may poll the execution status page (linked from
the default webapp start page) and see that it indeed reports proper percentages [1].

In order to set the percentage, your processlet must call

public void process( ProcessletInputs in, ProcessletOutputs out, ProcessletExecutionInfo info )
                            throws ProcessletException {
...
  info.setPercentCompleted( percentCompleted );
...
}

in regular intervals. You may also provide start/stop detail messages [2].

Best regards,
Markus

[1] http://web.lat-lon.de/~schneider/pics/wps_status.png
[2] http://download.deegree.org/deegree3/nightly/services/javadoc/org/deegree/services/wps/ProcessletExecutionInfo.html

--
Markus Schneider

l a t / l o n  GmbH
Aennchenstrasse 19           53177 Bonn, Germany
phone ++49 [phone]       fax ++49 [phone]
http://www.lat-lon.de        http://www.deegree.org



------------------------------------------------------------------------------
Come build with us! The BlackBerry® Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9-12, 2009. Register now!
http://p.sf.net/sfu/devconf
_______________________________________________
deegree-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/deegree-devel