finally

7 messages Options
Embed this post
Permalink
Haris Kurtagic

finally

Reply Threaded More More options
Print post
Permalink
Some javascript/style in this post has been disabled (why?)

Finally we found what was causing MapGuide server to crash while reading data from MapGuide.

It took us lot of energy and time.

 

Problem will appear in multiuser environment with concurent access when you are using provider which is per command threaded or sharing (caching) FDO schema accross multiple connections.

 

Issue is in function: MgServerGetFeatures::SerializeToXml(FdoClassDefinition* classDef) which will in order to serialize class to xml remove that class from schema, put it to temporary schema serialize and return class to original schema. While doing this if another thread goes to execute, problem starts.

 

Workaround is to set provider to per connection threaded or not cache schema at all, and both comes with perfomance penalties.

 

I think that few things are important in this case beside the obvious issue. One of them is missing constant pointers in FDO API. Schema returned from provider shouldn't be allowed to be changed by fdo clients. That is something were problems appeared already.

Also, I think that serialization using XML is not right way to go. XML we could use to serialize data to end-user not for internal MG processes.

 

Haris


_______________________________________________
fdo-users mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/fdo-users
doggybs

Re: finally

Reply Threaded More More options
Print post
Permalink
That is good news! I think I have been casing this problem for a while, when mapguide will lockup and crash when multiple users are viewing data from the postgis FDO.

Would it be possible for you to give some more details on how to setup the work arounds you have suggested.

Many Thanks
James


Haris Kurtagic wrote:
Finally we found what was causing MapGuide server to crash while reading
data from MapGuide.

It took us lot of energy and time.

 

Problem will appear in multiuser environment with concurent access when
you are using provider which is per command threaded or sharing
(caching) FDO schema accross multiple connections.

 

Issue is in function:
MgServerGetFeatures::SerializeToXml(FdoClassDefinition* classDef) which
will in order to serialize class to xml remove that class from schema,
put it to temporary schema serialize and return class to original
schema. While doing this if another thread goes to execute, problem
starts.

 

Workaround is to set provider to per connection threaded or not cache
schema at all, and both comes with perfomance penalties.

 

I think that few things are important in this case beside the obvious
issue. One of them is missing constant pointers in FDO API. Schema
returned from provider shouldn't be allowed to be changed by fdo
clients. That is something were problems appeared already.

Also, I think that serialization using XML is not right way to go. XML
we could use to serialize data to end-user not for internal MG
processes.

 

Haris


_______________________________________________
fdo-users mailing list
fdo-users@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/fdo-users
Haris Kurtagic

RE: finally

Reply Threaded More More options
Print post
Permalink
I guess PostGis provider is caching schema too. I don't know if it
allows for multiple simultaneous commands to run over one connection but
if it does then that could be same problem.

Real fix needs to be done in MapGuide. For our project I have changed
King.Oracle but that is just quick workaround to get us running.

I just realized I posted to FDO/MapGuide-user lists.
I should post to internals too ?

Haris

-----Original Message-----
From: [hidden email]
[mailto:[hidden email]] On Behalf Of doggybs
Sent: Tuesday, May 19, 2009 2:19 PM
To: [hidden email]
Subject: Re: [fdo-users] finally


That is good news! I think I have been casing this problem for a while,
when
mapguide will lockup and crash when multiple users are viewing data from
the
postgis FDO.

Would it be possible for you to give some more details on how to setup
the
work arounds you have suggested.

Many Thanks
James



Haris Kurtagic wrote:
>
> Finally we found what was causing MapGuide server to crash while
reading
> data from MapGuide.
>
> It took us lot of energy and time.
>
>  
>
> Problem will appear in multiuser environment with concurent access
when
> you are using provider which is per command threaded or sharing
> (caching) FDO schema accross multiple connections.
>
>  
>
> Issue is in function:
> MgServerGetFeatures::SerializeToXml(FdoClassDefinition* classDef)
which

> will in order to serialize class to xml remove that class from schema,
> put it to temporary schema serialize and return class to original
> schema. While doing this if another thread goes to execute, problem
> starts.
>
>  
>
> Workaround is to set provider to per connection threaded or not cache
> schema at all, and both comes with perfomance penalties.
>
>  
>
> I think that few things are important in this case beside the obvious
> issue. One of them is missing constant pointers in FDO API. Schema
> returned from provider shouldn't be allowed to be changed by fdo
> clients. That is something were problems appeared already.
>
> Also, I think that serialization using XML is not right way to go. XML
> we could use to serialize data to end-user not for internal MG
> processes.
>
>  
>
> Haris
>
>
> _______________________________________________
> fdo-users mailing list
> [hidden email]
> http://lists.osgeo.org/mailman/listinfo/fdo-users
>
>

--
View this message in context:
http://n2.nabble.com/finally-tp2939043p2939178.html
Sent from the FDO Users mailing list archive at Nabble.com.

_______________________________________________
fdo-users mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/fdo-users
_______________________________________________
fdo-users mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/fdo-users
Traian Stanev

RE: finally

Reply Threaded More More options
Print post
Permalink

Hi Haris,

It's wrong if a provider internally caches the schema returned by the DescribeSchema call. I used to do this too in some providers, until I realized that people get the schema and then change it destructively -- they are unfortunately allowed to, since the schema returned is not  constant pointer. In my case it was FME doing that when creating a schema for a new SQLite data store, so it had nothing to do with multithreading access, yet was still causing problems. One thing we could do is provide an extra, faster API for DescribeScheme, which returns a cached pointer to the schema, which the caller is not allowed to modify.

I also agree that MapGuide should not be using the XML serialization of the FDO schema to pass around internally. Not only because of threading issues, but also because the performance hit of XML serialization is just too big.


Traian


-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Haris Kurtagic
Sent: Tuesday, May 19, 2009 4:01 PM
To: FDO Users Mail List
Subject: RE: [fdo-users] finally

I guess PostGis provider is caching schema too. I don't know if it
allows for multiple simultaneous commands to run over one connection but
if it does then that could be same problem.

Real fix needs to be done in MapGuide. For our project I have changed
King.Oracle but that is just quick workaround to get us running.

I just realized I posted to FDO/MapGuide-user lists.
I should post to internals too ?

Haris

-----Original Message-----
From: [hidden email]
[mailto:[hidden email]] On Behalf Of doggybs
Sent: Tuesday, May 19, 2009 2:19 PM
To: [hidden email]
Subject: Re: [fdo-users] finally


That is good news! I think I have been casing this problem for a while,
when
mapguide will lockup and crash when multiple users are viewing data from
the
postgis FDO.

Would it be possible for you to give some more details on how to setup
the
work arounds you have suggested.

Many Thanks
James



Haris Kurtagic wrote:
>
> Finally we found what was causing MapGuide server to crash while
reading
> data from MapGuide.
>
> It took us lot of energy and time.
>
>  
>
> Problem will appear in multiuser environment with concurent access
when
> you are using provider which is per command threaded or sharing
> (caching) FDO schema accross multiple connections.
>
>  
>
> Issue is in function:
> MgServerGetFeatures::SerializeToXml(FdoClassDefinition* classDef)
which

> will in order to serialize class to xml remove that class from schema,
> put it to temporary schema serialize and return class to original
> schema. While doing this if another thread goes to execute, problem
> starts.
>
>  
>
> Workaround is to set provider to per connection threaded or not cache
> schema at all, and both comes with perfomance penalties.
>
>  
>
> I think that few things are important in this case beside the obvious
> issue. One of them is missing constant pointers in FDO API. Schema
> returned from provider shouldn't be allowed to be changed by fdo
> clients. That is something were problems appeared already.
>
> Also, I think that serialization using XML is not right way to go. XML
> we could use to serialize data to end-user not for internal MG
> processes.
>
>  
>
> Haris
>
>
> _______________________________________________
> fdo-users mailing list
> [hidden email]
> http://lists.osgeo.org/mailman/listinfo/fdo-users
>
>

--
View this message in context:
http://n2.nabble.com/finally-tp2939043p2939178.html
Sent from the FDO Users mailing list archive at Nabble.com.

_______________________________________________
fdo-users mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/fdo-users
_______________________________________________
fdo-users mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/fdo-users
_______________________________________________
fdo-users mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/fdo-users
Haris Kurtagic

RE: finally

Reply Threaded More More options
Print post
Permalink
It is not DescribeSchema which is problem in this case. MG is taking
class from reader and is changing parent schema of it.
Multithreading is making this problem with reader to show up. Problem
you had in sqlite is solved same way as it was solved in King.Oracle
with deepcopy of schema. But that was another issue.

Of course, performance is reason to not use xml.

Haris.


-----Original Message-----
From: [hidden email]
[mailto:[hidden email]] On Behalf Of Traian Stanev
Sent: Tuesday, May 19, 2009 10:00 PM
To: FDO Users Mail List
Subject: RE: [fdo-users] finally


Hi Haris,

It's wrong if a provider internally caches the schema returned by the
DescribeSchema call. I used to do this too in some providers, until I
realized that people get the schema and then change it destructively --
they are unfortunately allowed to, since the schema returned is not
constant pointer. In my case it was FME doing that when creating a
schema for a new SQLite data store, so it had nothing to do with
multithreading access, yet was still causing problems. One thing we
could do is provide an extra, faster API for DescribeScheme, which
returns a cached pointer to the schema, which the caller is not allowed
to modify.

I also agree that MapGuide should not be using the XML serialization of
the FDO schema to pass around internally. Not only because of threading
issues, but also because the performance hit of XML serialization is
just too big.


Traian


-----Original Message-----
From: [hidden email]
[mailto:[hidden email]] On Behalf Of Haris Kurtagic
Sent: Tuesday, May 19, 2009 4:01 PM
To: FDO Users Mail List
Subject: RE: [fdo-users] finally

I guess PostGis provider is caching schema too. I don't know if it
allows for multiple simultaneous commands to run over one connection but
if it does then that could be same problem.

Real fix needs to be done in MapGuide. For our project I have changed
King.Oracle but that is just quick workaround to get us running.

I just realized I posted to FDO/MapGuide-user lists.
I should post to internals too ?

Haris

-----Original Message-----
From: [hidden email]
[mailto:[hidden email]] On Behalf Of doggybs
Sent: Tuesday, May 19, 2009 2:19 PM
To: [hidden email]
Subject: Re: [fdo-users] finally


That is good news! I think I have been casing this problem for a while,
when
mapguide will lockup and crash when multiple users are viewing data from
the
postgis FDO.

Would it be possible for you to give some more details on how to setup
the
work arounds you have suggested.

Many Thanks
James



Haris Kurtagic wrote:
>
> Finally we found what was causing MapGuide server to crash while
reading
> data from MapGuide.
>
> It took us lot of energy and time.
>
>  
>
> Problem will appear in multiuser environment with concurent access
when
> you are using provider which is per command threaded or sharing
> (caching) FDO schema accross multiple connections.
>
>  
>
> Issue is in function:
> MgServerGetFeatures::SerializeToXml(FdoClassDefinition* classDef)
which

> will in order to serialize class to xml remove that class from schema,
> put it to temporary schema serialize and return class to original
> schema. While doing this if another thread goes to execute, problem
> starts.
>
>  
>
> Workaround is to set provider to per connection threaded or not cache
> schema at all, and both comes with perfomance penalties.
>
>  
>
> I think that few things are important in this case beside the obvious
> issue. One of them is missing constant pointers in FDO API. Schema
> returned from provider shouldn't be allowed to be changed by fdo
> clients. That is something were problems appeared already.
>
> Also, I think that serialization using XML is not right way to go. XML
> we could use to serialize data to end-user not for internal MG
> processes.
>
>  
>
> Haris
>
>
> _______________________________________________
> fdo-users mailing list
> [hidden email]
> http://lists.osgeo.org/mailman/listinfo/fdo-users
>
>

--
View this message in context:
http://n2.nabble.com/finally-tp2939043p2939178.html
Sent from the FDO Users mailing list archive at Nabble.com.

_______________________________________________
fdo-users mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/fdo-users
_______________________________________________
fdo-users mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/fdo-users
_______________________________________________
fdo-users mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/fdo-users
_______________________________________________
fdo-users mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/fdo-users
Jason Birch

RE: finally

Reply Threaded More More options
Print post
Permalink
Just bringing MapGuide-Internals and FDO-Internals into this thread, in case the MapGuide devs aren't following the users list closely.

Jason

-----Original Message-----
From: Haris Kurtagic
Sent: Wednesday, May 20, 2009 12:14 AM
Subject: RE: [fdo-users] finally

It is not DescribeSchema which is problem in this case. MG is taking
class from reader and is changing parent schema of it.
Multithreading is making this problem with reader to show up. Problem
you had in sqlite is solved same way as it was solved in King.Oracle
with deepcopy of schema. But that was another issue.

Of course, performance is reason to not use xml.
_______________________________________________
fdo-users mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/fdo-users
doggybs

RE: finally

Reply Threaded More More options
Print post
Permalink
This is a real problem to us, as we are hoping to go live with our new system in a few weeks. I don't suppose there are any plans to release a patched postgis FDO with the work around Haris has suggested until mapguide is fixed?

Many Thanks
James

Jason Birch wrote:
Just bringing MapGuide-Internals and FDO-Internals into this thread, in case the MapGuide devs aren't following the users list closely.

Jason

-----Original Message-----
From: Haris Kurtagic
Sent: Wednesday, May 20, 2009 12:14 AM
Subject: RE: [fdo-users] finally

It is not DescribeSchema which is problem in this case. MG is taking
class from reader and is changing parent schema of it.
Multithreading is making this problem with reader to show up. Problem
you had in sqlite is solved same way as it was solved in King.Oracle
with deepcopy of schema. But that was another issue.

Of course, performance is reason to not use xml.
_______________________________________________
fdo-users mailing list
fdo-users@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/fdo-users