MapGuide RFC 65 - ACE 5.6 upgrade is ready for review

[Bruce Dechant]

Hi All,



Please review MapGuide RFC 65 - ACE 5.6 upgrade.

http://trac.osgeo.org/mapguide/wiki/MapGuideRfc65



Post any comments or questions to the mailing list.



Thanks,

Bruce


_______________________________________________
mapguide-internals mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/mapguide-internals

[Trevor Wekel-3]

Hi Bruce,

As part of the upgrade we may want to switch ACE on Windows to use either ACE_TP_Reactor or ACE_Select_Reactor instead of the ACE_WFMO_Reactor.  The WFMO reactor can only handle 62 concurrent connections whereas the TP and Select reactors can handle 1024 connections.

Thanks,
Trevor  


-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Bruce Dechant
Sent: Thursday, June 04, 2009 10:42 AM
To: MapGuide Internals Mail List
Subject: [mapguide-internals] MapGuide RFC 65 - ACE 5.6 upgrade is ready for review

Hi All,



Please review MapGuide RFC 65 - ACE 5.6 upgrade.

http://trac.osgeo.org/mapguide/wiki/MapGuideRfc65



Post any comments or questions to the mailing list.



Thanks,

Bruce


_______________________________________________
mapguide-internals mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/mapguide-internals
_______________________________________________
mapguide-internals mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/mapguide-internals

[Jason Birch]

That would be nice.  

Although the recent http connection fixes seem to have worked around the problems fairly well and reduced the number of hard failures for end users, I've noticed that in some cases a user's performance degrades considerably (hangs for N seconds) while waiting for connections that are inactive but still open to be closed properly.

It's also my understanding (from previous conversations) that some newer hardware can actually handle more than 62 connections worth of work, and that the current limitation would be artificially constraining the performance of these boxes.
 
Jason

-----Original Message-----
From: Trevor Wekel
Sent: Thursday, June 04, 2009 12:40 PM
To: MapGuide Internals Mail List
Subject: RE: [mapguide-internals] MapGuide RFC 65 - ACE 5.6 upgrade is ready for review

As part of the upgrade we may want to switch ACE on Windows to use either ACE_TP_Reactor or ACE_Select_Reactor instead of the ACE_WFMO_Reactor.  The WFMO reactor can only handle 62 concurrent connections whereas the TP and Select reactors can handle 1024 connections.
_______________________________________________
mapguide-internals mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/mapguide-internals

[Trevor Wekel-3]

Yes.  Intel Nehalems are 4 core w/ hyperthreading and AMD Istanbuls are 6 core chips.  Both of these are available in two socket configuration and the AMD chips are also available in 4 and 8 socket configurations.  This brings the total concurrent "core" count to 12, 16, 24, or 48 depending on configuration.

If we assume a 24 "core" variant then the current WMFO reactor will only be able to supply two concurrent threads of work per core.  Depending on IO constraints, this may not be enough to fully utilize all of the CPU on these newer boxes.


Thanks,
Trevor


-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Jason Birch
Sent: Thursday, June 04, 2009 1:52 PM
To: MapGuide Internals Mail List
Subject: RE: [mapguide-internals] MapGuide RFC 65 - ACE 5.6 upgrade is ready for review

That would be nice.  

Although the recent http connection fixes seem to have worked around the problems fairly well and reduced the number of hard failures for end users, I've noticed that in some cases a user's performance degrades considerably (hangs for N seconds) while waiting for connections that are inactive but still open to be closed properly.

It's also my understanding (from previous conversations) that some newer hardware can actually handle more than 62 connections worth of work, and that the current limitation would be artificially constraining the performance of these boxes.
 
Jason

-----Original Message-----
From: Trevor Wekel
Sent: Thursday, June 04, 2009 12:40 PM
To: MapGuide Internals Mail List
Subject: RE: [mapguide-internals] MapGuide RFC 65 - ACE 5.6 upgrade is ready for review

As part of the upgrade we may want to switch ACE on Windows to use either ACE_TP_Reactor or ACE_Select_Reactor instead of the ACE_WFMO_Reactor.  The WFMO reactor can only handle 62 concurrent connections whereas the TP and Select reactors can handle 1024 connections.
_______________________________________________
mapguide-internals mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/mapguide-internals
_______________________________________________
mapguide-internals mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/mapguide-internals

[Traian Stanev]

While this is true, don't we also have the big giant lock in the code, which makes sure that if you have rasters for example (where I/O matters), you will only be working on one request at a time anyway?


Traian


-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Trevor Wekel
Sent: Thursday, June 04, 2009 4:10 PM
To: MapGuide Internals Mail List
Subject: RE: [mapguide-internals] MapGuide RFC 65 - ACE 5.6 upgrade is ready for review

Yes.  Intel Nehalems are 4 core w/ hyperthreading and AMD Istanbuls are 6 core chips.  Both of these are available in two socket configuration and the AMD chips are also available in 4 and 8 socket configurations.  This brings the total concurrent "core" count to 12, 16, 24, or 48 depending on configuration.

If we assume a 24 "core" variant then the current WMFO reactor will only be able to supply two concurrent threads of work per core.  Depending on IO constraints, this may not be enough to fully utilize all of the CPU on these newer boxes.


Thanks,
Trevor


-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Jason Birch
Sent: Thursday, June 04, 2009 1:52 PM
To: MapGuide Internals Mail List
Subject: RE: [mapguide-internals] MapGuide RFC 65 - ACE 5.6 upgrade is ready for review

That would be nice.  

Although the recent http connection fixes seem to have worked around the problems fairly well and reduced the number of hard failures for end users, I've noticed that in some cases a user's performance degrades considerably (hangs for N seconds) while waiting for connections that are inactive but still open to be closed properly.

It's also my understanding (from previous conversations) that some newer hardware can actually handle more than 62 connections worth of work, and that the current limitation would be artificially constraining the performance of these boxes.
 
Jason

-----Original Message-----
From: Trevor Wekel
Sent: Thursday, June 04, 2009 12:40 PM
To: MapGuide Internals Mail List
Subject: RE: [mapguide-internals] MapGuide RFC 65 - ACE 5.6 upgrade is ready for review

As part of the upgrade we may want to switch ACE on Windows to use either ACE_TP_Reactor or ACE_Select_Reactor instead of the ACE_WFMO_Reactor.  The WFMO reactor can only handle 62 concurrent connections whereas the TP and Select reactors can handle 1024 connections.
_______________________________________________
mapguide-internals mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/mapguide-internals
_______________________________________________
mapguide-internals mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/mapguide-internals
_______________________________________________
mapguide-internals mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/mapguide-internals

[Bruce Dechant]

Trevor,

That is outside the scope of the RFC - switching to a different Reactor would be an entirely different RFC.

Thanks,
Bruce


-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Trevor Wekel
Sent: Thursday, June 04, 2009 1:40 PM
To: MapGuide Internals Mail List
Subject: RE: [mapguide-internals] MapGuide RFC 65 - ACE 5.6 upgrade is ready for review

Hi Bruce,

As part of the upgrade we may want to switch ACE on Windows to use either ACE_TP_Reactor or ACE_Select_Reactor instead of the ACE_WFMO_Reactor.  The WFMO reactor can only handle 62 concurrent connections whereas the TP and Select reactors can handle 1024 connections.

Thanks,
Trevor  


-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Bruce Dechant
Sent: Thursday, June 04, 2009 10:42 AM
To: MapGuide Internals Mail List
Subject: [mapguide-internals] MapGuide RFC 65 - ACE 5.6 upgrade is ready for review

Hi All,



Please review MapGuide RFC 65 - ACE 5.6 upgrade.

http://trac.osgeo.org/mapguide/wiki/MapGuideRfc65



Post any comments or questions to the mailing list.



Thanks,

Bruce


_______________________________________________
mapguide-internals mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/mapguide-internals
_______________________________________________
mapguide-internals mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/mapguide-internals
_______________________________________________
mapguide-internals mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/mapguide-internals

[Trevor Wekel-3]

Big giant locks would certainly affect multi-core scalability.  I know Bruce did a whole ton of work on the Fdo connection management in MapGuide to improve Gdal threading issues.  I don't know if the big giant lock went it.

Thanks,
Trevor  

-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Traian Stanev
Sent: Thursday, June 04, 2009 2:15 PM
To: MapGuide Internals Mail List
Subject: RE: [mapguide-internals] MapGuide RFC 65 - ACE 5.6 upgrade is ready for review


While this is true, don't we also have the big giant lock in the code, which makes sure that if you have rasters for example (where I/O matters), you will only be working on one request at a time anyway?


Traian


-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Trevor Wekel
Sent: Thursday, June 04, 2009 4:10 PM
To: MapGuide Internals Mail List
Subject: RE: [mapguide-internals] MapGuide RFC 65 - ACE 5.6 upgrade is ready for review

Yes.  Intel Nehalems are 4 core w/ hyperthreading and AMD Istanbuls are 6 core chips.  Both of these are available in two socket configuration and the AMD chips are also available in 4 and 8 socket configurations.  This brings the total concurrent "core" count to 12, 16, 24, or 48 depending on configuration.

If we assume a 24 "core" variant then the current WMFO reactor will only be able to supply two concurrent threads of work per core.  Depending on IO constraints, this may not be enough to fully utilize all of the CPU on these newer boxes.


Thanks,
Trevor


-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Jason Birch
Sent: Thursday, June 04, 2009 1:52 PM
To: MapGuide Internals Mail List
Subject: RE: [mapguide-internals] MapGuide RFC 65 - ACE 5.6 upgrade is ready for review

That would be nice.  

Although the recent http connection fixes seem to have worked around the problems fairly well and reduced the number of hard failures for end users, I've noticed that in some cases a user's performance degrades considerably (hangs for N seconds) while waiting for connections that are inactive but still open to be closed properly.

It's also my understanding (from previous conversations) that some newer hardware can actually handle more than 62 connections worth of work, and that the current limitation would be artificially constraining the performance of these boxes.
 
Jason

-----Original Message-----
From: Trevor Wekel
Sent: Thursday, June 04, 2009 12:40 PM
To: MapGuide Internals Mail List
Subject: RE: [mapguide-internals] MapGuide RFC 65 - ACE 5.6 upgrade is ready for review

As part of the upgrade we may want to switch ACE on Windows to use either ACE_TP_Reactor or ACE_Select_Reactor instead of the ACE_WFMO_Reactor.  The WFMO reactor can only handle 62 concurrent connections whereas the TP and Select reactors can handle 1024 connections.
_______________________________________________
mapguide-internals mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/mapguide-internals
_______________________________________________
mapguide-internals mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/mapguide-internals
_______________________________________________
mapguide-internals mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/mapguide-internals
_______________________________________________
mapguide-internals mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/mapguide-internals

[Jason Birch]

Looks like one of the locks is still there:

http://trac.osgeo.org/mapguide/browser/branches/2.1/MgDev/Server/src/Services/Mapping/MappingUtil.cpp#L675

Was there something else in there limiting the raster provider to a single connection?

Jason

-----Original Message-----
From: Trevor Wekel
Sent: Thursday, June 04, 2009 1:35 PM
To: MapGuide Internals Mail List
Subject: RE: [mapguide-internals] MapGuide RFC 65 - ACE 5.6 upgrade is ready for review

Big giant locks would certainly affect multi-core scalability.  I know Bruce did a whole ton of work on the Fdo connection management in MapGuide to improve Gdal threading issues.  I don't know if the big giant lock went it.

-----Original Message-----
From: Traian Stanev
Sent: Thursday, June 04, 2009 2:15 PM
To: MapGuide Internals Mail List
Subject: RE: [mapguide-internals] MapGuide RFC 65 - ACE 5.6 upgrade is ready for review

While this is true, don't we also have the big giant lock in the code, which makes sure that if you have rasters for example (where I/O matters), you will only be working on one request at a time anyway?
_______________________________________________
mapguide-internals mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/mapguide-internals

[Trevor Wekel-3]

Good catch Jason.  The ACE_GUARD will make all raster processing single threaded.  I believe Gdal and WMS would fall into this category.  Serverconfig.ini settings can also affect the number of concurrent connections.

Thanks,
Trevor

-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Jason Birch
Sent: Thursday, June 04, 2009 2:45 PM
To: MapGuide Internals Mail List
Subject: RE: [mapguide-internals] MapGuide RFC 65 - ACE 5.6 upgrade is ready for review

Looks like one of the locks is still there:

http://trac.osgeo.org/mapguide/browser/branches/2.1/MgDev/Server/src/Services/Mapping/MappingUtil.cpp#L675

Was there something else in there limiting the raster provider to a single connection?

Jason

-----Original Message-----
From: Trevor Wekel
Sent: Thursday, June 04, 2009 1:35 PM
To: MapGuide Internals Mail List
Subject: RE: [mapguide-internals] MapGuide RFC 65 - ACE 5.6 upgrade is ready for review

Big giant locks would certainly affect multi-core scalability.  I know Bruce did a whole ton of work on the Fdo connection management in MapGuide to improve Gdal threading issues.  I don't know if the big giant lock went it.

-----Original Message-----
From: Traian Stanev
Sent: Thursday, June 04, 2009 2:15 PM
To: MapGuide Internals Mail List
Subject: RE: [mapguide-internals] MapGuide RFC 65 - ACE 5.6 upgrade is ready for review

While this is true, don't we also have the big giant lock in the code, which makes sure that if you have rasters for example (where I/O matters), you will only be working on one request at a time anyway?
_______________________________________________
mapguide-internals mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/mapguide-internals
_______________________________________________
mapguide-internals mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/mapguide-internals

[Traian Stanev]

It is possible to run without the lock (and also without a similar lock that exists on the other side of the GDAL provider), provided that a fix is made to the GDAL provider refcounting. This was held back for the 2.1 release due to concerns by Haris and Jason (and others) about removing the lock being risky. There was a long thread about this a month or so ago.

Traian


________________________________________
From: [hidden email] [[hidden email]] On Behalf Of Trevor Wekel
Sent: Thursday, June 04, 2009 4:49 PM
To: MapGuide Internals Mail List
Subject: RE: [mapguide-internals] MapGuide RFC 65 - ACE 5.6 upgrade is ready    for     review

Good catch Jason.  The ACE_GUARD will make all raster processing single threaded.  I believe Gdal and WMS would fall into this category.  Serverconfig.ini settings can also affect the number of concurrent connections.

Thanks,
Trevor

-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Jason Birch
Sent: Thursday, June 04, 2009 2:45 PM
To: MapGuide Internals Mail List
Subject: RE: [mapguide-internals] MapGuide RFC 65 - ACE 5.6 upgrade is ready for review

Looks like one of the locks is still there:

http://trac.osgeo.org/mapguide/browser/branches/2.1/MgDev/Server/src/Services/Mapping/MappingUtil.cpp#L675

Was there something else in there limiting the raster provider to a single connection?

Jason

-----Original Message-----
From: Trevor Wekel
Sent: Thursday, June 04, 2009 1:35 PM
To: MapGuide Internals Mail List
Subject: RE: [mapguide-internals] MapGuide RFC 65 - ACE 5.6 upgrade is ready for review

Big giant locks would certainly affect multi-core scalability.  I know Bruce did a whole ton of work on the Fdo connection management in MapGuide to improve Gdal threading issues.  I don't know if the big giant lock went it.

-----Original Message-----
From: Traian Stanev
Sent: Thursday, June 04, 2009 2:15 PM
To: MapGuide Internals Mail List
Subject: RE: [mapguide-internals] MapGuide RFC 65 - ACE 5.6 upgrade is ready for review

While this is true, don't we also have the big giant lock in the code, which makes sure that if you have rasters for example (where I/O matters), you will only be working on one request at a time anyway?
_______________________________________________
mapguide-internals mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/mapguide-internals
_______________________________________________
mapguide-internals mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/mapguide-internals
_______________________________________________
mapguide-internals mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/mapguide-internals

[Jason Birch]

Yeah, I know, I'm whiny and conservative. :)  

I don't think I was as concerned about the risk of removing the lock as I was that it would have masked an underlying deficiency in the MapGuide code, which was only emerged by the GDAL provider because it was single-threaded.  I believe that the other providers had the same problems, just with less regularity.  I also believe that Bruce's work addressed these concerns, and made MapGuide non-reliant on refcounting from the FDO providers.

If that's the case, I wouldn't have a problem with removing the MapGuide lock, which I think would then open the way to an experimental threaded GDAL provider being distributed for testing without changes to the MapGuide code.  

I seem to recall that there was some other logic in MapGuide that explicitly set the raster provider to only use a single pool, but can't track it down.

Jason

-----Original Message-----
From: Traian Stanev
Sent: Thursday, June 04, 2009 4:28 PM
To: MapGuide Internals Mail List
Subject: RE: [mapguide-internals] MapGuide RFC 65 - ACE 5.6 upgrade is ready for review


It is possible to run without the lock (and also without a similar lock that exists on the other side of the GDAL provider), provided that a fix is made to the GDAL provider refcounting. This was held back for the 2.1 release due to concerns by Haris and Jason (and others) about removing the lock being risky. There was a long thread about this a month or so ago.
_______________________________________________
mapguide-internals mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/mapguide-internals

[Traian Stanev]

About the last issue -- GDAL provider being single-threaded, it is set in the serverconfig.ini.

Simply removing the lock would not have "masked" any problem. Removing the lock AND fixing the refcounting problem in the GDAL provider itself would have made the connection pooling problems in MapGuide go away.

The refcounting problem in the provider needs to be fixed no matter what. I can check in that fix, and remove the global GDAL lock, if you guys no longer mind. The global GDAL lock is unnecessary due to there being the global raster lock in MapGuide as well.

Traian

________________________________________
From: [hidden email] [[hidden email]] On Behalf Of Jason Birch [[hidden email]]
Sent: Thursday, June 04, 2009 7:43 PM
To: MapGuide Internals Mail List
Subject: RE: [mapguide-internals] MapGuide RFC 65 - ACE 5.6 upgrade is ready    for     review

Yeah, I know, I'm whiny and conservative. :)

I don't think I was as concerned about the risk of removing the lock as I was that it would have masked an underlying deficiency in the MapGuide code, which was only emerged by the GDAL provider because it was single-threaded.  I believe that the other providers had the same problems, just with less regularity.  I also believe that Bruce's work addressed these concerns, and made MapGuide non-reliant on refcounting from the FDO providers.

If that's the case, I wouldn't have a problem with removing the MapGuide lock, which I think would then open the way to an experimental threaded GDAL provider being distributed for testing without changes to the MapGuide code.

I seem to recall that there was some other logic in MapGuide that explicitly set the raster provider to only use a single pool, but can't track it down.

Jason

-----Original Message-----
From: Traian Stanev
Sent: Thursday, June 04, 2009 4:28 PM
To: MapGuide Internals Mail List
Subject: RE: [mapguide-internals] MapGuide RFC 65 - ACE 5.6 upgrade is ready for review


It is possible to run without the lock (and also without a similar lock that exists on the other side of the GDAL provider), provided that a fix is made to the GDAL provider refcounting. This was held back for the 2.1 release due to concerns by Haris and Jason (and others) about removing the lock being risky. There was a long thread about this a month or so ago.
_______________________________________________
mapguide-internals mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/mapguide-internals
_______________________________________________
mapguide-internals mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/mapguide-internals

[UV-2]

somehow I see quite some analogy to a previous thread regarding having
different thread pools for different purposes.
There are obviously different degrees of parallelization for different
request types
and maybe we should discuss how to reflect that in the reaktor and
thread pool architecture to maintain a truely scalable system.

I think its clear by now that there is room for improvement regarding
parallelization.
What about introducing additional priorization attributes and
prioritized request queues.
This could go along the introduction of a new Reactor type.
any thoughts?

Bruce Dechant wrote:

> Trevor,
>
> That is outside the scope of the RFC - switching to a different Reactor would be an entirely different RFC.
>
> Thanks,
> Bruce
>
>
> -----Original Message-----
> From: [hidden email] [mailto:[hidden email]] On Behalf Of Trevor Wekel
> Sent: Thursday, June 04, 2009 1:40 PM
> To: MapGuide Internals Mail List
> Subject: RE: [mapguide-internals] MapGuide RFC 65 - ACE 5.6 upgrade is ready for review
>
> Hi Bruce,
>
> As part of the upgrade we may want to switch ACE on Windows to use either ACE_TP_Reactor or ACE_Select_Reactor instead of the ACE_WFMO_Reactor.  The WFMO reactor can only handle 62 concurrent connections whereas the TP and Select reactors can handle 1024 connections.
>
> Thanks,
> Trevor  
>
>
> -----Original Message-----
> From: [hidden email] [mailto:[hidden email]] On Behalf Of Bruce Dechant
> Sent: Thursday, June 04, 2009 10:42 AM
> To: MapGuide Internals Mail List
> Subject: [mapguide-internals] MapGuide RFC 65 - ACE 5.6 upgrade is ready for review
>
> Hi All,
>
>
>
> Please review MapGuide RFC 65 - ACE 5.6 upgrade.
>
> http://trac.osgeo.org/mapguide/wiki/MapGuideRfc65
>
>
>
> Post any comments or questions to the mailing list.
>
>
>
> Thanks,
>
> Bruce
>
>
> _______________________________________________
> mapguide-internals mailing list
> [hidden email]
> http://lists.osgeo.org/mailman/listinfo/mapguide-internals
> _______________________________________________
> mapguide-internals mailing list
> [hidden email]
> http://lists.osgeo.org/mailman/listinfo/mapguide-internals
> _______________________________________________
> mapguide-internals mailing list
> [hidden email]
> http://lists.osgeo.org/mailman/listinfo/mapguide-internals
>
>  

_______________________________________________
mapguide-internals mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/mapguide-internals