OSGeo.org › OSGeo Software and Data Projects › Feature Data Objects (FDO) › FDO Internals

New RFC 30 Posted

22 messages

1 2

Brent Robinson

New RFC 30 Posted

Reply Threaded

Some javascript/style in this post has been disabled (why?)

Hi,

RFC 30 (http://trac.osgeo.org/fdo/wiki/FDORfc30) is now ready for review. The main purpose is add FDO API functions to encapsulate conversions between different FDO data types. Please review and let me know if you have any questions or comments.

If approved, this one will likely be implemented 2 versions after FDO 3.3.

Thanks,

Brent.

_______________________________________________
fdo-internals mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/fdo-internals

Haris Kurtagic

RE: New RFC 30 Posted

Reply Threaded

Some javascript/style in this post has been disabled (why?)

Hi Brent,

not directly related to this RFC but still it is important issue regarding FDO data values.

I find a problem working with FDO Readers and commands is that I need to get value as basic type from reader and then convert to FdoDataValue etc...

I would like to be able to use data values and properties with values from readers directly and make those as input value for another FDO command.

Right now I need to check data type returned from property get that data as native convert back to fdo data value and put as input to FDO command.

Haris

From: [hidden email] [mailto:[hidden email]] On Behalf Of Brent Robinson
Sent: Friday, November 21, 2008 7:32 PM
To: FDO Internals Mail List
Subject: [fdo-internals] New RFC 30 Posted

Hi,

If approved, this one will likely be implemented 2 versions after FDO 3.3.

Thanks,

Brent.

_______________________________________________
fdo-internals mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/fdo-internals

Traian Stanev

RE: New RFC 30 Posted

Reply Threaded

Some javascript/style in this post has been disabled (why?)

An alternative to this suggestion is to make the destination command accept an FdoIFeatureReader (or something equivalent that provides access to property values in a procedural way) as argument, rather than a collection of PropertyValues. I was thinking of this when I was implementing the SDF->SQLite converter, which requires two translations from PropertyValue to raw data – once when initializing the inputs of the insert command and a second time inside the destination provider when matching the input property values to the destination schema.

Traian

From: [hidden email] [mailto:[hidden email]] On Behalf Of Haris Kurtagic
Sent: Monday, November 24, 2008 8:27 AM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] New RFC 30 Posted

Hi Brent,

not directly related to this RFC but still it is important issue regarding FDO data values.

I find a problem working with FDO Readers and commands is that I need to get value as basic type from reader and then convert to FdoDataValue etc...

I would like to be able to use data values and properties with values from readers directly and make those as input value for another FDO command.

Right now I need to check data type returned from property get that data as native convert back to fdo data value and put as input to FDO command.

Haris

Hi,

If approved, this one will likely be implemented 2 versions after FDO 3.3.

Thanks,

Brent.

_______________________________________________
fdo-internals mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/fdo-internals

Haris Kurtagic

RE: New RFC 30 Posted

Reply Threaded

Some javascript/style in this post has been disabled (why?)

Right now from Reader we can’t get PropertyValues only raw data.

To get Property value and property value collection which then could be passed to another command would be of great help and remove some of unnecessary data conversions.

Haris

From: [hidden email] [mailto:[hidden email]] On Behalf Of Traian Stanev
Sent: Monday, November 24, 2008 7:39 PM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] New RFC 30 Posted

Traian

Hi Brent,

not directly related to this RFC but still it is important issue regarding FDO data values.

I find a problem working with FDO Readers and commands is that I need to get value as basic type from reader and then convert to FdoDataValue etc...

I would like to be able to use data values and properties with values from readers directly and make those as input value for another FDO command.

Right now I need to check data type returned from property get that data as native convert back to fdo data value and put as input to FDO command.

Haris

Hi,

If approved, this one will likely be implemented 2 versions after FDO 3.3.

Thanks,

Brent.

_______________________________________________
fdo-internals mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/fdo-internals

Traian Stanev

RE: New RFC 30 Posted

Reply Threaded

Some javascript/style in this post has been disabled (why?)

Yes, but creating PropertyValues would be a performance overhead. That’s why I’m suggesting the alternative approach of allowing commands to pull the feature’s data directly from a FeatureReader (or equivalent).

Traian

From: [hidden email] [mailto:[hidden email]] On Behalf Of Haris Kurtagic
Sent: Monday, November 24, 2008 2:25 PM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] New RFC 30 Posted

Right now from Reader we can’t get PropertyValues only raw data.

To get Property value and property value collection which then could be passed to another command would be of great help and remove some of unnecessary data conversions.

Haris

Traian

Hi Brent,

not directly related to this RFC but still it is important issue regarding FDO data values.

I find a problem working with FDO Readers and commands is that I need to get value as basic type from reader and then convert to FdoDataValue etc...

I would like to be able to use data values and properties with values from readers directly and make those as input value for another FDO command.

Right now I need to check data type returned from property get that data as native convert back to fdo data value and put as input to FDO command.

Haris

Hi,

If approved, this one will likely be implemented 2 versions after FDO 3.3.

Thanks,

Brent.

_______________________________________________
fdo-internals mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/fdo-internals

Haris Kurtagic

RE: New RFC 30 Posted

Reply Threaded

Some javascript/style in this post has been disabled (why?)

Yes I understood that. I am afraid that it can be dangerous to use readers as input to another command.

One of thing would be that reader would not be closed before another command would be executed.

Not necessarily but I am worried that it could create problems with current readers/commands.

If we look further as you suggested to improve performance of readers, than it would be better to have them to returned shared pointers to data or to be able to set data buffers to be used by readers.

But even before that I would prefer to have index access to columns of readers as we speak of performance, we discussed it few times :)

Haris

From: [hidden email] [mailto:[hidden email]] On Behalf Of Traian Stanev
Sent: Monday, November 24, 2008 8:41 PM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] New RFC 30 Posted

Traian

Right now from Reader we can’t get PropertyValues only raw data.

To get Property value and property value collection which then could be passed to another command would be of great help and remove some of unnecessary data conversions.

Haris

Traian

Hi Brent,

not directly related to this RFC but still it is important issue regarding FDO data values.

I find a problem working with FDO Readers and commands is that I need to get value as basic type from reader and then convert to FdoDataValue etc...

I would like to be able to use data values and properties with values from readers directly and make those as input value for another FDO command.

Right now I need to check data type returned from property get that data as native convert back to fdo data value and put as input to FDO command.

Haris

Hi,

If approved, this one will likely be implemented 2 versions after FDO 3.3.

Thanks,

Brent.

_______________________________________________
fdo-internals mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/fdo-internals

Traian Stanev

RE: New RFC 30 Posted

Reply Threaded

Some javascript/style in this post has been disabled (why?)

Yes that’s a problem and that’s why I said “FdoFeatureReader (or equivalent)”. What I meant by “equivalent” was something that has the almost the same API as FdoIFeatureReader, without the ReadNext/Close functions. Essentially it would represent a single row of data, with an API that allows fast access in a way that is the same as reading features coming from a query.

Traian

From: [hidden email] [mailto:[hidden email]] On Behalf Of Haris Kurtagic
Sent: Monday, November 24, 2008 3:21 PM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] New RFC 30 Posted

Yes I understood that. I am afraid that it can be dangerous to use readers as input to another command.

One of thing would be that reader would not be closed before another command would be executed.

Not necessarily but I am worried that it could create problems with current readers/commands.

But even before that I would prefer to have index access to columns of readers as we speak of performance, we discussed it few times :)

Haris

Traian

Right now from Reader we can’t get PropertyValues only raw data.

To get Property value and property value collection which then could be passed to another command would be of great help and remove some of unnecessary data conversions.

Haris

Traian

Hi Brent,

not directly related to this RFC but still it is important issue regarding FDO data values.

I find a problem working with FDO Readers and commands is that I need to get value as basic type from reader and then convert to FdoDataValue etc...

I would like to be able to use data values and properties with values from readers directly and make those as input value for another FDO command.

Right now I need to check data type returned from property get that data as native convert back to fdo data value and put as input to FDO command.

Haris

Hi,

If approved, this one will likely be implemented 2 versions after FDO 3.3.

Thanks,

Brent.

_______________________________________________
fdo-internals mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/fdo-internals

Haris Kurtagic

RE: New RFC 30 Posted

Reply Threaded

Some javascript/style in this post has been disabled (why?)

I am affraid I don’t understand now. That equivalent of reader would be returned by command (than it is new reader) or it would be created from returned current Reader ?

Haris

From: [hidden email] [mailto:[hidden email]] On Behalf Of Traian Stanev
Sent: Monday, November 24, 2008 9:01 PM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] New RFC 30 Posted

Traian

Yes I understood that. I am afraid that it can be dangerous to use readers as input to another command.

One of thing would be that reader would not be closed before another command would be executed.

Not necessarily but I am worried that it could create problems with current readers/commands.

But even before that I would prefer to have index access to columns of readers as we speak of performance, we discussed it few times :)

Haris

Traian

Right now from Reader we can’t get PropertyValues only raw data.

To get Property value and property value collection which then could be passed to another command would be of great help and remove some of unnecessary data conversions.

Haris

Traian

Hi Brent,

not directly related to this RFC but still it is important issue regarding FDO data values.

I find a problem working with FDO Readers and commands is that I need to get value as basic type from reader and then convert to FdoDataValue etc...

I would like to be able to use data values and properties with values from readers directly and make those as input value for another FDO command.

Right now I need to check data type returned from property get that data as native convert back to fdo data value and put as input to FDO command.

Haris

Hi,

If approved, this one will likely be implemented 2 versions after FDO 3.3.

Thanks,

Brent.

_______________________________________________
fdo-internals mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/fdo-internals

Traian Stanev

RE: New RFC 30 Posted

Reply Threaded

Some javascript/style in this post has been disabled (why?)

Think of it as a data row – a feature reader lets you iterate through many such data rows via ReadNext(). It is also very similar to OGRFeature for example (http://www.gdal.org/ogr/classOGRFeature.html).

So, for example say you want to transfer a feature from one connection to another – you would do a select from one connection, and that will return a FeatureReader. You can wrap that in the data row object, which has essentially the same API as the feature reader, except that it does not allow ReadNext(). Then you can pass that wrapper to an FdoInsert which will take the data row and insert it into the destination connection.

Internally you could also wrap a PropertyValue collection in such a data row object – no matter whether it is backed by a FeatureReader or a PropertyValue collection, it would offer the same API. Also, you can have the data row convert all its data into a PropertyValueCollection which you can use in case you don’t need performance, thus enabling interop with existing code, which uses such PropertyValue collections.

You can also for example define this object so that it can do automatic type conversion for you – say you call GetInt32 on an Int16 property – it can automatically do the cast for you. Which would provide equivalent functionality to RFC 30 by the way, or it can use whatever code RFC30 adds to FDO. J

Traian

From: [hidden email] [mailto:[hidden email]] On Behalf Of Haris Kurtagic
Sent: Monday, November 24, 2008 3:36 PM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] New RFC 30 Posted

I am affraid I don’t understand now. That equivalent of reader would be returned by command (than it is new reader) or it would be created from returned current Reader ?

Haris

Traian

Yes I understood that. I am afraid that it can be dangerous to use readers as input to another command.

One of thing would be that reader would not be closed before another command would be executed.

Not necessarily but I am worried that it could create problems with current readers/commands.

But even before that I would prefer to have index access to columns of readers as we speak of performance, we discussed it few times :)

Haris

Traian

Right now from Reader we can’t get PropertyValues only raw data.

To get Property value and property value collection which then could be passed to another command would be of great help and remove some of unnecessary data conversions.

Haris

Traian

Hi Brent,

not directly related to this RFC but still it is important issue regarding FDO data values.

I find a problem working with FDO Readers and commands is that I need to get value as basic type from reader and then convert to FdoDataValue etc...

I would like to be able to use data values and properties with values from readers directly and make those as input value for another FDO command.

Right now I need to check data type returned from property get that data as native convert back to fdo data value and put as input to FDO command.

Haris

Hi,

If approved, this one will likely be implemented 2 versions after FDO 3.3.

Thanks,

Brent.

_______________________________________________
fdo-internals mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/fdo-internals

Haris Kurtagic

RE: New RFC 30 Posted

Reply Threaded

Some javascript/style in this post has been disabled (why?)

I guess there is lot of discus on that topic.

You are proposing an wrapper over feature reader which would expose data of reader ?

Than that wrapper would need to read rows (or one row ) from reader and copy data to new data buffer before it can be used in another command?

If it would fetch data to new data buffers than it is same performance problem of copying data. Same performance as to copy data to property values as we do already eventually if we want to do something else with that data.

Data type conversions can be implemented on existing reader too.

What I see as way to improve performance is to be able to set data buffers for select command so returned reader would used those buffers. So this same data without copying can be reused.

Haris

From: [hidden email] [mailto:[hidden email]] On Behalf Of Traian Stanev
Sent: Monday, November 24, 2008 9:34 PM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] New RFC 30 Posted

Traian

I am affraid I don’t understand now. That equivalent of reader would be returned by command (than it is new reader) or it would be created from returned current Reader ?

Haris

Traian

Yes I understood that. I am afraid that it can be dangerous to use readers as input to another command.

One of thing would be that reader would not be closed before another command would be executed.

Not necessarily but I am worried that it could create problems with current readers/commands.

But even before that I would prefer to have index access to columns of readers as we speak of performance, we discussed it few times :)

Haris

Traian

Right now from Reader we can’t get PropertyValues only raw data.

To get Property value and property value collection which then could be passed to another command would be of great help and remove some of unnecessary data conversions.

Haris

Traian

Hi Brent,

not directly related to this RFC but still it is important issue regarding FDO data values.

I find a problem working with FDO Readers and commands is that I need to get value as basic type from reader and then convert to FdoDataValue etc...

I would like to be able to use data values and properties with values from readers directly and make those as input value for another FDO command.

Right now I need to check data type returned from property get that data as native convert back to fdo data value and put as input to FDO command.

Haris

Hi,

If approved, this one will likely be implemented 2 versions after FDO 3.3.

Thanks,

Brent.

_______________________________________________
fdo-internals mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/fdo-internals

Traian Stanev

RE: New RFC 30 Posted

Reply Threaded

Some javascript/style in this post has been disabled (why?)

Yes, a wrapper around either FdoFeatureReader or a PropertyValueCollection – same API for both. However, it would not necessarily copy the data from the FeatureReader – in the case where it wraps a FeatureReader it could simply do a passthrough, for example:

virtual FdoString* FdoFRFeature::GetString(FdoString* propName) { return m_reader->GetString(propName); }

Where m_reader is a private member variable pointing to the FeatureReader.

In the case where it wraps a PropertyValueCollection, it would be *roughly*:

virtual FdoString* FdoPVCFeature::GetString(FdoString* propName) { return ((FdoStringValue*)m_propValCol->FindItem(propName))->GetString(); }

The actual implementation is not the point though, as long as the outfacing API allows for implementing it efficiently. I don’t think an API which uses PropertyValue objects has the maximum potential for efficiency because you would be incurring at least two virtual calls to refcounting functions on top of already expensive virtual calls to get the value.

Traian

From: [hidden email] [mailto:[hidden email]] On Behalf Of Haris Kurtagic
Sent: Monday, November 24, 2008 4:19 PM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] New RFC 30 Posted

I guess there is lot of discus on that topic.

You are proposing an wrapper over feature reader which would expose data of reader ?

Than that wrapper would need to read rows (or one row ) from reader and copy data to new data buffer before it can be used in another command?

Data type conversions can be implemented on existing reader too.

What I see as way to improve performance is to be able to set data buffers for select command so returned reader would used those buffers. So this same data without copying can be reused.

Haris

Traian

I am affraid I don’t understand now. That equivalent of reader would be returned by command (than it is new reader) or it would be created from returned current Reader ?

Haris

Traian

Yes I understood that. I am afraid that it can be dangerous to use readers as input to another command.

One of thing would be that reader would not be closed before another command would be executed.

Not necessarily but I am worried that it could create problems with current readers/commands.

But even before that I would prefer to have index access to columns of readers as we speak of performance, we discussed it few times :)

Haris

Traian

Right now from Reader we can’t get PropertyValues only raw data.

To get Property value and property value collection which then could be passed to another command would be of great help and remove some of unnecessary data conversions.

Haris

Traian

Hi Brent,

not directly related to this RFC but still it is important issue regarding FDO data values.

I find a problem working with FDO Readers and commands is that I need to get value as basic type from reader and then convert to FdoDataValue etc...

I would like to be able to use data values and properties with values from readers directly and make those as input value for another FDO command.

Right now I need to check data type returned from property get that data as native convert back to fdo data value and put as input to FDO command.

Haris

Hi,

If approved, this one will likely be implemented 2 versions after FDO 3.3.

Thanks,

Brent.

_______________________________________________
fdo-internals mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/fdo-internals

Haris Kurtagic

RE: New RFC 30 Posted

Reply Threaded

Some javascript/style in this post has been disabled (why?)

If that wrapper would return same data buffer than we are into same problems as if we would send reader to another command.

Feature reader data buffer would be corrupted on next call and that wrapper in another command would give wrong result.

When I was talking about feature reader returning fdo data value, was primarily to avoid what we do now: switch(data type), we create new fdo data type , copy raw data to new fdo data value and pass that data to another command.

Intention was to reduce that switch and data copy code in every fdo application.

I think to improve performance of reader, would be that reader will be able to fetch data in preallocated shared data buffers.

Haris

From: [hidden email] [mailto:[hidden email]] On Behalf Of Traian Stanev
Sent: Monday, November 24, 2008 10:15 PM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] New RFC 30 Posted

virtual FdoString* FdoFRFeature::GetString(FdoString* propName) { return m_reader->GetString(propName); }

Where m_reader is a private member variable pointing to the FeatureReader.

In the case where it wraps a PropertyValueCollection, it would be *roughly*:

virtual FdoString* FdoPVCFeature::GetString(FdoString* propName) { return ((FdoStringValue*)m_propValCol->FindItem(propName))->GetString(); }

Traian

I guess there is lot of discus on that topic.

You are proposing an wrapper over feature reader which would expose data of reader ?

Than that wrapper would need to read rows (or one row ) from reader and copy data to new data buffer before it can be used in another command?

Data type conversions can be implemented on existing reader too.

What I see as way to improve performance is to be able to set data buffers for select command so returned reader would used those buffers. So this same data without copying can be reused.

Haris

Traian

I am affraid I don’t understand now. That equivalent of reader would be returned by command (than it is new reader) or it would be created from returned current Reader ?

Haris

Traian

Yes I understood that. I am afraid that it can be dangerous to use readers as input to another command.

One of thing would be that reader would not be closed before another command would be executed.

Not necessarily but I am worried that it could create problems with current readers/commands.

But even before that I would prefer to have index access to columns of readers as we speak of performance, we discussed it few times :)

Haris

Traian

Right now from Reader we can’t get PropertyValues only raw data.

To get Property value and property value collection which then could be passed to another command would be of great help and remove some of unnecessary data conversions.

Haris

Traian

Hi Brent,

not directly related to this RFC but still it is important issue regarding FDO data values.

I find a problem working with FDO Readers and commands is that I need to get value as basic type from reader and then convert to FdoDataValue etc...

I would like to be able to use data values and properties with values from readers directly and make those as input value for another FDO command.

Right now I need to check data type returned from property get that data as native convert back to fdo data value and put as input to FDO command.

Haris

Hi,

If approved, this one will likely be implemented 2 versions after FDO 3.3.

Thanks,

Brent.

_______________________________________________
fdo-internals mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/fdo-internals

Traian Stanev

RE: New RFC 30 Posted

Reply Threaded

Some javascript/style in this post has been disabled (why?)

Yes, such corruption is a possible problem. Hoewever, if your scheme is based on cached (pre-created) PropertyValue objects, then it has the same problem – someone can hold on to the PropertyValue object while it changes from under them. If you want to fix that, you’d have to do some really ugly refcounting hacks.

Besides, there is precedent for allowing corruptible memory – one of the GetGeometry calls in FdoFeatureReader is only guaranteed to hold good data until the next ReadNext call, and callers must be aware of that. This allows providers like SDF and SQLite for example to return pointers directly to database memory, thus avoiding doing even a single copy of the data. This is also how the GetString() calls works.

Traian

From: [hidden email] [mailto:[hidden email]] On Behalf Of Haris Kurtagic
Sent: Monday, November 24, 2008 5:15 PM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] New RFC 30 Posted

If that wrapper would return same data buffer than we are into same problems as if we would send reader to another command.

Feature reader data buffer would be corrupted on next call and that wrapper in another command would give wrong result.

Intention was to reduce that switch and data copy code in every fdo application.

I think to improve performance of reader, would be that reader will be able to fetch data in preallocated shared data buffers.

Haris

virtual FdoString* FdoFRFeature::GetString(FdoString* propName) { return m_reader->GetString(propName); }

Where m_reader is a private member variable pointing to the FeatureReader.

In the case where it wraps a PropertyValueCollection, it would be *roughly*:

virtual FdoString* FdoPVCFeature::GetString(FdoString* propName) { return ((FdoStringValue*)m_propValCol->FindItem(propName))->GetString(); }

Traian

I guess there is lot of discus on that topic.

You are proposing an wrapper over feature reader which would expose data of reader ?

Than that wrapper would need to read rows (or one row ) from reader and copy data to new data buffer before it can be used in another command?

Data type conversions can be implemented on existing reader too.

What I see as way to improve performance is to be able to set data buffers for select command so returned reader would used those buffers. So this same data without copying can be reused.

Haris

Traian

I am affraid I don’t understand now. That equivalent of reader would be returned by command (than it is new reader) or it would be created from returned current Reader ?

Haris

Traian

Yes I understood that. I am afraid that it can be dangerous to use readers as input to another command.

One of thing would be that reader would not be closed before another command would be executed.

Not necessarily but I am worried that it could create problems with current readers/commands.

But even before that I would prefer to have index access to columns of readers as we speak of performance, we discussed it few times :)

Haris

Traian

Right now from Reader we can’t get PropertyValues only raw data.

To get Property value and property value collection which then could be passed to another command would be of great help and remove some of unnecessary data conversions.

Haris

Traian

Hi Brent,

not directly related to this RFC but still it is important issue regarding FDO data values.

I find a problem working with FDO Readers and commands is that I need to get value as basic type from reader and then convert to FdoDataValue etc...

I would like to be able to use data values and properties with values from readers directly and make those as input value for another FDO command.

Right now I need to check data type returned from property get that data as native convert back to fdo data value and put as input to FDO command.

Haris

Hi,

If approved, this one will likely be implemented 2 versions after FDO 3.3.

Thanks,

Brent.

_______________________________________________
fdo-internals mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/fdo-internals

Haris Kurtagic

RE: New RFC 30 Posted

Reply Threaded

Some javascript/style in this post has been disabled (why?)

Still even with SDF fdo clients are calling GetGeometry which returns FdoByteArray * which allocates new memory buffer and data is copied into new buffer.

Problem is not that data is copied once but that it is copied multiple times (not case with FdoByteArray ) but with other data types.

With preallocated shared data buffers, data returned by reader can be reused from same buffers without copying data even if reader is closed and destroyed.

Haris

From: [hidden email] [mailto:[hidden email]] On Behalf Of Traian Stanev
Sent: Monday, November 24, 2008 11:15 PM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] New RFC 30 Posted

Traian

If that wrapper would return same data buffer than we are into same problems as if we would send reader to another command.

Feature reader data buffer would be corrupted on next call and that wrapper in another command would give wrong result.

Intention was to reduce that switch and data copy code in every fdo application.

I think to improve performance of reader, would be that reader will be able to fetch data in preallocated shared data buffers.

Haris

virtual FdoString* FdoFRFeature::GetString(FdoString* propName) { return m_reader->GetString(propName); }

Where m_reader is a private member variable pointing to the FeatureReader.

In the case where it wraps a PropertyValueCollection, it would be *roughly*:

virtual FdoString* FdoPVCFeature::GetString(FdoString* propName) { return ((FdoStringValue*)m_propValCol->FindItem(propName))->GetString(); }

Traian

I guess there is lot of discus on that topic.

You are proposing an wrapper over feature reader which would expose data of reader ?

Than that wrapper would need to read rows (or one row ) from reader and copy data to new data buffer before it can be used in another command?

Data type conversions can be implemented on existing reader too.

What I see as way to improve performance is to be able to set data buffers for select command so returned reader would used those buffers. So this same data without copying can be reused.

Haris

Traian

I am affraid I don’t understand now. That equivalent of reader would be returned by command (than it is new reader) or it would be created from returned current Reader ?

Haris

Traian

Yes I understood that. I am afraid that it can be dangerous to use readers as input to another command.

One of thing would be that reader would not be closed before another command would be executed.

Not necessarily but I am worried that it could create problems with current readers/commands.

But even before that I would prefer to have index access to columns of readers as we speak of performance, we discussed it few times :)

Haris

Traian

Right now from Reader we can’t get PropertyValues only raw data.

To get Property value and property value collection which then could be passed to another command would be of great help and remove some of unnecessary data conversions.

Haris

Traian

Hi Brent,

not directly related to this RFC but still it is important issue regarding FDO data values.

I find a problem working with FDO Readers and commands is that I need to get value as basic type from reader and then convert to FdoDataValue etc...

I would like to be able to use data values and properties with values from readers directly and make those as input value for another FDO command.

Right now I need to check data type returned from property get that data as native convert back to fdo data value and put as input to FDO command.

Haris

Hi,

If approved, this one will likely be implemented 2 versions after FDO 3.3.

Thanks,

Brent.

_______________________________________________
fdo-internals mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/fdo-internals

Traian Stanev

RE: New RFC 30 Posted

Reply Threaded

GetGeometry has two overloads. One of them returns a raw pointer:

virtual const FdoByte* GetGeometry (FdoString* propertyName, FdoInt32* len);

This is the API I always use in client code. Also, GetString() returns a raw pointer which is also only valid till the following ReadNext():

virtual FdoString* GetString (FdoString* propertyName);

However, we are digressing from the topic...

Traian

From: [hidden email] [mailto:[hidden email]] On Behalf Of Haris Kurtagic
Sent: Monday, November 24, 2008 6:04 PM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] New RFC 30 Posted

Still even with SDF fdo clients are calling GetGeometry which returns FdoByteArray * which allocates new memory buffer and data is copied into new buffer.
Problem is not that data is copied once but that it is copied multiple times (not case with FdoByteArray ) but with other data types.

With preallocated shared data buffers, data returned by reader can be reused from same buffers without copying data even if reader is closed and destroyed.

Haris

From: [hidden email] [mailto:[hidden email]] On Behalf Of Traian Stanev
Sent: Monday, November 24, 2008 11:15 PM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] New RFC 30 Posted

Yes, such corruption is a possible problem. Hoewever, if your scheme is based on cached (pre-created) PropertyValue objects, then it has the same problem – someone can hold on to the PropertyValue object while it changes from under them. If you want to fix that, you’d have to do some really ugly refcounting hacks.

Besides, there is precedent for allowing corruptible memory – one of the GetGeometry calls in FdoFeatureReader is only guaranteed to hold good data until the next ReadNext call, and callers must be aware of that. This allows providers like SDF and SQLite for example to return pointers directly to database memory, thus avoiding doing even a single copy of the data. This is also how the GetString() calls works.

Traian

From: [hidden email] [mailto:[hidden email]] On Behalf Of Haris Kurtagic
Sent: Monday, November 24, 2008 5:15 PM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] New RFC 30 Posted

If that wrapper would return same data buffer than we are into same problems as if we would send reader to another command.
Feature reader data buffer would be corrupted on next call and that wrapper in another command would give wrong result.

When I was talking about feature reader returning fdo data value, was primarily to avoid what we do now: switch(data type), we create new fdo data type , copy raw data to new fdo data value and pass that data to another command.
Intention was to reduce that switch and data copy code in every fdo application.

I think to improve performance of reader, would be that reader will be able to fetch data in preallocated shared data buffers.

Haris

From: [hidden email] [mailto:[hidden email]] On Behalf Of Traian Stanev
Sent: Monday, November 24, 2008 10:15 PM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] New RFC 30 Posted

Yes, a wrapper around either FdoFeatureReader or a PropertyValueCollection – same API for both. However, it would not necessarily copy the data from the FeatureReader – in the case where it wraps a FeatureReader it could simply do a passthrough, for example:

virtual FdoString* FdoFRFeature::GetString(FdoString* propName) { return m_reader->GetString(propName); }

Where m_reader is a private member variable pointing to the FeatureReader.

In the case where it wraps a PropertyValueCollection, it would be *roughly*:

virtual FdoString* FdoPVCFeature::GetString(FdoString* propName) { return ((FdoStringValue*)m_propValCol->FindItem(propName))->GetString(); }

The actual implementation is not the point though, as long as the outfacing API allows for implementing it efficiently. I don’t think an API which uses PropertyValue objects has the maximum potential for efficiency because you would be incurring at least two virtual calls to refcounting functions on top of already expensive virtual calls to get the value.

Traian

From: [hidden email] [mailto:[hidden email]] On Behalf Of Haris Kurtagic
Sent: Monday, November 24, 2008 4:19 PM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] New RFC 30 Posted

I guess there is lot of discus on that topic.
You are proposing an wrapper over feature reader which would expose data of reader ?
Than that wrapper would need to read rows (or one row ) from reader and copy data to new data buffer before it can be used in another command?
If it would fetch data to new data buffers than it is same performance problem of copying data. Same performance as to copy data to property values as we do already eventually if we want to do something else with that data.
Data type conversions can be implemented on existing reader too.

What I see as way to improve performance is to be able to set data buffers for select command so returned reader would used those buffers. So this same data without copying can be reused.

Haris

From: [hidden email] [mailto:[hidden email]] On Behalf Of Traian Stanev
Sent: Monday, November 24, 2008 9:34 PM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] New RFC 30 Posted

Think of it as a data row – a feature reader lets you iterate through many such data rows via ReadNext(). It is also very similar to OGRFeature for example (http://www.gdal.org/ogr/classOGRFeature.html).

So, for example say you want to transfer a feature from one connection to another – you would do a select from one connection, and that will return a FeatureReader. You can wrap that in the data row object, which has essentially the same API as the feature reader, except that it does not allow ReadNext(). Then you can pass that wrapper to an FdoInsert which will take the data row and insert it into the destination connection.

Internally you could also wrap a PropertyValue collection in such a data row object – no matter whether it is backed by a FeatureReader or a PropertyValue collection, it would offer the same API. Also, you can have the data row convert all its data into a PropertyValueCollection which you can use in case you don’t need performance, thus enabling interop with existing code, which uses such PropertyValue collections.

You can also for example define this object so that it can do automatic type conversion for you – say you call GetInt32 on an Int16 property – it can automatically do the cast for you. Which would provide equivalent functionality to RFC 30 by the way, or it can use whatever code RFC30 adds to FDO. ☺

Traian

From: [hidden email] [mailto:[hidden email]] On Behalf Of Haris Kurtagic
Sent: Monday, November 24, 2008 3:36 PM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] New RFC 30 Posted

I am affraid I don’t understand now. That equivalent of reader would be returned by command (than it is new reader) or it would be created from returned current Reader ?

Haris

From: [hidden email] [mailto:[hidden email]] On Behalf Of Traian Stanev
Sent: Monday, November 24, 2008 9:01 PM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] New RFC 30 Posted

Yes that’s a problem and that’s why I said “FdoFeatureReader (or equivalent)”. What I meant by “equivalent” was something that has the almost the same API as FdoIFeatureReader, without the ReadNext/Close functions. Essentially it would represent a single row of data, with an API that allows fast access in a way that is the same as reading features coming from a query.

Traian

From: [hidden email] [mailto:[hidden email]] On Behalf Of Haris Kurtagic
Sent: Monday, November 24, 2008 3:21 PM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] New RFC 30 Posted

Yes I understood that. I am afraid that it can be dangerous to use readers as input to another command.
One of thing would be that reader would not be closed before another command would be executed.
Not necessarily but I am worried that it could create problems with current readers/commands.

If we look further as you suggested to improve performance of readers, than it would be better to have them to returned shared pointers to data or to be able to set data buffers to be used by readers.
But even before that I would prefer to have index access to columns of readers as we speak of performance, we discussed it few times :)

Haris

From: [hidden email] [mailto:[hidden email]] On Behalf Of Traian Stanev
Sent: Monday, November 24, 2008 8:41 PM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] New RFC 30 Posted

Yes, but creating PropertyValues would be a performance overhead. That’s why I’m suggesting the alternative approach of allowing commands to pull the feature’s data directly from a FeatureReader (or equivalent).

Traian

From: [hidden email] [mailto:[hidden email]] On Behalf Of Haris Kurtagic
Sent: Monday, November 24, 2008 2:25 PM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] New RFC 30 Posted

Right now from Reader we can’t get PropertyValues only raw data.
To get Property value and property value collection which then could be passed to another command would be of great help and remove some of unnecessary data conversions.

Haris

From: [hidden email] [mailto:[hidden email]] On Behalf Of Traian Stanev
Sent: Monday, November 24, 2008 7:39 PM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] New RFC 30 Posted

An alternative to this suggestion is to make the destination command accept an FdoIFeatureReader (or something equivalent that provides access to property values in a procedural way) as argument, rather than a collection of PropertyValues. I was thinking of this when I was implementing the SDF->SQLite converter, which requires two translations from PropertyValue to raw data – once when initializing the inputs of the insert command and a second time inside the destination provider when matching the input property values to the destination schema.

Traian

From: [hidden email] [mailto:[hidden email]] On Behalf Of Haris Kurtagic
Sent: Monday, November 24, 2008 8:27 AM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] New RFC 30 Posted

Hi Brent,
not directly related to this RFC but still it is important issue regarding FDO data values.

I find a problem working with FDO Readers and commands is that I need to get value as basic type from reader and then convert to FdoDataValue etc...
I would like to be able to use data values and properties with values from readers directly and make those as input value for another FDO command.
Right now I need to check data type returned from property get that data as native convert back to fdo data value and put as input to FDO command.

Haris

From: [hidden email] [mailto:[hidden email]] On Behalf Of Brent Robinson
Sent: Friday, November 21, 2008 7:32 PM
To: FDO Internals Mail List
Subject: [fdo-internals] New RFC 30 Posted

Hi,

RFC 30 (http://trac.osgeo.org/fdo/wiki/FDORfc30) is now ready for review. The main purpose is add FDO API functions to encapsulate conversions between different FDO data types. Please review and let me know if you have any questions or comments.

If approved, this one will likely be implemented 2 versions after FDO 3.3.

Thanks,
Brent.

_______________________________________________
fdo-internals mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/fdo-internals

Greg Boone

MOTION: FDO RFC 30 - Data Value Type Conversion

Reply Threaded

In reply to this post by Brent Robinson

Some javascript/style in this post has been disabled (why?)

Hi All,

I would like to motion that RFC 30 be formally adopted for inclusion in FDO 3.5.0.

Greg

From: [hidden email] [mailto:[hidden email]] On Behalf Of Brent Robinson
Sent: Friday, November 21, 2008 1:32 PM
To: FDO Internals Mail List
Subject: [fdo-internals] New RFC 30 Posted

Hi,

If approved, this one will likely be implemented 2 versions after FDO 3.3.

Thanks,

Brent.

_______________________________________________
fdo-internals mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/fdo-internals

Jackie Ng

Re: MOTION: FDO RFC 30 - Data Value Type Conversion

Reply Threaded

+1 Jackie

Greg Boone wrote:

Hi All,

I would like to motion that RFC 30 be formally adopted for inclusion in FDO 3.5.0.

Greg

From: fdo-internals-bounces@lists.osgeo.org [mailto:fdo-internals-bounces@lists.osgeo.org] On Behalf Of Brent Robinson
Sent: Friday, November 21, 2008 1:32 PM
To: FDO Internals Mail List
Subject: [fdo-internals] New RFC 30 Posted

Hi,

RFC 30 (http://trac.osgeo.org/fdo/wiki/FDORfc30) is now ready for review. The main purpose is add FDO API functions to encapsulate conversions between different FDO data types. Please review and let me know if you have any questions or comments.

If approved, this one will likely be implemented 2 versions after FDO 3.3.

Thanks,
Brent.

_______________________________________________
fdo-internals mailing list
fdo-internals@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/fdo-internals

Jason Birch

RE: MOTION: FDO RFC 30 - Data Value Type Conversion

Reply Threaded

In reply to this post by Greg Boone

Some javascript/style in this post has been disabled (why?)

Heh. I'd forgotten about that one.

Jason

From: [hidden email] [mailto:[hidden email]] On Behalf Of Greg Boone
Sent: April-09-09 8:20 AM
To: FDO Internals Mail List
Subject: [fdo-internals] MOTION: FDO RFC 30 - Data Value Type Conversion

Hi All,

I would like to motion that RFC 30 be formally adopted for inclusion in FDO 3.5.0.

Greg

Hi,

If approved, this one will likely be implemented 2 versions after FDO 3.3.

Thanks,

Brent.

_______________________________________________
fdo-internals mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/fdo-internals

Traian Stanev

RE: MOTION: FDO RFC 30 - Data Value Type Conversion

Reply Threaded

In reply to this post by Jackie Ng

+1 Traian

________________________________________
From: [hidden email] [[hidden email]] On Behalf Of Jackie Ng [[hidden email]]
Sent: Thursday, April 09, 2009 11:07 PM
To: [hidden email]
Subject: Re: [fdo-internals] MOTION: FDO RFC 30 - Data Value Type Conversion

+1 Jackie

Hi All,

I would like to motion that RFC 30 be formally adopted for inclusion in FDO 3.5.0.

Greg

From: [hidden email] [mailto:[hidden email]] On Behalf Of Brent Robinson
Sent: Friday, November 21, 2008 1:32 PM
To: FDO Internals Mail List
Subject: [fdo-internals] New RFC 30 Posted

Hi,

RFC 30 (http://trac.osgeo.org/fdo/wiki/FDORfc30) is now ready for review. The main purpose is add FDO API functions to encapsulate conversions between different FDO data types. Please review and let me know if you have any questions or comments.

If approved, this one will likely be implemented 2 versions after FDO 3.3.

Thanks,
Brent.

_______________________________________________
fdo-internals mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/fdo-internals

--
View this message in context: http://n2.nabble.com/New-RFC-30-Posted-tp2051948p2614617.html
Sent from the FDO Internals mailing list archive at Nabble.com.

_______________________________________________
fdo-internals mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/fdo-internals
_______________________________________________
fdo-internals mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/fdo-internals

Haris Kurtagic

RE: MOTION: FDO RFC 30 - Data Value Type Conversion

Reply Threaded

+1 Haris

-----Original Message-----
From: [hidden email]
[mailto:[hidden email]] On Behalf Of Traian
Stanev
Sent: Monday, April 13, 2009 5:16 PM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] MOTION: FDO RFC 30 - Data Value Type
Conversion

+1 Traian

________________________________________
From: [hidden email]
[[hidden email]] On Behalf Of Jackie Ng
[[hidden email]]
Sent: Thursday, April 09, 2009 11:07 PM
To: [hidden email]
Subject: Re: [fdo-internals] MOTION: FDO RFC 30 - Data Value Type
Conversion

+1 Jackie

Hi All,

I would like to motion that RFC 30 be formally adopted for inclusion in
FDO 3.5.0.

Greg

From: [hidden email]
[mailto:[hidden email]] On Behalf Of Brent
Robinson
Sent: Friday, November 21, 2008 1:32 PM
To: FDO Internals Mail List
Subject: [fdo-internals] New RFC 30 Posted

Hi,

RFC 30 (http://trac.osgeo.org/fdo/wiki/FDORfc30) is now ready for
review. The main purpose is add FDO API functions to encapsulate
conversions between different FDO data types. Please review and let me
know if you have any questions or comments.

If approved, this one will likely be implemented 2 versions after FDO
3.3.

Thanks,
Brent.

_______________________________________________
fdo-internals mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/fdo-internals

--
View this message in context:
http://n2.nabble.com/New-RFC-30-Posted-tp2051948p2614617.html
Sent from the FDO Internals mailing list archive at Nabble.com.

_______________________________________________
fdo-internals mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/fdo-internals
_______________________________________________
fdo-internals mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/fdo-internals
_______________________________________________
fdo-internals mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/fdo-internals

1 2