Use of SQL pass through

14 messages Options
Embed this post
Permalink
Orest Halustchak

Use of SQL pass through

Reply Threaded More More options
Print post
Permalink
Some javascript/style in this post has been disabled (why?)

Hi,

 

I would like to solicit feedback, discussion, and suggestions on requirements and issues that we are seeing with the FdoISQLCommand which is used for the purpose of SQL pass through for some of the RDBMS-based providers.

 

Please see the discussion below.

 

Thanks,

Orest.

 

 

 

Enhanced SQL Command Support Discussion

Overview

The FDO API currently defines support for a SQL command that allows for pass-through execution of SQL statements either through a non-query execution of the SQL statement directly in the underlying Data Store, or through a query mechanism that returns a simple data reader listing the properties returned as a result of the SQL execution. The definition of the SQL command has remained fairly static over the last number of releases as primary development efforts have focused on extending other aspects of the FDO API, implementing new providers, etc. However, requirements for change to the SQL command have accumulated for us as RDBMS providers have implemented SQL pass through support and clients have attempted to integrate use SQL pass through into their applications in a seamless manner. While we generally expect applications to be using the generic FDO Select and other commands using FDO expressions, application still need to execute direct SQL against RDBMS-based providers on an exception basis for things that cannot do through FDO. While we hope to improve the FDO expression capabilities over time, there will always be this need for direct SQL processing for exception cases.

This email is meant to start discussion on how to handle the requirements that we’ve been seeing and to get feedback on how to modify the FDO api to handle these requirements. We have some ideas as described below, but would like to get other feedback. We’ll draft an RFC once we get close to a consensus on how it should look.

Requirements

One key request has been the desire to have the FDO API support SQL pass-through commands that return an FDO feature reader, referencing a proper FDO schema, not simply an FDO data reader. The feature reader will contain proper geometry properties, relations and associations. This enhancement is also intended to allow client applications that use FDO Feature Readers to code their applications in a generic manner so that data coming back from Select or SQL Pass Through statements can be processed in a uniform manner, thus reducing complexity, costs and time to implement.

Other more SQL specific requirements related to this have also arisen, including:

  • A need to allow a client to set the size of the fetch array an FDO provider will use when executing SQL statements ([OH] I still have an issue with this one. Fetch sizes are internal tuning parameters and are not FDO api concepts. Other providers that deal with select or insert buffering have default internal sizes.)
  • A need to specify bind variables for the SQL command, including arrays of bind variables. Since SQL commands may include stored procedure execution, bind variables need indication of whether they are input, output, or return parameters.

Discussion

In general, the intent of SQL pass-through is to deal with physical schemas. There is no parsing of the SQL statements, and what are identified are physical schema tables, columns, functions, commands, etc. The SQL statement can be any SQL command, not just select, but any DML (select, insert, update, delete) or any DDL (create, drop, alter, …). However, FDO feature readers deal with FDO logical schemas, which is at a different conceptual level. It’s the mixing of these levels that causes difficulty for applications using the FDO API. Applications are required to use different code pathways to handle select statement as opposed to direct SQL execution. If clients could use the result of either of these types of operations in a generic routine or component, such as a Data Table or Symbolization packages, applications would be provided a much more seamless and dynamic mechanism on which they can build and extend their applications.

To a certain degree, the current FDO feature reader implementation assumes an FDO class definition is directly related to the properties being returned. With physical schema SQL (let’s say just Select command), there isn’t necessarily any FDO class definition that applies. This is why currently the SQL command's Execute method returns a data reader, which handles any generically returned data. Note that the FDO select aggregates command doesn’t return a feature reader either, since it’s returning computed data.

One response to this issue has been to suggest that providers reverse engineer the select results and attempt to find the FDO class definition (coming from a describe schema request) that matches it. Other suggestions have been to construct a class definition on the fly, one that would not result from a call to describe schema. If the select is from a single table, providers would find the class definition that is based on that table and use it. However, then matching up the columns being selected with the properties in the logical schema, there may be some slight differences. This would result in the mixing of physical schema and logical schema elements together. For example, class names that are not the same as table names, property names that are not the same as column names, the use of additional computed columns, inclusion of pkey columns, etc. Granted that in many cases, the logical and physical views would be virtually identical. However, since that cannot be guaranteed, the design that is adopted will have to be able to handle the general case.

To account for these scenarios, providers will need to be modified to reverse engineer class definitions from the selected data and not attempt to match the select request to an existing FDO class. In reality, FDO Providers will be required to handle this in any case since a generic SQL select may not match up at all with an existing FDO class. An example of this can be found when selecting from a table with an owner that is different from the connected data store. For example, Connect to data store called Denver and select from Boulder.Roads – the schema may or may not be similar to Denver.Roads. In these types of circumstances, providers should manufacture a new class definition.

In cases where the resulting columns come from an existing FDO feature table, a provider can return the class definition corresponding to that table. In cases where the columns come from an unknown table, a class definition can be constructed on the fly. By definition, the FDO class definition returned by a feature reader does not necessarily correspond exactly to an existing FDO class definition. Existing class definitions may contain the properties that were asked for in the Select command, plus additional computed properties. It is perfectly legal to return a constructed class definition, which is only valid for the select that was executed, and not usable for further updates or inserts.

In the circumstance that a computed class is generated, the FDO class definition’s IsComputed property will return true. In this manner, applications are able to distinguish the content of the feature reader responses coming from the providers and tailor their implementation accordingly. In such a situation, some care will also need to be given to the name of the generated FDO schema and class definitions. At this point no standards exist for naming auto-generated schema, class and property names. It would be beneficial if, as a result of this RFC, some uniform naming conventions could be adopted.

Providers that do return feature readers from SQL commands will need to come up with the appropriate class definition that the feature reader could expose. Here are a few general use cases:

  • Select is against a table that has an existing class definition and the select returns the same information as defined by the class definition (e.g. select * from roads).
  • Select is against a table that has an existing class definition, but the select returns other information such as a subset of properties or additional computed properties. The FDO select command handles this type of case as well. The class definition returned by the feature reader includes the specific properties for that select.
  • Select is against a table that doesn’t have an existing class definition. Providers will be required to generate a temporary class definition.
  • Select is against a table that has an existing class definition, but the select is an aggregation that doesn’t return actual objects (e.g. select count(*), max(length) from roads). In this case it doesn’t make sense for the provider to return the same class name as for the underlying table since it is not actually returning road objects at all. This is basically another temporary class definition.

FDO API Ideas

In order to support the SQL pass-through objective outlined above, the FDO API will need to be modified to provide a new capability function to state that this is supported, and to provide a way to return an explicit FDO Feature Reader. Two possibilities have been identified for returning the feature reader. One idea is to add a new ExecuteFeatureReader (or similar name) method to FdoISQLCommand. The existing ExecuteReader method will remain as is for backwards compatibility. Another possibility is to keep the SQL pass through interface unchanged but provide a utility that will convert the returned SQL Data Reader into a Feature Reader after the fact.

A related question is around the class definition that would be generated from the SQL since it often would be a temporary class definition for the command and not something found from the FDO Describe Schema command. The feature reader would expose that class definition, but would applications need to get that earlier, e.g. with a new method on FdoISQLCommand to describe the generated class definition prior to getting the reader?

Parameter Direction

FDO already includes API’s for defining and using parameters (bind variables). However, there is no indication of direction (input versus output) and if an application is planning to use stored procedure calls in the SQL pass through, they would need to provide some indication of input, output, or return parameters. A possibility is to add a new FdoParameterDirection enumeration and add set/get methods to the FdoParameterValue or FdoParameterValueCollection interface.

A related question is whether FdoISQLCommand needs the application to tell it whether the SQL command being executed is a stored procedure call or some other type of SQL? We want to avoid having providers be forced to parse SQL where currently they probably all just send the SQL directly to the underlying server. The ExecuteNonQuery versus ExecuteReader methods that the application uses already tell the provider whether to expect results back from the SQL or not. A stored procedure call is a “non-query” but can return results, so is it another case?

 

 


_______________________________________________
fdo-internals mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/fdo-internals
Jackie Ng

Re: Use of SQL pass through

Reply Threaded More More options
Print post
Permalink
Hi Orest,

I am happy that this issue is finally being addressed. It has been an absolute pain to not be able to treat feature and sql query results the same manner.

On a somewhat unrelated note, are we able to fix up the FdoISQLDataReader interface? It shares the same method interface as FdoIFeatureReader (FdoIReader), yet it does not inherit from FdoIReader. So I have to treat these readers separately even though they should have a common FdoIReader base interface. (ref: http://trac.osgeo.org/fdo/ticket/359)

- Jackie

Orest Halustchak wrote:
Hi,

I would like to solicit feedback, discussion, and suggestions on requirements and issues that we are seeing with the FdoISQLCommand which is used for the purpose of SQL pass through for some of the RDBMS-based providers.

Please see the discussion below.

Thanks,
Orest.



Enhanced SQL Command Support Discussion
Overview
The FDO API currently defines support for a SQL command that allows for pass-through execution of SQL statements either through a non-query execution of the SQL statement directly in the underlying Data Store, or through a query mechanism that returns a simple data reader listing the properties returned as a result of the SQL execution. The definition of the SQL command has remained fairly static over the last number of releases as primary development efforts have focused on extending other aspects of the FDO API, implementing new providers, etc. However, requirements for change to the SQL command have accumulated for us as RDBMS providers have implemented SQL pass through support and clients have attempted to integrate use SQL pass through into their applications in a seamless manner. While we generally expect applications to be using the generic FDO Select and other commands using FDO expressions, application still need to execute direct SQL against RDBMS-based providers on an exception basis for things that cannot do through FDO. While we hope to improve the FDO expression capabilities over time, there will always be this need for direct SQL processing for exception cases.
This email is meant to start discussion on how to handle the requirements that we've been seeing and to get feedback on how to modify the FDO api to handle these requirements. We have some ideas as described below, but would like to get other feedback. We'll draft an RFC once we get close to a consensus on how it should look.
Requirements
One key request has been the desire to have the FDO API support SQL pass-through commands that return an FDO feature reader, referencing a proper FDO schema, not simply an FDO data reader. The feature reader will contain proper geometry properties, relations and associations. This enhancement is also intended to allow client applications that use FDO Feature Readers to code their applications in a generic manner so that data coming back from Select or SQL Pass Through statements can be processed in a uniform manner, thus reducing complexity, costs and time to implement.
Other more SQL specific requirements related to this have also arisen, including:

 *   A need to allow a client to set the size of the fetch array an FDO provider will use when executing SQL statements ([OH] I still have an issue with this one. Fetch sizes are internal tuning parameters and are not FDO api concepts. Other providers that deal with select or insert buffering have default internal sizes.)
 *   A need to specify bind variables for the SQL command, including arrays of bind variables. Since SQL commands may include stored procedure execution, bind variables need indication of whether they are input, output, or return parameters.
Discussion
In general, the intent of SQL pass-through is to deal with physical schemas. There is no parsing of the SQL statements, and what are identified are physical schema tables, columns, functions, commands, etc. The SQL statement can be any SQL command, not just select, but any DML (select, insert, update, delete) or any DDL (create, drop, alter, ...). However, FDO feature readers deal with FDO logical schemas, which is at a different conceptual level. It's the mixing of these levels that causes difficulty for applications using the FDO API. Applications are required to use different code pathways to handle select statement as opposed to direct SQL execution. If clients could use the result of either of these types of operations in a generic routine or component, such as a Data Table or Symbolization packages, applications would be provided a much more seamless and dynamic mechanism on which they can build and extend their applications.
To a certain degree, the current FDO feature reader implementation assumes an FDO class definition is directly related to the properties being returned. With physical schema SQL (let's say just Select command), there isn't necessarily any FDO class definition that applies. This is why currently the SQL command's Execute method returns a data reader, which handles any generically returned data. Note that the FDO select aggregates command doesn't return a feature reader either, since it's returning computed data.
One response to this issue has been to suggest that providers reverse engineer the select results and attempt to find the FDO class definition (coming from a describe schema request) that matches it. Other suggestions have been to construct a class definition on the fly, one that would not result from a call to describe schema. If the select is from a single table, providers would find the class definition that is based on that table and use it. However, then matching up the columns being selected with the properties in the logical schema, there may be some slight differences. This would result in the mixing of physical schema and logical schema elements together. For example, class names that are not the same as table names, property names that are not the same as column names, the use of additional computed columns, inclusion of pkey columns, etc. Granted that in many cases, the logical and physical views would be virtually identical. However, since that cannot be guaranteed, the design that is adopted will have to be able to handle the general case.
To account for these scenarios, providers will need to be modified to reverse engineer class definitions from the selected data and not attempt to match the select request to an existing FDO class. In reality, FDO Providers will be required to handle this in any case since a generic SQL select may not match up at all with an existing FDO class. An example of this can be found when selecting from a table with an owner that is different from the connected data store. For example, Connect to data store called Denver and select from Boulder.Roads - the schema may or may not be similar to Denver.Roads. In these types of circumstances, providers should manufacture a new class definition.
In cases where the resulting columns come from an existing FDO feature table, a provider can return the class definition corresponding to that table. In cases where the columns come from an unknown table, a class definition can be constructed on the fly. By definition, the FDO class definition returned by a feature reader does not necessarily correspond exactly to an existing FDO class definition. Existing class definitions may contain the properties that were asked for in the Select command, plus additional computed properties. It is perfectly legal to return a constructed class definition, which is only valid for the select that was executed, and not usable for further updates or inserts.
In the circumstance that a computed class is generated, the FDO class definition's IsComputed property will return true. In this manner, applications are able to distinguish the content of the feature reader responses coming from the providers and tailor their implementation accordingly. In such a situation, some care will also need to be given to the name of the generated FDO schema and class definitions. At this point no standards exist for naming auto-generated schema, class and property names. It would be beneficial if, as a result of this RFC, some uniform naming conventions could be adopted.
Providers that do return feature readers from SQL commands will need to come up with the appropriate class definition that the feature reader could expose. Here are a few general use cases:

 *   Select is against a table that has an existing class definition and the select returns the same information as defined by the class definition (e.g. select * from roads).
 *   Select is against a table that has an existing class definition, but the select returns other information such as a subset of properties or additional computed properties. The FDO select command handles this type of case as well. The class definition returned by the feature reader includes the specific properties for that select.
 *   Select is against a table that doesn't have an existing class definition. Providers will be required to generate a temporary class definition.
 *   Select is against a table that has an existing class definition, but the select is an aggregation that doesn't return actual objects (e.g. select count(*), max(length) from roads). In this case it doesn't make sense for the provider to return the same class name as for the underlying table since it is not actually returning road objects at all. This is basically another temporary class definition.
FDO API Ideas
In order to support the SQL pass-through objective outlined above, the FDO API will need to be modified to provide a new capability function to state that this is supported, and to provide a way to return an explicit FDO Feature Reader. Two possibilities have been identified for returning the feature reader. One idea is to add a new ExecuteFeatureReader (or similar name) method to FdoISQLCommand. The existing ExecuteReader method will remain as is for backwards compatibility. Another possibility is to keep the SQL pass through interface unchanged but provide a utility that will convert the returned SQL Data Reader into a Feature Reader after the fact.
A related question is around the class definition that would be generated from the SQL since it often would be a temporary class definition for the command and not something found from the FDO Describe Schema command. The feature reader would expose that class definition, but would applications need to get that earlier, e.g. with a new method on FdoISQLCommand to describe the generated class definition prior to getting the reader?
Parameter Direction
FDO already includes API's for defining and using parameters (bind variables). However, there is no indication of direction (input versus output) and if an application is planning to use stored procedure calls in the SQL pass through, they would need to provide some indication of input, output, or return parameters. A possibility is to add a new FdoParameterDirection enumeration and add set/get methods to the FdoParameterValue or FdoParameterValueCollection interface.
A related question is whether FdoISQLCommand needs the application to tell it whether the SQL command being executed is a stored procedure call or some other type of SQL? We want to avoid having providers be forced to parse SQL where currently they probably all just send the SQL directly to the underlying server. The ExecuteNonQuery versus ExecuteReader methods that the application uses already tell the provider whether to expect results back from the SQL or not. A stored procedure call is a "non-query" but can return results, so is it another case?



_______________________________________________
fdo-internals mailing list
fdo-internals@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/fdo-internals
Haris Kurtagic

RE: Use of SQL pass through

Reply Threaded More More options
Print post
Permalink
In reply to this post by Orest Halustchak
Some javascript/style in this post has been disabled (why?)

I would like to divide this and answer in two parts.

 

Part with Parameters direction is something we need. As it is already written , mostly to be able to execute and get results from stored procedures.

 

About executing SQL statements and trying to squeeze result of it inside FDO class/schema, I think it is too complex and in my mind without chance to be successful. There is so many cases in which it can't be done properly. If we are missing some info about result of execution of SQL we could look into existing specs like ODBC and add those. I agree FDO application should be able to get all necessary info about executed SQL so app can be written in generic way but I don't see putting that info in FDO class or reengineering sql etc.. I believe what can be done with api's like odbc is ok for fdo api too.

 

Haris

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Orest Halustchak
Sent: Wednesday, June 03, 2009 3:28 PM
To: FDO Internals Mail List
Subject: [fdo-internals] Use of SQL pass through

 

Hi,

 

I would like to solicit feedback, discussion, and suggestions on requirements and issues that we are seeing with the FdoISQLCommand which is used for the purpose of SQL pass through for some of the RDBMS-based providers.

 

Please see the discussion below.

 

Thanks,

Orest.

 

 

 

Enhanced SQL Command Support Discussion

Overview

The FDO API currently defines support for a SQL command that allows for pass-through execution of SQL statements either through a non-query execution of the SQL statement directly in the underlying Data Store, or through a query mechanism that returns a simple data reader listing the properties returned as a result of the SQL execution. The definition of the SQL command has remained fairly static over the last number of releases as primary development efforts have focused on extending other aspects of the FDO API, implementing new providers, etc. However, requirements for change to the SQL command have accumulated for us as RDBMS providers have implemented SQL pass through support and clients have attempted to integrate use SQL pass through into their applications in a seamless manner. While we generally expect applications to be using the generic FDO Select and other commands using FDO expressions, application still need to execute direct SQL against RDBMS-based providers on an exception basis for things that cannot do through FDO. While we hope to improve the FDO expression capabilities over time, there will always be this need for direct SQL processing for exception cases.

This email is meant to start discussion on how to handle the requirements that we’ve been seeing and to get feedback on how to modify the FDO api to handle these requirements. We have some ideas as described below, but would like to get other feedback. We’ll draft an RFC once we get close to a consensus on how it should look.

Requirements

One key request has been the desire to have the FDO API support SQL pass-through commands that return an FDO feature reader, referencing a proper FDO schema, not simply an FDO data reader. The feature reader will contain proper geometry properties, relations and associations. This enhancement is also intended to allow client applications that use FDO Feature Readers to code their applications in a generic manner so that data coming back from Select or SQL Pass Through statements can be processed in a uniform manner, thus reducing complexity, costs and time to implement.

Other more SQL specific requirements related to this have also arisen, including:

  • A need to allow a client to set the size of the fetch array an FDO provider will use when executing SQL statements ([OH] I still have an issue with this one. Fetch sizes are internal tuning parameters and are not FDO api concepts. Other providers that deal with select or insert buffering have default internal sizes.)
  • A need to specify bind variables for the SQL command, including arrays of bind variables. Since SQL commands may include stored procedure execution, bind variables need indication of whether they are input, output, or return parameters.

Discussion

In general, the intent of SQL pass-through is to deal with physical schemas. There is no parsing of the SQL statements, and what are identified are physical schema tables, columns, functions, commands, etc. The SQL statement can be any SQL command, not just select, but any DML (select, insert, update, delete) or any DDL (create, drop, alter, …). However, FDO feature readers deal with FDO logical schemas, which is at a different conceptual level. It’s the mixing of these levels that causes difficulty for applications using the FDO API. Applications are required to use different code pathways to handle select statement as opposed to direct SQL execution. If clients could use the result of either of these types of operations in a generic routine or component, such as a Data Table or Symbolization packages, applications would be provided a much more seamless and dynamic mechanism on which they can build and extend their applications.

To a certain degree, the current FDO feature reader implementation assumes an FDO class definition is directly related to the properties being returned. With physical schema SQL (let’s say just Select command), there isn’t necessarily any FDO class definition that applies. This is why currently the SQL command's Execute method returns a data reader, which handles any generically returned data. Note that the FDO select aggregates command doesn’t return a feature reader either, since it’s returning computed data.

One response to this issue has been to suggest that providers reverse engineer the select results and attempt to find the FDO class definition (coming from a describe schema request) that matches it. Other suggestions have been to construct a class definition on the fly, one that would not result from a call to describe schema. If the select is from a single table, providers would find the class definition that is based on that table and use it. However, then matching up the columns being selected with the properties in the logical schema, there may be some slight differences. This would result in the mixing of physical schema and logical schema elements together. For example, class names that are not the same as table names, property names that are not the same as column names, the use of additional computed columns, inclusion of pkey columns, etc. Granted that in many cases, the logical and physical views would be virtually identical. However, since that cannot be guaranteed, the design that is adopted will have to be able to handle the general case.

To account for these scenarios, providers will need to be modified to reverse engineer class definitions from the selected data and not attempt to match the select request to an existing FDO class. In reality, FDO Providers will be required to handle this in any case since a generic SQL select may not match up at all with an existing FDO class. An example of this can be found when selecting from a table with an owner that is different from the connected data store. For example, Connect to data store called Denver and select from Boulder.Roads – the schema may or may not be similar to Denver.Roads. In these types of circumstances, providers should manufacture a new class definition.

In cases where the resulting columns come from an existing FDO feature table, a provider can return the class definition corresponding to that table. In cases where the columns come from an unknown table, a class definition can be constructed on the fly. By definition, the FDO class definition returned by a feature reader does not necessarily correspond exactly to an existing FDO class definition. Existing class definitions may contain the properties that were asked for in the Select command, plus additional computed properties. It is perfectly legal to return a constructed class definition, which is only valid for the select that was executed, and not usable for further updates or inserts.

In the circumstance that a computed class is generated, the FDO class definition’s IsComputed property will return true. In this manner, applications are able to distinguish the content of the feature reader responses coming from the providers and tailor their implementation accordingly. In such a situation, some care will also need to be given to the name of the generated FDO schema and class definitions. At this point no standards exist for naming auto-generated schema, class and property names. It would be beneficial if, as a result of this RFC, some uniform naming conventions could be adopted.

Providers that do return feature readers from SQL commands will need to come up with the appropriate class definition that the feature reader could expose. Here are a few general use cases:

  • Select is against a table that has an existing class definition and the select returns the same information as defined by the class definition (e.g. select * from roads).
  • Select is against a table that has an existing class definition, but the select returns other information such as a subset of properties or additional computed properties. The FDO select command handles this type of case as well. The class definition returned by the feature reader includes the specific properties for that select.
  • Select is against a table that doesn’t have an existing class definition. Providers will be required to generate a temporary class definition.
  • Select is against a table that has an existing class definition, but the select is an aggregation that doesn’t return actual objects (e.g. select count(*), max(length) from roads). In this case it doesn’t make sense for the provider to return the same class name as for the underlying table since it is not actually returning road objects at all. This is basically another temporary class definition.

FDO API Ideas

In order to support the SQL pass-through objective outlined above, the FDO API will need to be modified to provide a new capability function to state that this is supported, and to provide a way to return an explicit FDO Feature Reader. Two possibilities have been identified for returning the feature reader. One idea is to add a new ExecuteFeatureReader (or similar name) method to FdoISQLCommand. The existing ExecuteReader method will remain as is for backwards compatibility. Another possibility is to keep the SQL pass through interface unchanged but provide a utility that will convert the returned SQL Data Reader into a Feature Reader after the fact.

A related question is around the class definition that would be generated from the SQL since it often would be a temporary class definition for the command and not something found from the FDO Describe Schema command. The feature reader would expose that class definition, but would applications need to get that earlier, e.g. with a new method on FdoISQLCommand to describe the generated class definition prior to getting the reader?

Parameter Direction

FDO already includes API’s for defining and using parameters (bind variables). However, there is no indication of direction (input versus output) and if an application is planning to use stored procedure calls in the SQL pass through, they would need to provide some indication of input, output, or return parameters. A possibility is to add a new FdoParameterDirection enumeration and add set/get methods to the FdoParameterValue or FdoParameterValueCollection interface.

A related question is whether FdoISQLCommand needs the application to tell it whether the SQL command being executed is a stored procedure call or some other type of SQL? We want to avoid having providers be forced to parse SQL where currently they probably all just send the SQL directly to the underlying server. The ExecuteNonQuery versus ExecuteReader methods that the application uses already tell the provider whether to expect results back from the SQL or not. A stored procedure call is a “non-query” but can return results, so is it another case?

 

 


_______________________________________________
fdo-internals mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/fdo-internals
Greg Boone

RE: Use of SQL pass through

Reply Threaded More More options
Print post
Permalink
Some javascript/style in this post has been disabled (why?)

Hi Haris,

 

Can you provide some examples of non-DDL use cases that would make the proposal too complex? I also feel we should limit the discussion at this time to read only DML statements such as Select.

 

Greg

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Haris Kurtagic
Sent: Thursday, June 04, 2009 5:41 AM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] Use of SQL pass through

 

I would like to divide this and answer in two parts.

 

Part with Parameters direction is something we need. As it is already written , mostly to be able to execute and get results from stored procedures.

 

About executing SQL statements and trying to squeeze result of it inside FDO class/schema, I think it is too complex and in my mind without chance to be successful. There is so many cases in which it can't be done properly. If we are missing some info about result of execution of SQL we could look into existing specs like ODBC and add those. I agree FDO application should be able to get all necessary info about executed SQL so app can be written in generic way but I don't see putting that info in FDO class or reengineering sql etc.. I believe what can be done with api's like odbc is ok for fdo api too.

 

Haris

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Orest Halustchak
Sent: Wednesday, June 03, 2009 3:28 PM
To: FDO Internals Mail List
Subject: [fdo-internals] Use of SQL pass through

 

Hi,

 

I would like to solicit feedback, discussion, and suggestions on requirements and issues that we are seeing with the FdoISQLCommand which is used for the purpose of SQL pass through for some of the RDBMS-based providers.

 

Please see the discussion below.

 

Thanks,

Orest.

 

 

 

Enhanced SQL Command Support Discussion

Overview

The FDO API currently defines support for a SQL command that allows for pass-through execution of SQL statements either through a non-query execution of the SQL statement directly in the underlying Data Store, or through a query mechanism that returns a simple data reader listing the properties returned as a result of the SQL execution. The definition of the SQL command has remained fairly static over the last number of releases as primary development efforts have focused on extending other aspects of the FDO API, implementing new providers, etc. However, requirements for change to the SQL command have accumulated for us as RDBMS providers have implemented SQL pass through support and clients have attempted to integrate use SQL pass through into their applications in a seamless manner. While we generally expect applications to be using the generic FDO Select and other commands using FDO expressions, application still need to execute direct SQL against RDBMS-based providers on an exception basis for things that cannot do through FDO. While we hope to improve the FDO expression capabilities over time, there will always be this need for direct SQL processing for exception cases.

This email is meant to start discussion on how to handle the requirements that we’ve been seeing and to get feedback on how to modify the FDO api to handle these requirements. We have some ideas as described below, but would like to get other feedback. We’ll draft an RFC once we get close to a consensus on how it should look.

Requirements

One key request has been the desire to have the FDO API support SQL pass-through commands that return an FDO feature reader, referencing a proper FDO schema, not simply an FDO data reader. The feature reader will contain proper geometry properties, relations and associations. This enhancement is also intended to allow client applications that use FDO Feature Readers to code their applications in a generic manner so that data coming back from Select or SQL Pass Through statements can be processed in a uniform manner, thus reducing complexity, costs and time to implement.

Other more SQL specific requirements related to this have also arisen, including:

  • A need to allow a client to set the size of the fetch array an FDO provider will use when executing SQL statements ([OH] I still have an issue with this one. Fetch sizes are internal tuning parameters and are not FDO api concepts. Other providers that deal with select or insert buffering have default internal sizes.)
  • A need to specify bind variables for the SQL command, including arrays of bind variables. Since SQL commands may include stored procedure execution, bind variables need indication of whether they are input, output, or return parameters.

Discussion

In general, the intent of SQL pass-through is to deal with physical schemas. There is no parsing of the SQL statements, and what are identified are physical schema tables, columns, functions, commands, etc. The SQL statement can be any SQL command, not just select, but any DML (select, insert, update, delete) or any DDL (create, drop, alter, …). However, FDO feature readers deal with FDO logical schemas, which is at a different conceptual level. It’s the mixing of these levels that causes difficulty for applications using the FDO API. Applications are required to use different code pathways to handle select statement as opposed to direct SQL execution. If clients could use the result of either of these types of operations in a generic routine or component, such as a Data Table or Symbolization packages, applications would be provided a much more seamless and dynamic mechanism on which they can build and extend their applications.

To a certain degree, the current FDO feature reader implementation assumes an FDO class definition is directly related to the properties being returned. With physical schema SQL (let’s say just Select command), there isn’t necessarily any FDO class definition that applies. This is why currently the SQL command's Execute method returns a data reader, which handles any generically returned data. Note that the FDO select aggregates command doesn’t return a feature reader either, since it’s returning computed data.

One response to this issue has been to suggest that providers reverse engineer the select results and attempt to find the FDO class definition (coming from a describe schema request) that matches it. Other suggestions have been to construct a class definition on the fly, one that would not result from a call to describe schema. If the select is from a single table, providers would find the class definition that is based on that table and use it. However, then matching up the columns being selected with the properties in the logical schema, there may be some slight differences. This would result in the mixing of physical schema and logical schema elements together. For example, class names that are not the same as table names, property names that are not the same as column names, the use of additional computed columns, inclusion of pkey columns, etc. Granted that in many cases, the logical and physical views would be virtually identical. However, since that cannot be guaranteed, the design that is adopted will have to be able to handle the general case.

To account for these scenarios, providers will need to be modified to reverse engineer class definitions from the selected data and not attempt to match the select request to an existing FDO class. In reality, FDO Providers will be required to handle this in any case since a generic SQL select may not match up at all with an existing FDO class. An example of this can be found when selecting from a table with an owner that is different from the connected data store. For example, Connect to data store called Denver and select from Boulder.Roads – the schema may or may not be similar to Denver.Roads. In these types of circumstances, providers should manufacture a new class definition.

In cases where the resulting columns come from an existing FDO feature table, a provider can return the class definition corresponding to that table. In cases where the columns come from an unknown table, a class definition can be constructed on the fly. By definition, the FDO class definition returned by a feature reader does not necessarily correspond exactly to an existing FDO class definition. Existing class definitions may contain the properties that were asked for in the Select command, plus additional computed properties. It is perfectly legal to return a constructed class definition, which is only valid for the select that was executed, and not usable for further updates or inserts.

In the circumstance that a computed class is generated, the FDO class definition’s IsComputed property will return true. In this manner, applications are able to distinguish the content of the feature reader responses coming from the providers and tailor their implementation accordingly. In such a situation, some care will also need to be given to the name of the generated FDO schema and class definitions. At this point no standards exist for naming auto-generated schema, class and property names. It would be beneficial if, as a result of this RFC, some uniform naming conventions could be adopted.

Providers that do return feature readers from SQL commands will need to come up with the appropriate class definition that the feature reader could expose. Here are a few general use cases:

  • Select is against a table that has an existing class definition and the select returns the same information as defined by the class definition (e.g. select * from roads).
  • Select is against a table that has an existing class definition, but the select returns other information such as a subset of properties or additional computed properties. The FDO select command handles this type of case as well. The class definition returned by the feature reader includes the specific properties for that select.
  • Select is against a table that doesn’t have an existing class definition. Providers will be required to generate a temporary class definition.
  • Select is against a table that has an existing class definition, but the select is an aggregation that doesn’t return actual objects (e.g. select count(*), max(length) from roads). In this case it doesn’t make sense for the provider to return the same class name as for the underlying table since it is not actually returning road objects at all. This is basically another temporary class definition.

FDO API Ideas

In order to support the SQL pass-through objective outlined above, the FDO API will need to be modified to provide a new capability function to state that this is supported, and to provide a way to return an explicit FDO Feature Reader. Two possibilities have been identified for returning the feature reader. One idea is to add a new ExecuteFeatureReader (or similar name) method to FdoISQLCommand. The existing ExecuteReader method will remain as is for backwards compatibility. Another possibility is to keep the SQL pass through interface unchanged but provide a utility that will convert the returned SQL Data Reader into a Feature Reader after the fact.

A related question is around the class definition that would be generated from the SQL since it often would be a temporary class definition for the command and not something found from the FDO Describe Schema command. The feature reader would expose that class definition, but would applications need to get that earlier, e.g. with a new method on FdoISQLCommand to describe the generated class definition prior to getting the reader?

Parameter Direction

FDO already includes API’s for defining and using parameters (bind variables). However, there is no indication of direction (input versus output) and if an application is planning to use stored procedure calls in the SQL pass through, they would need to provide some indication of input, output, or return parameters. A possibility is to add a new FdoParameterDirection enumeration and add set/get methods to the FdoParameterValue or FdoParameterValueCollection interface.

A related question is whether FdoISQLCommand needs the application to tell it whether the SQL command being executed is a stored procedure call or some other type of SQL? We want to avoid having providers be forced to parse SQL where currently they probably all just send the SQL directly to the underlying server. The ExecuteNonQuery versus ExecuteReader methods that the application uses already tell the provider whether to expect results back from the SQL or not. A stored procedure call is a “non-query” but can return results, so is it another case?

 

 


_______________________________________________
fdo-internals mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/fdo-internals
Orest Halustchak

RE: Use of SQL pass through

Reply Threaded More More options
Print post
Permalink
Some javascript/style in this post has been disabled (why?)

Hi,

 

There hasn’t been too much feedback on this.

 

Haris, for SQL selects, to return data in a feature reader, we either can simply describe the results of the select and produce a temporary fdo class for the reader or if the SQL select happens to correspond cleanly to an existing feature class (e.g. something as simple as select * from roads), then we can use that fdo class instead. The goal is to make it easier for application developers that can use either fdo selects or sql selects and process the results with a common reader.

 

Anybody else have comments? I know we got some feedback from Jackie.

 

Thanks,

Orest.

 

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Greg Boone
Sent: Thursday, June 04, 2009 11:39 AM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] Use of SQL pass through

 

Hi Haris,

 

Can you provide some examples of non-DDL use cases that would make the proposal too complex? I also feel we should limit the discussion at this time to read only DML statements such as Select.

 

Greg

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Haris Kurtagic
Sent: Thursday, June 04, 2009 5:41 AM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] Use of SQL pass through

 

I would like to divide this and answer in two parts.

 

Part with Parameters direction is something we need. As it is already written , mostly to be able to execute and get results from stored procedures.

 

About executing SQL statements and trying to squeeze result of it inside FDO class/schema, I think it is too complex and in my mind without chance to be successful. There is so many cases in which it can't be done properly. If we are missing some info about result of execution of SQL we could look into existing specs like ODBC and add those. I agree FDO application should be able to get all necessary info about executed SQL so app can be written in generic way but I don't see putting that info in FDO class or reengineering sql etc.. I believe what can be done with api's like odbc is ok for fdo api too.

 

Haris

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Orest Halustchak
Sent: Wednesday, June 03, 2009 3:28 PM
To: FDO Internals Mail List
Subject: [fdo-internals] Use of SQL pass through

 

Hi,

 

I would like to solicit feedback, discussion, and suggestions on requirements and issues that we are seeing with the FdoISQLCommand which is used for the purpose of SQL pass through for some of the RDBMS-based providers.

 

Please see the discussion below.

 

Thanks,

Orest.

 

 

 

Enhanced SQL Command Support Discussion

Overview

The FDO API currently defines support for a SQL command that allows for pass-through execution of SQL statements either through a non-query execution of the SQL statement directly in the underlying Data Store, or through a query mechanism that returns a simple data reader listing the properties returned as a result of the SQL execution. The definition of the SQL command has remained fairly static over the last number of releases as primary development efforts have focused on extending other aspects of the FDO API, implementing new providers, etc. However, requirements for change to the SQL command have accumulated for us as RDBMS providers have implemented SQL pass through support and clients have attempted to integrate use SQL pass through into their applications in a seamless manner. While we generally expect applications to be using the generic FDO Select and other commands using FDO expressions, application still need to execute direct SQL against RDBMS-based providers on an exception basis for things that cannot do through FDO. While we hope to improve the FDO expression capabilities over time, there will always be this need for direct SQL processing for exception cases.

This email is meant to start discussion on how to handle the requirements that we’ve been seeing and to get feedback on how to modify the FDO api to handle these requirements. We have some ideas as described below, but would like to get other feedback. We’ll draft an RFC once we get close to a consensus on how it should look.

Requirements

One key request has been the desire to have the FDO API support SQL pass-through commands that return an FDO feature reader, referencing a proper FDO schema, not simply an FDO data reader. The feature reader will contain proper geometry properties, relations and associations. This enhancement is also intended to allow client applications that use FDO Feature Readers to code their applications in a generic manner so that data coming back from Select or SQL Pass Through statements can be processed in a uniform manner, thus reducing complexity, costs and time to implement.

Other more SQL specific requirements related to this have also arisen, including:

  • A need to allow a client to set the size of the fetch array an FDO provider will use when executing SQL statements ([OH] I still have an issue with this one. Fetch sizes are internal tuning parameters and are not FDO api concepts. Other providers that deal with select or insert buffering have default internal sizes.)
  • A need to specify bind variables for the SQL command, including arrays of bind variables. Since SQL commands may include stored procedure execution, bind variables need indication of whether they are input, output, or return parameters.

Discussion

In general, the intent of SQL pass-through is to deal with physical schemas. There is no parsing of the SQL statements, and what are identified are physical schema tables, columns, functions, commands, etc. The SQL statement can be any SQL command, not just select, but any DML (select, insert, update, delete) or any DDL (create, drop, alter, …). However, FDO feature readers deal with FDO logical schemas, which is at a different conceptual level. It’s the mixing of these levels that causes difficulty for applications using the FDO API. Applications are required to use different code pathways to handle select statement as opposed to direct SQL execution. If clients could use the result of either of these types of operations in a generic routine or component, such as a Data Table or Symbolization packages, applications would be provided a much more seamless and dynamic mechanism on which they can build and extend their applications.

To a certain degree, the current FDO feature reader implementation assumes an FDO class definition is directly related to the properties being returned. With physical schema SQL (let’s say just Select command), there isn’t necessarily any FDO class definition that applies. This is why currently the SQL command's Execute method returns a data reader, which handles any generically returned data. Note that the FDO select aggregates command doesn’t return a feature reader either, since it’s returning computed data.

One response to this issue has been to suggest that providers reverse engineer the select results and attempt to find the FDO class definition (coming from a describe schema request) that matches it. Other suggestions have been to construct a class definition on the fly, one that would not result from a call to describe schema. If the select is from a single table, providers would find the class definition that is based on that table and use it. However, then matching up the columns being selected with the properties in the logical schema, there may be some slight differences. This would result in the mixing of physical schema and logical schema elements together. For example, class names that are not the same as table names, property names that are not the same as column names, the use of additional computed columns, inclusion of pkey columns, etc. Granted that in many cases, the logical and physical views would be virtually identical. However, since that cannot be guaranteed, the design that is adopted will have to be able to handle the general case.

To account for these scenarios, providers will need to be modified to reverse engineer class definitions from the selected data and not attempt to match the select request to an existing FDO class. In reality, FDO Providers will be required to handle this in any case since a generic SQL select may not match up at all with an existing FDO class. An example of this can be found when selecting from a table with an owner that is different from the connected data store. For example, Connect to data store called Denver and select from Boulder.Roads – the schema may or may not be similar to Denver.Roads. In these types of circumstances, providers should manufacture a new class definition.

In cases where the resulting columns come from an existing FDO feature table, a provider can return the class definition corresponding to that table. In cases where the columns come from an unknown table, a class definition can be constructed on the fly. By definition, the FDO class definition returned by a feature reader does not necessarily correspond exactly to an existing FDO class definition. Existing class definitions may contain the properties that were asked for in the Select command, plus additional computed properties. It is perfectly legal to return a constructed class definition, which is only valid for the select that was executed, and not usable for further updates or inserts.

In the circumstance that a computed class is generated, the FDO class definition’s IsComputed property will return true. In this manner, applications are able to distinguish the content of the feature reader responses coming from the providers and tailor their implementation accordingly. In such a situation, some care will also need to be given to the name of the generated FDO schema and class definitions. At this point no standards exist for naming auto-generated schema, class and property names. It would be beneficial if, as a result of this RFC, some uniform naming conventions could be adopted.

Providers that do return feature readers from SQL commands will need to come up with the appropriate class definition that the feature reader could expose. Here are a few general use cases:

  • Select is against a table that has an existing class definition and the select returns the same information as defined by the class definition (e.g. select * from roads).
  • Select is against a table that has an existing class definition, but the select returns other information such as a subset of properties or additional computed properties. The FDO select command handles this type of case as well. The class definition returned by the feature reader includes the specific properties for that select.
  • Select is against a table that doesn’t have an existing class definition. Providers will be required to generate a temporary class definition.
  • Select is against a table that has an existing class definition, but the select is an aggregation that doesn’t return actual objects (e.g. select count(*), max(length) from roads). In this case it doesn’t make sense for the provider to return the same class name as for the underlying table since it is not actually returning road objects at all. This is basically another temporary class definition.

FDO API Ideas

In order to support the SQL pass-through objective outlined above, the FDO API will need to be modified to provide a new capability function to state that this is supported, and to provide a way to return an explicit FDO Feature Reader. Two possibilities have been identified for returning the feature reader. One idea is to add a new ExecuteFeatureReader (or similar name) method to FdoISQLCommand. The existing ExecuteReader method will remain as is for backwards compatibility. Another possibility is to keep the SQL pass through interface unchanged but provide a utility that will convert the returned SQL Data Reader into a Feature Reader after the fact.

A related question is around the class definition that would be generated from the SQL since it often would be a temporary class definition for the command and not something found from the FDO Describe Schema command. The feature reader would expose that class definition, but would applications need to get that earlier, e.g. with a new method on FdoISQLCommand to describe the generated class definition prior to getting the reader?

Parameter Direction

FDO already includes API’s for defining and using parameters (bind variables). However, there is no indication of direction (input versus output) and if an application is planning to use stored procedure calls in the SQL pass through, they would need to provide some indication of input, output, or return parameters. A possibility is to add a new FdoParameterDirection enumeration and add set/get methods to the FdoParameterValue or FdoParameterValueCollection interface.

A related question is whether FdoISQLCommand needs the application to tell it whether the SQL command being executed is a stored procedure call or some other type of SQL? We want to avoid having providers be forced to parse SQL where currently they probably all just send the SQL directly to the underlying server. The ExecuteNonQuery versus ExecuteReader methods that the application uses already tell the provider whether to expect results back from the SQL or not. A stored procedure call is a “non-query” but can return results, so is it another case?

 

 


_______________________________________________
fdo-internals mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/fdo-internals
Haris Kurtagic

RE: Use of SQL pass through

Reply Threaded More More options
Print post
Permalink
Some javascript/style in this post has been disabled (why?)

Hi,

 

To return FDO class from any SQL will require extra processing from provider. This processing will in best case add unnecessary overhead to provider.

My suggestion is to enable application to get info about returned columns trough feature readers. Same way as non-fdo applications are doing it right now.

I don't see any advantage in trying to squeeze SQL statement result into FDO class and I can see many problems there.

 

Even opposite, I would prefer to optimize FDO commands. Instead trying to guess what type of SLQ statement application has executed or parsing current FDO filters it would be better that we would have very simple "query window of data" type of command in FDO. That is something not directly related to this RFC but it is direction where I think FDO could go. Less overhead and simpler commands for most usual  tasks.

 

Haris

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Orest Halustchak
Sent: Thursday, June 25, 2009 3:17 PM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] Use of SQL pass through

 

Hi,

 

There hasn’t been too much feedback on this.

 

Haris, for SQL selects, to return data in a feature reader, we either can simply describe the results of the select and produce a temporary fdo class for the reader or if the SQL select happens to correspond cleanly to an existing feature class (e.g. something as simple as select * from roads), then we can use that fdo class instead. The goal is to make it easier for application developers that can use either fdo selects or sql selects and process the results with a common reader.

 

Anybody else have comments? I know we got some feedback from Jackie.

 

Thanks,

Orest.

 

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Greg Boone
Sent: Thursday, June 04, 2009 11:39 AM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] Use of SQL pass through

 

Hi Haris,

 

Can you provide some examples of non-DDL use cases that would make the proposal too complex? I also feel we should limit the discussion at this time to read only DML statements such as Select.

 

Greg

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Haris Kurtagic
Sent: Thursday, June 04, 2009 5:41 AM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] Use of SQL pass through

 

I would like to divide this and answer in two parts.

 

Part with Parameters direction is something we need. As it is already written , mostly to be able to execute and get results from stored procedures.

 

About executing SQL statements and trying to squeeze result of it inside FDO class/schema, I think it is too complex and in my mind without chance to be successful. There is so many cases in which it can't be done properly. If we are missing some info about result of execution of SQL we could look into existing specs like ODBC and add those. I agree FDO application should be able to get all necessary info about executed SQL so app can be written in generic way but I don't see putting that info in FDO class or reengineering sql etc.. I believe what can be done with api's like odbc is ok for fdo api too.

 

Haris

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Orest Halustchak
Sent: Wednesday, June 03, 2009 3:28 PM
To: FDO Internals Mail List
Subject: [fdo-internals] Use of SQL pass through

 

Hi,

 

I would like to solicit feedback, discussion, and suggestions on requirements and issues that we are seeing with the FdoISQLCommand which is used for the purpose of SQL pass through for some of the RDBMS-based providers.

 

Please see the discussion below.

 

Thanks,

Orest.

 

 

 

Enhanced SQL Command Support Discussion

Overview

The FDO API currently defines support for a SQL command that allows for pass-through execution of SQL statements either through a non-query execution of the SQL statement directly in the underlying Data Store, or through a query mechanism that returns a simple data reader listing the properties returned as a result of the SQL execution. The definition of the SQL command has remained fairly static over the last number of releases as primary development efforts have focused on extending other aspects of the FDO API, implementing new providers, etc. However, requirements for change to the SQL command have accumulated for us as RDBMS providers have implemented SQL pass through support and clients have attempted to integrate use SQL pass through into their applications in a seamless manner. While we generally expect applications to be using the generic FDO Select and other commands using FDO expressions, application still need to execute direct SQL against RDBMS-based providers on an exception basis for things that cannot do through FDO. While we hope to improve the FDO expression capabilities over time, there will always be this need for direct SQL processing for exception cases.

This email is meant to start discussion on how to handle the requirements that we’ve been seeing and to get feedback on how to modify the FDO api to handle these requirements. We have some ideas as described below, but would like to get other feedback. We’ll draft an RFC once we get close to a consensus on how it should look.

Requirements

One key request has been the desire to have the FDO API support SQL pass-through commands that return an FDO feature reader, referencing a proper FDO schema, not simply an FDO data reader. The feature reader will contain proper geometry properties, relations and associations. This enhancement is also intended to allow client applications that use FDO Feature Readers to code their applications in a generic manner so that data coming back from Select or SQL Pass Through statements can be processed in a uniform manner, thus reducing complexity, costs and time to implement.

Other more SQL specific requirements related to this have also arisen, including:

  • A need to allow a client to set the size of the fetch array an FDO provider will use when executing SQL statements ([OH] I still have an issue with this one. Fetch sizes are internal tuning parameters and are not FDO api concepts. Other providers that deal with select or insert buffering have default internal sizes.)
  • A need to specify bind variables for the SQL command, including arrays of bind variables. Since SQL commands may include stored procedure execution, bind variables need indication of whether they are input, output, or return parameters.

Discussion

In general, the intent of SQL pass-through is to deal with physical schemas. There is no parsing of the SQL statements, and what are identified are physical schema tables, columns, functions, commands, etc. The SQL statement can be any SQL command, not just select, but any DML (select, insert, update, delete) or any DDL (create, drop, alter, …). However, FDO feature readers deal with FDO logical schemas, which is at a different conceptual level. It’s the mixing of these levels that causes difficulty for applications using the FDO API. Applications are required to use different code pathways to handle select statement as opposed to direct SQL execution. If clients could use the result of either of these types of operations in a generic routine or component, such as a Data Table or Symbolization packages, applications would be provided a much more seamless and dynamic mechanism on which they can build and extend their applications.

To a certain degree, the current FDO feature reader implementation assumes an FDO class definition is directly related to the properties being returned. With physical schema SQL (let’s say just Select command), there isn’t necessarily any FDO class definition that applies. This is why currently the SQL command's Execute method returns a data reader, which handles any generically returned data. Note that the FDO select aggregates command doesn’t return a feature reader either, since it’s returning computed data.

One response to this issue has been to suggest that providers reverse engineer the select results and attempt to find the FDO class definition (coming from a describe schema request) that matches it. Other suggestions have been to construct a class definition on the fly, one that would not result from a call to describe schema. If the select is from a single table, providers would find the class definition that is based on that table and use it. However, then matching up the columns being selected with the properties in the logical schema, there may be some slight differences. This would result in the mixing of physical schema and logical schema elements together. For example, class names that are not the same as table names, property names that are not the same as column names, the use of additional computed columns, inclusion of pkey columns, etc. Granted that in many cases, the logical and physical views would be virtually identical. However, since that cannot be guaranteed, the design that is adopted will have to be able to handle the general case.

To account for these scenarios, providers will need to be modified to reverse engineer class definitions from the selected data and not attempt to match the select request to an existing FDO class. In reality, FDO Providers will be required to handle this in any case since a generic SQL select may not match up at all with an existing FDO class. An example of this can be found when selecting from a table with an owner that is different from the connected data store. For example, Connect to data store called Denver and select from Boulder.Roads – the schema may or may not be similar to Denver.Roads. In these types of circumstances, providers should manufacture a new class definition.

In cases where the resulting columns come from an existing FDO feature table, a provider can return the class definition corresponding to that table. In cases where the columns come from an unknown table, a class definition can be constructed on the fly. By definition, the FDO class definition returned by a feature reader does not necessarily correspond exactly to an existing FDO class definition. Existing class definitions may contain the properties that were asked for in the Select command, plus additional computed properties. It is perfectly legal to return a constructed class definition, which is only valid for the select that was executed, and not usable for further updates or inserts.

In the circumstance that a computed class is generated, the FDO class definition’s IsComputed property will return true. In this manner, applications are able to distinguish the content of the feature reader responses coming from the providers and tailor their implementation accordingly. In such a situation, some care will also need to be given to the name of the generated FDO schema and class definitions. At this point no standards exist for naming auto-generated schema, class and property names. It would be beneficial if, as a result of this RFC, some uniform naming conventions could be adopted.

Providers that do return feature readers from SQL commands will need to come up with the appropriate class definition that the feature reader could expose. Here are a few general use cases:

  • Select is against a table that has an existing class definition and the select returns the same information as defined by the class definition (e.g. select * from roads).
  • Select is against a table that has an existing class definition, but the select returns other information such as a subset of properties or additional computed properties. The FDO select command handles this type of case as well. The class definition returned by the feature reader includes the specific properties for that select.
  • Select is against a table that doesn’t have an existing class definition. Providers will be required to generate a temporary class definition.
  • Select is against a table that has an existing class definition, but the select is an aggregation that doesn’t return actual objects (e.g. select count(*), max(length) from roads). In this case it doesn’t make sense for the provider to return the same class name as for the underlying table since it is not actually returning road objects at all. This is basically another temporary class definition.

FDO API Ideas

In order to support the SQL pass-through objective outlined above, the FDO API will need to be modified to provide a new capability function to state that this is supported, and to provide a way to return an explicit FDO Feature Reader. Two possibilities have been identified for returning the feature reader. One idea is to add a new ExecuteFeatureReader (or similar name) method to FdoISQLCommand. The existing ExecuteReader method will remain as is for backwards compatibility. Another possibility is to keep the SQL pass through interface unchanged but provide a utility that will convert the returned SQL Data Reader into a Feature Reader after the fact.

A related question is around the class definition that would be generated from the SQL since it often would be a temporary class definition for the command and not something found from the FDO Describe Schema command. The feature reader would expose that class definition, but would applications need to get that earlier, e.g. with a new method on FdoISQLCommand to describe the generated class definition prior to getting the reader?

Parameter Direction

FDO already includes API’s for defining and using parameters (bind variables). However, there is no indication of direction (input versus output) and if an application is planning to use stored procedure calls in the SQL pass through, they would need to provide some indication of input, output, or return parameters. A possibility is to add a new FdoParameterDirection enumeration and add set/get methods to the FdoParameterValue or FdoParameterValueCollection interface.

A related question is whether FdoISQLCommand needs the application to tell it whether the SQL command being executed is a stored procedure call or some other type of SQL? We want to avoid having providers be forced to parse SQL where currently they probably all just send the SQL directly to the underlying server. The ExecuteNonQuery versus ExecuteReader methods that the application uses already tell the provider whether to expect results back from the SQL or not. A stored procedure call is a “non-query” but can return results, so is it another case?

 

 


_______________________________________________
fdo-internals mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/fdo-internals
Orest Halustchak

RE: Use of SQL pass through

Reply Threaded More More options
Print post
Permalink
Some javascript/style in this post has been disabled (why?)

Hi,

 

But the class definition includes additional information that you wouldn’t find just by getting information a property at a time. The idea with the feature reader is that it is returning “objects” based on a class definition rather than arbitrary collections of columns. The addition of getting a feature reader from the SQL command is intended to be optional for the caller, they don’t need to use it if they don’t want. But the class definition includes extra information such as the identity properties (primary key) so that the caller can determine the unique identity of these objects, which is important if the caller is going to draw these objects and then enable a user to edit them or go back to the data store for other information about the object.

 

Thanks,

Orest.

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Haris Kurtagic
Sent: Thursday, June 25, 2009 10:10 AM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] Use of SQL pass through

 

Hi,

 

To return FDO class from any SQL will require extra processing from provider. This processing will in best case add unnecessary overhead to provider.

My suggestion is to enable application to get info about returned columns trough feature readers. Same way as non-fdo applications are doing it right now.

I don't see any advantage in trying to squeeze SQL statement result into FDO class and I can see many problems there.

 

Even opposite, I would prefer to optimize FDO commands. Instead trying to guess what type of SLQ statement application has executed or parsing current FDO filters it would be better that we would have very simple "query window of data" type of command in FDO. That is something not directly related to this RFC but it is direction where I think FDO could go. Less overhead and simpler commands for most usual  tasks.

 

Haris

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Orest Halustchak
Sent: Thursday, June 25, 2009 3:17 PM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] Use of SQL pass through

 

Hi,

 

There hasn’t been too much feedback on this.

 

Haris, for SQL selects, to return data in a feature reader, we either can simply describe the results of the select and produce a temporary fdo class for the reader or if the SQL select happens to correspond cleanly to an existing feature class (e.g. something as simple as select * from roads), then we can use that fdo class instead. The goal is to make it easier for application developers that can use either fdo selects or sql selects and process the results with a common reader.

 

Anybody else have comments? I know we got some feedback from Jackie.

 

Thanks,

Orest.

 

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Greg Boone
Sent: Thursday, June 04, 2009 11:39 AM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] Use of SQL pass through

 

Hi Haris,

 

Can you provide some examples of non-DDL use cases that would make the proposal too complex? I also feel we should limit the discussion at this time to read only DML statements such as Select.

 

Greg

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Haris Kurtagic
Sent: Thursday, June 04, 2009 5:41 AM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] Use of SQL pass through

 

I would like to divide this and answer in two parts.

 

Part with Parameters direction is something we need. As it is already written , mostly to be able to execute and get results from stored procedures.

 

About executing SQL statements and trying to squeeze result of it inside FDO class/schema, I think it is too complex and in my mind without chance to be successful. There is so many cases in which it can't be done properly. If we are missing some info about result of execution of SQL we could look into existing specs like ODBC and add those. I agree FDO application should be able to get all necessary info about executed SQL so app can be written in generic way but I don't see putting that info in FDO class or reengineering sql etc.. I believe what can be done with api's like odbc is ok for fdo api too.

 

Haris

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Orest Halustchak
Sent: Wednesday, June 03, 2009 3:28 PM
To: FDO Internals Mail List
Subject: [fdo-internals] Use of SQL pass through

 

Hi,

 

I would like to solicit feedback, discussion, and suggestions on requirements and issues that we are seeing with the FdoISQLCommand which is used for the purpose of SQL pass through for some of the RDBMS-based providers.

 

Please see the discussion below.

 

Thanks,

Orest.

 

… snipped …

 


_______________________________________________
fdo-internals mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/fdo-internals
Haris Kurtagic

RE: Use of SQL pass through

Reply Threaded More More options
Print post
Permalink
Some javascript/style in this post has been disabled (why?)

Orest,

 

Provider would return primary key from arbitrary SQL statement ? This RFC is about executing any type of SQL statement and returning some type of FDO class for it, if so then no primary key there.

 

" don’t need to use it if they don’t want", but provider would need to do some processing for it and it eventually would not be used.

 

Class definition can include information which it can extract from database after executing SQL statement. Objects to be created can include this information.

Again, I don't see additional value if those information about executed SQL statement would be put inside FDO object (in many cases it wouldn't be even possible or even desirable) over what is currently posible with "regular" database drivers. I can see lot of additional work to make it possible and even than in many cases those won't be used. There is also performance penalty.

 

Haris

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Orest Halustchak
Sent: Thursday, June 25, 2009 4:35 PM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] Use of SQL pass through

 

Hi,

 

But the class definition includes additional information that you wouldn’t find just by getting information a property at a time. The idea with the feature reader is that it is returning “objects” based on a class definition rather than arbitrary collections of columns. The addition of getting a feature reader from the SQL command is intended to be optional for the caller, they don’t need to use it if they don’t want. But the class definition includes extra information such as the identity properties (primary key) so that the caller can determine the unique identity of these objects, which is important if the caller is going to draw these objects and then enable a user to edit them or go back to the data store for other information about the object.

 

Thanks,

Orest.

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Haris Kurtagic
Sent: Thursday, June 25, 2009 10:10 AM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] Use of SQL pass through

 

Hi,

 

To return FDO class from any SQL will require extra processing from provider. This processing will in best case add unnecessary overhead to provider.

My suggestion is to enable application to get info about returned columns trough feature readers. Same way as non-fdo applications are doing it right now.

I don't see any advantage in trying to squeeze SQL statement result into FDO class and I can see many problems there.

 

Even opposite, I would prefer to optimize FDO commands. Instead trying to guess what type of SLQ statement application has executed or parsing current FDO filters it would be better that we would have very simple "query window of data" type of command in FDO. That is something not directly related to this RFC but it is direction where I think FDO could go. Less overhead and simpler commands for most usual  tasks.

 

Haris

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Orest Halustchak
Sent: Thursday, June 25, 2009 3:17 PM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] Use of SQL pass through

 

Hi,

 

There hasn’t been too much feedback on this.

 

Haris, for SQL selects, to return data in a feature reader, we either can simply describe the results of the select and produce a temporary fdo class for the reader or if the SQL select happens to correspond cleanly to an existing feature class (e.g. something as simple as select * from roads), then we can use that fdo class instead. The goal is to make it easier for application developers that can use either fdo selects or sql selects and process the results with a common reader.

 

Anybody else have comments? I know we got some feedback from Jackie.

 

Thanks,

Orest.

 

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Greg Boone
Sent: Thursday, June 04, 2009 11:39 AM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] Use of SQL pass through

 

Hi Haris,

 

Can you provide some examples of non-DDL use cases that would make the proposal too complex? I also feel we should limit the discussion at this time to read only DML statements such as Select.

 

Greg

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Haris Kurtagic
Sent: Thursday, June 04, 2009 5:41 AM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] Use of SQL pass through

 

I would like to divide this and answer in two parts.

 

Part with Parameters direction is something we need. As it is already written , mostly to be able to execute and get results from stored procedures.

 

About executing SQL statements and trying to squeeze result of it inside FDO class/schema, I think it is too complex and in my mind without chance to be successful. There is so many cases in which it can't be done properly. If we are missing some info about result of execution of SQL we could look into existing specs like ODBC and add those. I agree FDO application should be able to get all necessary info about executed SQL so app can be written in generic way but I don't see putting that info in FDO class or reengineering sql etc.. I believe what can be done with api's like odbc is ok for fdo api too.

 

Haris

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Orest Halustchak
Sent: Wednesday, June 03, 2009 3:28 PM
To: FDO Internals Mail List
Subject: [fdo-internals] Use of SQL pass through

 

Hi,

 

I would like to solicit feedback, discussion, and suggestions on requirements and issues that we are seeing with the FdoISQLCommand which is used for the purpose of SQL pass through for some of the RDBMS-based providers.

 

Please see the discussion below.

 

Thanks,

Orest.

 

… snipped …

 


_______________________________________________
fdo-internals mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/fdo-internals
Alan Quinton

RE: Use of SQL pass through

Reply Threaded More More options
Print post
Permalink
Some javascript/style in this post has been disabled (why?)

Hi,

   I don’t understand what you mean by “extra processing” or “overhead” – do you mean memory overhead or performance or both? Because if you don’t call GetClassDefinition on the feature reader, then you are not incurring anything, other having to take the trouble to write the function. And I think there is some guidance there as to how to do this, even for arbitrary columns. So yes, you would have to create this class definition object on the fly, if the client called the function, which they would only need to do once presumably, when they get the reader. So I’m not understanding where the performance problem is. Are you saying that building this class def once would be a performance problem?

 

As far as I’m aware, column metadata is only available from class definitions, so instead of doing that, are you suggesting FDO define an entirely new way to get column metadata? That’s how I read: “My suggestion is to enable application to get info about returned columns trough feature readers”.

 

The point is, as Orest stated, to process the results of both fdo selects and sql selects with a common reader.  Downstream code that draws or edits features and which works off FDO feature readers would not need to be modified. It would ‘just work”. For example, you might define an SQL select which selects the primary key and geometry from one table, then joins to other tables, using sub-selects and any features of SQL that the data store supports. If you produced a feature reader for this data, clients could draw this data, stylize it, and even potentially edit it without having to modify their code that does this since it just works off a feature reader with a “pseudo” class definition. This is a powerful capability since it essentially lets you define dynamic views of data using SQL.

 

Thanks,
Alan

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Haris Kurtagic
Sent: Thursday, June 25, 2009 8:05 AM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] Use of SQL pass through

 

Orest,

 

Provider would return primary key from arbitrary SQL statement ? This RFC is about executing any type of SQL statement and returning some type of FDO class for it, if so then no primary key there.

 

" don’t need to use it if they don’t want", but provider would need to do some processing for it and it eventually would not be used.

 

Class definition can include information which it can extract from database after executing SQL statement. Objects to be created can include this information.

Again, I don't see additional value if those information about executed SQL statement would be put inside FDO object (in many cases it wouldn't be even possible or even desirable) over what is currently posible with "regular" database drivers. I can see lot of additional work to make it possible and even than in many cases those won't be used. There is also performance penalty.

 

Haris

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Orest Halustchak
Sent: Thursday, June 25, 2009 4:35 PM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] Use of SQL pass through

 

Hi,

 

But the class definition includes additional information that you wouldn’t find just by getting information a property at a time. The idea with the feature reader is that it is returning “objects” based on a class definition rather than arbitrary collections of columns. The addition of getting a feature reader from the SQL command is intended to be optional for the caller, they don’t need to use it if they don’t want. But the class definition includes extra information such as the identity properties (primary key) so that the caller can determine the unique identity of these objects, which is important if the caller is going to draw these objects and then enable a user to edit them or go back to the data store for other information about the object.

 

Thanks,

Orest.

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Haris Kurtagic
Sent: Thursday, June 25, 2009 10:10 AM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] Use of SQL pass through

 

Hi,

 

To return FDO class from any SQL will require extra processing from provider. This processing will in best case add unnecessary overhead to provider.

My suggestion is to enable application to get info about returned columns trough feature readers. Same way as non-fdo applications are doing it right now.

I don't see any advantage in trying to squeeze SQL statement result into FDO class and I can see many problems there.

 

Even opposite, I would prefer to optimize FDO commands. Instead trying to guess what type of SLQ statement application has executed or parsing current FDO filters it would be better that we would have very simple "query window of data" type of command in FDO. That is something not directly related to this RFC but it is direction where I think FDO could go. Less overhead and simpler commands for most usual  tasks.

 

Haris

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Orest Halustchak
Sent: Thursday, June 25, 2009 3:17 PM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] Use of SQL pass through

 

Hi,

 

There hasn’t been too much feedback on this.

 

Haris, for SQL selects, to return data in a feature reader, we either can simply describe the results of the select and produce a temporary fdo class for the reader or if the SQL select happens to correspond cleanly to an existing feature class (e.g. something as simple as select * from roads), then we can use that fdo class instead. The goal is to make it easier for application developers that can use either fdo selects or sql selects and process the results with a common reader.

 

Anybody else have comments? I know we got some feedback from Jackie.

 

Thanks,

Orest.

 

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Greg Boone
Sent: Thursday, June 04, 2009 11:39 AM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] Use of SQL pass through

 

Hi Haris,

 

Can you provide some examples of non-DDL use cases that would make the proposal too complex? I also feel we should limit the discussion at this time to read only DML statements such as Select.

 

Greg

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Haris Kurtagic
Sent: Thursday, June 04, 2009 5:41 AM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] Use of SQL pass through

 

I would like to divide this and answer in two parts.

 

Part with Parameters direction is something we need. As it is already written , mostly to be able to execute and get results from stored procedures.

 

About executing SQL statements and trying to squeeze result of it inside FDO class/schema, I think it is too complex and in my mind without chance to be successful. There is so many cases in which it can't be done properly. If we are missing some info about result of execution of SQL we could look into existing specs like ODBC and add those. I agree FDO application should be able to get all necessary info about executed SQL so app can be written in generic way but I don't see putting that info in FDO class or reengineering sql etc.. I believe what can be done with api's like odbc is ok for fdo api too.

 

Haris

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Orest Halustchak
Sent: Wednesday, June 03, 2009 3:28 PM
To: FDO Internals Mail List
Subject: [fdo-internals] Use of SQL pass through

 

Hi,

 

I would like to solicit feedback, discussion, and suggestions on requirements and issues that we are seeing with the FdoISQLCommand which is used for the purpose of SQL pass through for some of the RDBMS-based providers.

 

Please see the discussion below.

 

Thanks,

Orest.

 

… snipped …

 


_______________________________________________
fdo-internals mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/fdo-internals
Haris Kurtagic

RE: Use of SQL pass through

Reply Threaded More More options
Print post
Permalink
Some javascript/style in this post has been disabled (why?)

Hi Alan,

By extra processing and overhead I mean code to be executed to try to create FDO class from arbitrary SQL statement.

 

I am suggesting to add functions ((if missing) to retrieve column info about executed sql statement.

 

Most of current code won't work. You give could example: "Would select primary key" - no normal way for provider to understand it is primary key selected in arbitrary SQL statement and set it as such in FDO class.

 

If we are talking about use case where application would allow end-user to create arbitrary SQL to be executed and display I don't see any problem with executing SQL statement, Iterating trough columns, getting info about columns and display geometry and attributes. I really don't see point in provider in name of application iterating trough columns and try to create FDO class.

 

 

I may not understand well what is your goal but :

 

- if it is to have only one type of reader then I think different approach is needed.

 

- if it is to create application which would allow end-user to enter specific select sql statement and create fdo class for it and  later use it as other fdo classes, I think then it is up to application to create such fdo class.   Such class would be created using column info returned by reader and  in some cases with some additional interaction with end-user.

 

I think it is very important for FDO to be simple and exact as much as possible. Complicating FDO and providers with some specific use cases is something to be avoided.

 

Haris

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Alan Quinton
Sent: Thursday, June 25, 2009 6:20 PM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] Use of SQL pass through

 

Hi,

   I don’t understand what you mean by “extra processing” or “overhead” – do you mean memory overhead or performance or both? Because if you don’t call GetClassDefinition on the feature reader, then you are not incurring anything, other having to take the trouble to write the function. And I think there is some guidance there as to how to do this, even for arbitrary columns. So yes, you would have to create this class definition object on the fly, if the client called the function, which they would only need to do once presumably, when they get the reader. So I’m not understanding where the performance problem is. Are you saying that building this class def once would be a performance problem?

 

As far as I’m aware, column metadata is only available from class definitions, so instead of doing that, are you suggesting FDO define an entirely new way to get column metadata? That’s how I read: “My suggestion is to enable application to get info about returned columns trough feature readers”.

 

The point is, as Orest stated, to process the results of both fdo selects and sql selects with a common reader.  Downstream code that draws or edits features and which works off FDO feature readers would not need to be modified. It would ‘just work”. For example, you might define an SQL select which selects the primary key and geometry from one table, then joins to other tables, using sub-selects and any features of SQL that the data store supports. If you produced a feature reader for this data, clients could draw this data, stylize it, and even potentially edit it without having to modify their code that does this since it just works off a feature reader with a “pseudo” class definition. This is a powerful capability since it essentially lets you define dynamic views of data using SQL.

 

Thanks,
Alan

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Haris Kurtagic
Sent: Thursday, June 25, 2009 8:05 AM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] Use of SQL pass through

 

Orest,

 

Provider would return primary key from arbitrary SQL statement ? This RFC is about executing any type of SQL statement and returning some type of FDO class for it, if so then no primary key there.

 

" don’t need to use it if they don’t want", but provider would need to do some processing for it and it eventually would not be used.

 

Class definition can include information which it can extract from database after executing SQL statement. Objects to be created can include this information.

Again, I don't see additional value if those information about executed SQL statement would be put inside FDO object (in many cases it wouldn't be even possible or even desirable) over what is currently posible with "regular" database drivers. I can see lot of additional work to make it possible and even than in many cases those won't be used. There is also performance penalty.

 

Haris

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Orest Halustchak
Sent: Thursday, June 25, 2009 4:35 PM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] Use of SQL pass through

 

Hi,

 

But the class definition includes additional information that you wouldn’t find just by getting information a property at a time. The idea with the feature reader is that it is returning “objects” based on a class definition rather than arbitrary collections of columns. The addition of getting a feature reader from the SQL command is intended to be optional for the caller, they don’t need to use it if they don’t want. But the class definition includes extra information such as the identity properties (primary key) so that the caller can determine the unique identity of these objects, which is important if the caller is going to draw these objects and then enable a user to edit them or go back to the data store for other information about the object.

 

Thanks,

Orest.

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Haris Kurtagic
Sent: Thursday, June 25, 2009 10:10 AM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] Use of SQL pass through

 

Hi,

 

To return FDO class from any SQL will require extra processing from provider. This processing will in best case add unnecessary overhead to provider.

My suggestion is to enable application to get info about returned columns trough feature readers. Same way as non-fdo applications are doing it right now.

I don't see any advantage in trying to squeeze SQL statement result into FDO class and I can see many problems there.

 

Even opposite, I would prefer to optimize FDO commands. Instead trying to guess what type of SLQ statement application has executed or parsing current FDO filters it would be better that we would have very simple "query window of data" type of command in FDO. That is something not directly related to this RFC but it is direction where I think FDO could go. Less overhead and simpler commands for most usual  tasks.

 

Haris

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Orest Halustchak
Sent: Thursday, June 25, 2009 3:17 PM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] Use of SQL pass through

 

Hi,

 

There hasn’t been too much feedback on this.

 

Haris, for SQL selects, to return data in a feature reader, we either can simply describe the results of the select and produce a temporary fdo class for the reader or if the SQL select happens to correspond cleanly to an existing feature class (e.g. something as simple as select * from roads), then we can use that fdo class instead. The goal is to make it easier for application developers that can use either fdo selects or sql selects and process the results with a common reader.

 

Anybody else have comments? I know we got some feedback from Jackie.

 

Thanks,

Orest.

 

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Greg Boone
Sent: Thursday, June 04, 2009 11:39 AM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] Use of SQL pass through

 

Hi Haris,

 

Can you provide some examples of non-DDL use cases that would make the proposal too complex? I also feel we should limit the discussion at this time to read only DML statements such as Select.

 

Greg

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Haris Kurtagic
Sent: Thursday, June 04, 2009 5:41 AM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] Use of SQL pass through

 

I would like to divide this and answer in two parts.

 

Part with Parameters direction is something we need. As it is already written , mostly to be able to execute and get results from stored procedures.

 

About executing SQL statements and trying to squeeze result of it inside FDO class/schema, I think it is too complex and in my mind without chance to be successful. There is so many cases in which it can't be done properly. If we are missing some info about result of execution of SQL we could look into existing specs like ODBC and add those. I agree FDO application should be able to get all necessary info about executed SQL so app can be written in generic way but I don't see putting that info in FDO class or reengineering sql etc.. I believe what can be done with api's like odbc is ok for fdo api too.

 

Haris

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Orest Halustchak
Sent: Wednesday, June 03, 2009 3:28 PM
To: FDO Internals Mail List
Subject: [fdo-internals] Use of SQL pass through

 

Hi,

 

I would like to solicit feedback, discussion, and suggestions on requirements and issues that we are seeing with the FdoISQLCommand which is used for the purpose of SQL pass through for some of the RDBMS-based providers.

 

Please see the discussion below.

 

Thanks,

Orest.

 

… snipped …

 


_______________________________________________
fdo-internals mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/fdo-internals
Orest Halustchak

RE: Use of SQL pass through

Reply Threaded More More options
Print post
Permalink
Some javascript/style in this post has been disabled (why?)

Hi,

 

Well, the goal is to get back one reader, a feature reader in this case, so that the client code is simplified. We’re not specifically trying to replace both the data reader and feature reader with one reader as in your first bullet, however, maybe that is an alternative. What suggestions would you have around that? I don’t want to lose the concept that a feature reader is returning “objects” based on some class definition.

 

Thanks,

Orest.

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Haris Kurtagic
Sent: Thursday, June 25, 2009 1:15 PM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] Use of SQL pass through

 

Hi Alan,

By extra processing and overhead I mean code to be executed to try to create FDO class from arbitrary SQL statement.

 

I am suggesting to add functions ((if missing) to retrieve column info about executed sql statement.

 

Most of current code won't work. You give could example: "Would select primary key" - no normal way for provider to understand it is primary key selected in arbitrary SQL statement and set it as such in FDO class.

 

If we are talking about use case where application would allow end-user to create arbitrary SQL to be executed and display I don't see any problem with executing SQL statement, Iterating trough columns, getting info about columns and display geometry and attributes. I really don't see point in provider in name of application iterating trough columns and try to create FDO class.

 

 

I may not understand well what is your goal but :

 

- if it is to have only one type of reader then I think different approach is needed.

 

- if it is to create application which would allow end-user to enter specific select sql statement and create fdo class for it and  later use it as other fdo classes, I think then it is up to application to create such fdo class.   Such class would be created using column info returned by reader and  in some cases with some additional interaction with end-user.

 

I think it is very important for FDO to be simple and exact as much as possible. Complicating FDO and providers with some specific use cases is something to be avoided.

 

Haris

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Alan Quinton
Sent: Thursday, June 25, 2009 6:20 PM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] Use of SQL pass through

 

Hi,

   I don’t understand what you mean by “extra processing” or “overhead” – do you mean memory overhead or performance or both? Because if you don’t call GetClassDefinition on the feature reader, then you are not incurring anything, other having to take the trouble to write the function. And I think there is some guidance there as to how to do this, even for arbitrary columns. So yes, you would have to create this class definition object on the fly, if the client called the function, which they would only need to do once presumably, when they get the reader. So I’m not understanding where the performance problem is. Are you saying that building this class def once would be a performance problem?

 

As far as I’m aware, column metadata is only available from class definitions, so instead of doing that, are you suggesting FDO define an entirely new way to get column metadata? That’s how I read: “My suggestion is to enable application to get info about returned columns trough feature readers”.

 

The point is, as Orest stated, to process the results of both fdo selects and sql selects with a common reader.  Downstream code that draws or edits features and which works off FDO feature readers would not need to be modified. It would ‘just work”. For example, you might define an SQL select which selects the primary key and geometry from one table, then joins to other tables, using sub-selects and any features of SQL that the data store supports. If you produced a feature reader for this data, clients could draw this data, stylize it, and even potentially edit it without having to modify their code that does this since it just works off a feature reader with a “pseudo” class definition. This is a powerful capability since it essentially lets you define dynamic views of data using SQL.

 

Thanks,
Alan

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Haris Kurtagic
Sent: Thursday, June 25, 2009 8:05 AM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] Use of SQL pass through

 

Orest,

 

Provider would return primary key from arbitrary SQL statement ? This RFC is about executing any type of SQL statement and returning some type of FDO class for it, if so then no primary key there.

 

" don’t need to use it if they don’t want", but provider would need to do some processing for it and it eventually would not be used.

 

Class definition can include information which it can extract from database after executing SQL statement. Objects to be created can include this information.

Again, I don't see additional value if those information about executed SQL statement would be put inside FDO object (in many cases it wouldn't be even possible or even desirable) over what is currently posible with "regular" database drivers. I can see lot of additional work to make it possible and even than in many cases those won't be used. There is also performance penalty.

 

Haris

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Orest Halustchak
Sent: Thursday, June 25, 2009 4:35 PM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] Use of SQL pass through

 

Hi,

 

But the class definition includes additional information that you wouldn’t find just by getting information a property at a time. The idea with the feature reader is that it is returning “objects” based on a class definition rather than arbitrary collections of columns. The addition of getting a feature reader from the SQL command is intended to be optional for the caller, they don’t need to use it if they don’t want. But the class definition includes extra information such as the identity properties (primary key) so that the caller can determine the unique identity of these objects, which is important if the caller is going to draw these objects and then enable a user to edit them or go back to the data store for other information about the object.

 

Thanks,

Orest.

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Haris Kurtagic
Sent: Thursday, June 25, 2009 10:10 AM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] Use of SQL pass through

 

Hi,

 

To return FDO class from any SQL will require extra processing from provider. This processing will in best case add unnecessary overhead to provider.

My suggestion is to enable application to get info about returned columns trough feature readers. Same way as non-fdo applications are doing it right now.

I don't see any advantage in trying to squeeze SQL statement result into FDO class and I can see many problems there.

 

Even opposite, I would prefer to optimize FDO commands. Instead trying to guess what type of SLQ statement application has executed or parsing current FDO filters it would be better that we would have very simple "query window of data" type of command in FDO. That is something not directly related to this RFC but it is direction where I think FDO could go. Less overhead and simpler commands for most usual  tasks.

 

Haris

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Orest Halustchak
Sent: Thursday, June 25, 2009 3:17 PM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] Use of SQL pass through

 

Hi,

 

There hasn’t been too much feedback on this.

 

Haris, for SQL selects, to return data in a feature reader, we either can simply describe the results of the select and produce a temporary fdo class for the reader or if the SQL select happens to correspond cleanly to an existing feature class (e.g. something as simple as select * from roads), then we can use that fdo class instead. The goal is to make it easier for application developers that can use either fdo selects or sql selects and process the results with a common reader.

 

Anybody else have comments? I know we got some feedback from Jackie.

 

Thanks,

Orest.

 

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Greg Boone
Sent: Thursday, June 04, 2009 11:39 AM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] Use of SQL pass through

 

Hi Haris,

 

Can you provide some examples of non-DDL use cases that would make the proposal too complex? I also feel we should limit the discussion at this time to read only DML statements such as Select.

 

Greg

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Haris Kurtagic
Sent: Thursday, June 04, 2009 5:41 AM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] Use of SQL pass through

 

I would like to divide this and answer in two parts.

 

Part with Parameters direction is something we need. As it is already written , mostly to be able to execute and get results from stored procedures.

 

About executing SQL statements and trying to squeeze result of it inside FDO class/schema, I think it is too complex and in my mind without chance to be successful. There is so many cases in which it can't be done properly. If we are missing some info about result of execution of SQL we could look into existing specs like ODBC and add those. I agree FDO application should be able to get all necessary info about executed SQL so app can be written in generic way but I don't see putting that info in FDO class or reengineering sql etc.. I believe what can be done with api's like odbc is ok for fdo api too.

 

Haris

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Orest Halustchak
Sent: Wednesday, June 03, 2009 3:28 PM
To: FDO Internals Mail List
Subject: [fdo-internals] Use of SQL pass through

 

Hi,

 

I would like to solicit feedback, discussion, and suggestions on requirements and issues that we are seeing with the FdoISQLCommand which is used for the purpose of SQL pass through for some of the RDBMS-based providers.

 

Please see the discussion below.

 

Thanks,

Orest.

 

… snipped …

 


_______________________________________________
fdo-internals mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/fdo-internals
Greg Boone

RE: Use of SQL pass through

Reply Threaded More More options
Print post
Permalink
Some javascript/style in this post has been disabled (why?)

Where are we on this thread?  Haris, can you summarize the remaining issues you have with RFC 33? We have to come to some agreement on how to proceed.

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Orest Halustchak
Sent: Thursday, June 25, 2009 1:35 PM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] Use of SQL pass through

 

Hi,

 

Well, the goal is to get back one reader, a feature reader in this case, so that the client code is simplified. We’re not specifically trying to replace both the data reader and feature reader with one reader as in your first bullet, however, maybe that is an alternative. What suggestions would you have around that? I don’t want to lose the concept that a feature reader is returning “objects” based on some class definition.

 

Thanks,

Orest.

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Haris Kurtagic
Sent: Thursday, June 25, 2009 1:15 PM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] Use of SQL pass through

 

Hi Alan,

By extra processing and overhead I mean code to be executed to try to create FDO class from arbitrary SQL statement.

 

I am suggesting to add functions ((if missing) to retrieve column info about executed sql statement.

 

Most of current code won't work. You give could example: "Would select primary key" - no normal way for provider to understand it is primary key selected in arbitrary SQL statement and set it as such in FDO class.

 

If we are talking about use case where application would allow end-user to create arbitrary SQL to be executed and display I don't see any problem with executing SQL statement, Iterating trough columns, getting info about columns and display geometry and attributes. I really don't see point in provider in name of application iterating trough columns and try to create FDO class.

 

 

I may not understand well what is your goal but :

 

- if it is to have only one type of reader then I think different approach is needed.

 

- if it is to create application which would allow end-user to enter specific select sql statement and create fdo class for it and  later use it as other fdo classes, I think then it is up to application to create such fdo class.   Such class would be created using column info returned by reader and  in some cases with some additional interaction with end-user.

 

I think it is very important for FDO to be simple and exact as much as possible. Complicating FDO and providers with some specific use cases is something to be avoided.

 

Haris

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Alan Quinton
Sent: Thursday, June 25, 2009 6:20 PM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] Use of SQL pass through

 

Hi,

   I don’t understand what you mean by “extra processing” or “overhead” – do you mean memory overhead or performance or both? Because if you don’t call GetClassDefinition on the feature reader, then you are not incurring anything, other having to take the trouble to write the function. And I think there is some guidance there as to how to do this, even for arbitrary columns. So yes, you would have to create this class definition object on the fly, if the client called the function, which they would only need to do once presumably, when they get the reader. So I’m not understanding where the performance problem is. Are you saying that building this class def once would be a performance problem?

 

As far as I’m aware, column metadata is only available from class definitions, so instead of doing that, are you suggesting FDO define an entirely new way to get column metadata? That’s how I read: “My suggestion is to enable application to get info about returned columns trough feature readers”.

 

The point is, as Orest stated, to process the results of both fdo selects and sql selects with a common reader.  Downstream code that draws or edits features and which works off FDO feature readers would not need to be modified. It would ‘just work”. For example, you might define an SQL select which selects the primary key and geometry from one table, then joins to other tables, using sub-selects and any features of SQL that the data store supports. If you produced a feature reader for this data, clients could draw this data, stylize it, and even potentially edit it without having to modify their code that does this since it just works off a feature reader with a “pseudo” class definition. This is a powerful capability since it essentially lets you define dynamic views of data using SQL.

 

Thanks,
Alan

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Haris Kurtagic
Sent: Thursday, June 25, 2009 8:05 AM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] Use of SQL pass through

 

Orest,

 

Provider would return primary key from arbitrary SQL statement ? This RFC is about executing any type of SQL statement and returning some type of FDO class for it, if so then no primary key there.

 

" don’t need to use it if they don’t want", but provider would need to do some processing for it and it eventually would not be used.

 

Class definition can include information which it can extract from database after executing SQL statement. Objects to be created can include this information.

Again, I don't see additional value if those information about executed SQL statement would be put inside FDO object (in many cases it wouldn't be even possible or even desirable) over what is currently posible with "regular" database drivers. I can see lot of additional work to make it possible and even than in many cases those won't be used. There is also performance penalty.

 

Haris

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Orest Halustchak
Sent: Thursday, June 25, 2009 4:35 PM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] Use of SQL pass through

 

Hi,

 

But the class definition includes additional information that you wouldn’t find just by getting information a property at a time. The idea with the feature reader is that it is returning “objects” based on a class definition rather than arbitrary collections of columns. The addition of getting a feature reader from the SQL command is intended to be optional for the caller, they don’t need to use it if they don’t want. But the class definition includes extra information such as the identity properties (primary key) so that the caller can determine the unique identity of these objects, which is important if the caller is going to draw these objects and then enable a user to edit them or go back to the data store for other information about the object.

 

Thanks,

Orest.

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Haris Kurtagic
Sent: Thursday, June 25, 2009 10:10 AM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] Use of SQL pass through

 

Hi,

 

To return FDO class from any SQL will require extra processing from provider. This processing will in best case add unnecessary overhead to provider.

My suggestion is to enable application to get info about returned columns trough feature readers. Same way as non-fdo applications are doing it right now.

I don't see any advantage in trying to squeeze SQL statement result into FDO class and I can see many problems there.

 

Even opposite, I would prefer to optimize FDO commands. Instead trying to guess what type of SLQ statement application has executed or parsing current FDO filters it would be better that we would have very simple "query window of data" type of command in FDO. That is something not directly related to this RFC but it is direction where I think FDO could go. Less overhead and simpler commands for most usual  tasks.

 

Haris

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Orest Halustchak
Sent: Thursday, June 25, 2009 3:17 PM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] Use of SQL pass through

 

Hi,

 

There hasn’t been too much feedback on this.

 

Haris, for SQL selects, to return data in a feature reader, we either can simply describe the results of the select and produce a temporary fdo class for the reader or if the SQL select happens to correspond cleanly to an existing feature class (e.g. something as simple as select * from roads), then we can use that fdo class instead. The goal is to make it easier for application developers that can use either fdo selects or sql selects and process the results with a common reader.

 

Anybody else have comments? I know we got some feedback from Jackie.

 

Thanks,

Orest.

 

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Greg Boone
Sent: Thursday, June 04, 2009 11:39 AM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] Use of SQL pass through

 

Hi Haris,

 

Can you provide some examples of non-DDL use cases that would make the proposal too complex? I also feel we should limit the discussion at this time to read only DML statements such as Select.

 

Greg

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Haris Kurtagic
Sent: Thursday, June 04, 2009 5:41 AM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] Use of SQL pass through

 

I would like to divide this and answer in two parts.

 

Part with Parameters direction is something we need. As it is already written , mostly to be able to execute and get results from stored procedures.

 

About executing SQL statements and trying to squeeze result of it inside FDO class/schema, I think it is too complex and in my mind without chance to be successful. There is so many cases in which it can't be done properly. If we are missing some info about result of execution of SQL we could look into existing specs like ODBC and add those. I agree FDO application should be able to get all necessary info about executed SQL so app can be written in generic way but I don't see putting that info in FDO class or reengineering sql etc.. I believe what can be done with api's like odbc is ok for fdo api too.

 

Haris

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Orest Halustchak
Sent: Wednesday, June 03, 2009 3:28 PM
To: FDO Internals Mail List
Subject: [fdo-internals] Use of SQL pass through

 

Hi,

 

I would like to solicit feedback, discussion, and suggestions on requirements and issues that we are seeing with the FdoISQLCommand which is used for the purpose of SQL pass through for some of the RDBMS-based providers.

 

Please see the discussion below.

 

Thanks,

Orest.

 

… snipped …

 


_______________________________________________
fdo-internals mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/fdo-internals
Jason Birch

RE: Use of SQL pass through

Reply Threaded More More options
Print post
Permalink
Some javascript/style in this post has been disabled (why?)

I think Haris may be out of email contact until some time next week.

 

From: Greg Boone
Sent: Wednesday, July 08, 2009 2:12 PM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] Use of SQL pass through

 

Where are we on this thread?  Haris, can you summarize the remaining issues you have with RFC 33? We have to come to some agreement on how to proceed.


_______________________________________________
fdo-internals mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/fdo-internals
Greg Boone

RE: Use of SQL pass through

Reply Threaded More More options
Print post
Permalink
Some javascript/style in this post has been disabled (why?)

Is it possible to work through Haris’ concerns this week? The resources that we have available to work towards the implementation of this enhancement will not be available indefinitely.

 

Are there alternate suggestions that are being proposed that would allow us to meet the same objectives?

 

Greg

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Jason Birch
Sent: Wednesday, July 08, 2009 5:46 PM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] Use of SQL pass through

 

I think Haris may be out of email contact until some time next week.

 

From: Greg Boone
Sent: Wednesday, July 08, 2009 2:12 PM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] Use of SQL pass through

 

Where are we on this thread?  Haris, can you summarize the remaining issues you have with RFC 33? We have to come to some agreement on how to proceed.


_______________________________________________
fdo-internals mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/fdo-internals