Sorting shapefile

[jericks]

I am trying to sort a shapefile or ArcSDE layer by an attribute.  I first tried adding a SortBy to a DefaultQuery, but the shapefile throws an error and the ArcSDE layer fails silently.  Next, I tried reading my FeatureSource into an ArrayList of SimpleFeatures, sorting them using a Comparator, and then writing the ArrayList back to an output FeatureStore, but the order is not preserved.  What is the correct way of sorting a FeatureSource?

Thanks,
Jared Erickson

[aaime]

jericks ha scritto:
> I am trying to sort a shapefile or ArcSDE layer by an attribute.  I first
> tried adding a SortBy to a DefaultQuery, but the shapefile throws an error
> and the ArcSDE layer fails silently.  Next, I tried reading my FeatureSource
> into an ArrayList of SimpleFeatures, sorting them using a Comparator, and
> then writing the ArrayList back to an output FeatureStore, but the order is
> not preserved.  What is the correct way of sorting a FeatureSource?

Native sorting is not supported on all stores, in particular, only
database backends do... not sure what goes on in SDE.
You can check at runtime if a feature source
supports sorting on a certain field by using
FeatureSource.getQueryCapabilities.supportsSorting(SortBy[]).

As for preserving order in writes, there is basically only shapefile
that do support it, but it's just an accident of how the writing
occurs. SDE is free to keep whatever order it wants storage wise,
and when you retrieve data the order will be very much dependent
on what indexes are used to execute the query.

So I guess in the end what you need is sort support in SDE.
By looking at the code in SDE it would seem sorting in never
supported (the query capabilities seem to always return false to
the above cited methos), but I may be wrong. I've cc'ed the module
maintainer, he should know more.

Cheers
Andrea


--
Andrea Aime
OpenGeo - http://opengeo.org
Expert service straight from the developers.

------------------------------------------------------------------------------
_______________________________________________
Geotools-gt2-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geotools-gt2-users

[Gabriel Roldan]

Andrea Aime wrote:

> jericks ha scritto:
>> I am trying to sort a shapefile or ArcSDE layer by an attribute.  I first
>> tried adding a SortBy to a DefaultQuery, but the shapefile throws an
>> error
>> and the ArcSDE layer fails silently.  Next, I tried reading my
>> FeatureSource
>> into an ArrayList of SimpleFeatures, sorting them using a Comparator, and
>> then writing the ArrayList back to an output FeatureStore, but the
>> order is
>> not preserved.  What is the correct way of sorting a FeatureSource?
>
> Native sorting is not supported on all stores, in particular, only
> database backends do... not sure what goes on in SDE.
> You can check at runtime if a feature source
> supports sorting on a certain field by using
> FeatureSource.getQueryCapabilities.supportsSorting(SortBy[]).
>
> As for preserving order in writes, there is basically only shapefile
> that do support it, but it's just an accident of how the writing
> occurs. SDE is free to keep whatever order it wants storage wise,
> and when you retrieve data the order will be very much dependent
> on what indexes are used to execute the query.
>
> So I guess in the end what you need is sort support in SDE.
> By looking at the code in SDE it would seem sorting in never
> supported (the query capabilities seem to always return false to
> the above cited methos), but I may be wrong. I've cc'ed the module
> maintainer, he should know more.

ArcSDE does not support NATURAL_ORDER (though it would be easy to add
support for it based on feature id). But it should definitely support
sorting by attributes.

If it is not, jericks could you please file a jira[1] issue on the
data-arcsde component and assign it to me? I think I can take care of it
quite soon.

Best regards,

Gabriel

[1] <http://jira.codehaus.org/browse/GEOS> and choose CREATE NEW ISSUE
>
> Cheers
> Andrea
>
>


--
Gabriel Roldan
OpenGeo - http://opengeo.org
Expert service straight from the developers.

------------------------------------------------------------------------------
_______________________________________________
Geotools-gt2-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geotools-gt2-users

[aaime]

Gabriel Roldan ha scritto:

> Andrea Aime wrote:
>> jericks ha scritto:
>>> I am trying to sort a shapefile or ArcSDE layer by an attribute.  I
>>> first
>>> tried adding a SortBy to a DefaultQuery, but the shapefile throws an
>>> error
>>> and the ArcSDE layer fails silently.  Next, I tried reading my
>>> FeatureSource
>>> into an ArrayList of SimpleFeatures, sorting them using a Comparator,
>>> and
>>> then writing the ArrayList back to an output FeatureStore, but the
>>> order is
>>> not preserved.  What is the correct way of sorting a FeatureSource?
>>
>> Native sorting is not supported on all stores, in particular, only
>> database backends do... not sure what goes on in SDE.
>> You can check at runtime if a feature source
>> supports sorting on a certain field by using
>> FeatureSource.getQueryCapabilities.supportsSorting(SortBy[]).
>>
>> As for preserving order in writes, there is basically only shapefile
>> that do support it, but it's just an accident of how the writing
>> occurs. SDE is free to keep whatever order it wants storage wise,
>> and when you retrieve data the order will be very much dependent
>> on what indexes are used to execute the query.
>>
>> So I guess in the end what you need is sort support in SDE.
>> By looking at the code in SDE it would seem sorting in never
>> supported (the query capabilities seem to always return false to
>> the above cited methos), but I may be wrong. I've cc'ed the module
>> maintainer, he should know more.
>
> ArcSDE does not support NATURAL_ORDER (though it would be easy to add
> support for it based on feature id). But it should definitely support
> sorting by attributes.

Gabriel,
the source code of ArcSDEFeatureSource on trunk has this:

this.queryCapabilities = new QueryCapabilities() {
     @Override
     public boolean supportsSorting(SortBy[] sortAttributes) {
         final SimpleFeatureType featureType = typeInfo.getFeatureType();
         for (int i = 0; i < sortAttributes.length; i++) {
             SortBy sortBy = sortAttributes[i];
             if (SortBy.NATURAL_ORDER == sortBy) {
                 // TODO: we should be able to support natural order
                 return false;
             }
             String attName = sortBy.getPropertyName().getPropertyName();
             if (featureType.getDescriptor(attName) == null) {
                 return false;
             }
         }
         return false;
     }
};

It would seem no matter what the capabilities will return only false.
That's why I assumed sorting was not supported.

Cheers
Andrea

--
Andrea Aime
OpenGeo - http://opengeo.org
Expert service straight from the developers.

------------------------------------------------------------------------------
_______________________________________________
Geotools-gt2-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geotools-gt2-users

[Gabriel Roldan]

know what, I didn't read the last return false at all, I sort of assumed
  it was a return true. Good catch.

And now that I remember that, since this one and the postgis one were
the same at the time of writing, I sort of remember fixing it for
postgis... ahh...

gonna file the appropriate jira so I force myself to write the
appropriate unit tests...

Gabriel

Andrea Aime wrote:

> Gabriel Roldan ha scritto:
>> Andrea Aime wrote:
>>> jericks ha scritto:
>>>> I am trying to sort a shapefile or ArcSDE layer by an attribute.  I
>>>> first
>>>> tried adding a SortBy to a DefaultQuery, but the shapefile throws an
>>>> error
>>>> and the ArcSDE layer fails silently.  Next, I tried reading my
>>>> FeatureSource
>>>> into an ArrayList of SimpleFeatures, sorting them using a
>>>> Comparator, and
>>>> then writing the ArrayList back to an output FeatureStore, but the
>>>> order is
>>>> not preserved.  What is the correct way of sorting a FeatureSource?
>>>
>>> Native sorting is not supported on all stores, in particular, only
>>> database backends do... not sure what goes on in SDE.
>>> You can check at runtime if a feature source
>>> supports sorting on a certain field by using
>>> FeatureSource.getQueryCapabilities.supportsSorting(SortBy[]).
>>>
>>> As for preserving order in writes, there is basically only shapefile
>>> that do support it, but it's just an accident of how the writing
>>> occurs. SDE is free to keep whatever order it wants storage wise,
>>> and when you retrieve data the order will be very much dependent
>>> on what indexes are used to execute the query.
>>>
>>> So I guess in the end what you need is sort support in SDE.
>>> By looking at the code in SDE it would seem sorting in never
>>> supported (the query capabilities seem to always return false to
>>> the above cited methos), but I may be wrong. I've cc'ed the module
>>> maintainer, he should know more.
>>
>> ArcSDE does not support NATURAL_ORDER (though it would be easy to add
>> support for it based on feature id). But it should definitely support
>> sorting by attributes.
>
> Gabriel,
> the source code of ArcSDEFeatureSource on trunk has this:
>
> this.queryCapabilities = new QueryCapabilities() {
>     @Override
>     public boolean supportsSorting(SortBy[] sortAttributes) {
>         final SimpleFeatureType featureType = typeInfo.getFeatureType();
>         for (int i = 0; i < sortAttributes.length; i++) {
>             SortBy sortBy = sortAttributes[i];
>             if (SortBy.NATURAL_ORDER == sortBy) {
>                 // TODO: we should be able to support natural order
>                 return false;
>             }
>             String attName = sortBy.getPropertyName().getPropertyName();
>             if (featureType.getDescriptor(attName) == null) {
>                 return false;
>             }
>         }
>         return false;
>     }
> };
>
> It would seem no matter what the capabilities will return only false.
> That's why I assumed sorting was not supported.
>
> Cheers
> Andrea
>

--
Gabriel Roldan
OpenGeo - http://opengeo.org
Expert service straight from the developers.

------------------------------------------------------------------------------
_______________________________________________
Geotools-gt2-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geotools-gt2-users

[aaime]

Gabriel Roldan ha scritto:
> know what, I didn't read the last return false at all, I sort of assumed
>  it was a return true. Good catch.
>
> And now that I remember that, since this one and the postgis one were
> the same at the time of writing, I sort of remember fixing it for
> postgis... ahh...
>
> gonna file the appropriate jira so I force myself to write the
> appropriate unit tests...

Afaik the are already working properly, but if tests are missing,
by any means, having some will be good :-)

NG datastores:
http://svn.osgeo.org/geotools/branches/2.5.x/modules/unsupported/jdbc-ng/jdbc-core/src/main/java/org/geotools/jdbc/JDBCQueryCapabilities.java

Old jdbc datastores
http://svn.osgeo.org/geotools/branches/2.5.x/modules/library/jdbc/src/main/java/org/geotools/data/jdbc/JDBCFeatureSource.java

Cheers
Andrea

--
Andrea Aime
OpenGeo - http://opengeo.org
Expert service straight from the developers.

------------------------------------------------------------------------------
_______________________________________________
Geotools-gt2-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geotools-gt2-users

[Gabriel Roldan]

Andrea Aime wrote:

> Gabriel Roldan ha scritto:
>> know what, I didn't read the last return false at all, I sort of
>> assumed  it was a return true. Good catch.
>>
>> And now that I remember that, since this one and the postgis one were
>> the same at the time of writing, I sort of remember fixing it for
>> postgis... ahh...
>>
>> gonna file the appropriate jira so I force myself to write the
>> appropriate unit tests...
>
> Afaik the are already working properly, but if tests are missing,
> by any means, having some will be good :-)

I mean tests for arcsde. It's not a jdbc one so it's on it's own.
jericks, I created http://jira.codehaus.org/browse/GEOT-2582 to track it
down, guess I'll be able to fix by the end of the week as I have some
outstanding commits right now. Feel free to add yourself as a watcher of
the issue so you get notified when it's fixed.

Regards,
Gabriel

>
> NG datastores:
> http://svn.osgeo.org/geotools/branches/2.5.x/modules/unsupported/jdbc-ng/jdbc-core/src/main/java/org/geotools/jdbc/JDBCQueryCapabilities.java 
>
>
> Old jdbc datastores
> http://svn.osgeo.org/geotools/branches/2.5.x/modules/library/jdbc/src/main/java/org/geotools/data/jdbc/JDBCFeatureSource.java 
>
>
> Cheers
> Andrea
>

--
Gabriel Roldan
OpenGeo - http://opengeo.org
Expert service straight from the developers.

------------------------------------------------------------------------------
_______________________________________________
Geotools-gt2-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geotools-gt2-users

[jericks]

Is there a way to read from one FeatureSource and write to another with sorting in tact?  Specifically, I need to read from an ArcSDE feature class, sort the data by an attribute, and then write the sorted FeatureCollection to a shapefile.  Currently, I lose any sorting when I write to a shapefile.

Thanks!
Jared

[Gabriel Roldan]

Just curious, why do you need so?
The more I think on what you're requiring the more it looks like bad
practice to me. I'm pretty sure that when you write the shapefile, the
features will be physically written in the same order you're writing
them. If writing to a database backend though, you can't really say. All
  in all, you shouldn't rely on the physical order in any case, as
that's implementation dependent and relying on it will be coincidental.
After all, that's what sorting is supposed to provide. Moreover, even if
the shapefile is gonna be written in the order you add features to it,
it doesn't imply querying it will return the same order, an index may be
in use (shapefile supports spatial indexing and afaik the index is open
if available).

So, what is it that makes you _need_ a certain physical order?

jericks wrote:
> Is there a way to read from one FeatureSource and write to another with
> sorting in tact?  Specifically, I need to read from an ArcSDE feature class,
> sort the data by an attribute, and then write the sorted FeatureCollection
> to a shapefile.  Currently, I lose any sorting when I write to a shapefile.
May it be instead that you're not "loosing" the order but that the
shapefile is using a spatial index? not sure, just an idea...

Cheers,
Gabriel
>
> Thanks!
> Jared


--
Gabriel Roldan
OpenGeo - http://opengeo.org
Expert service straight from the developers.

------------------------------------------------------------------------------
_______________________________________________
Geotools-gt2-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geotools-gt2-users

[jericks]

I am experimenting with creating statically cached tile map services using Mapnik. Order is very important to Mapnik (see http://mike.teczno.com/notes/mapnik.html) when labeling.  I found while trying to get interstates shields to draw that I had to sort my shapefile.  Mapnik doesn't work with SDE, so I used GeoTools to extract SDE layers into shapefiles.  I had to use GeoTools because OGR doesn't support NString (something I provided a patch for and you fixed months ago).  You can physically sort shapefiles with OGR using the following command:

ogr2ogr -sql "SELECT * FROM interstates ORDER BY NAME DESC" sorted_interstates.shp interstates.shp

I am trying to write a GeoTools version of this to remove my dependence on OGR (actually OSGEO4W).

I really appreciate the help!

Jared

Gabriel Roldan wrote:

Just curious, why do you need so?
The more I think on what you're requiring the more it looks like bad
practice to me. I'm pretty sure that when you write the shapefile, the
features will be physically written in the same order you're writing
them. If writing to a database backend though, you can't really say. All
  in all, you shouldn't rely on the physical order in any case, as
that's implementation dependent and relying on it will be coincidental.
After all, that's what sorting is supposed to provide. Moreover, even if
the shapefile is gonna be written in the order you add features to it,
it doesn't imply querying it will return the same order, an index may be
in use (shapefile supports spatial indexing and afaik the index is open
if available).

So, what is it that makes you _need_ a certain physical order?

jericks wrote:
> Is there a way to read from one FeatureSource and write to another with
> sorting in tact?  Specifically, I need to read from an ArcSDE feature class,
> sort the data by an attribute, and then write the sorted FeatureCollection
> to a shapefile.  Currently, I lose any sorting when I write to a shapefile.
May it be instead that you're not "loosing" the order but that the
shapefile is using a spatial index? not sure, just an idea...

Cheers,
Gabriel
>
> Thanks!
> Jared


--
Gabriel Roldan
OpenGeo - http://opengeo.org
Expert service straight from the developers.

------------------------------------------------------------------------------
_______________________________________________
Geotools-gt2-users mailing list
[e-mail]
https://lists.sourceforge.net/lists/listinfo/geotools-gt2-users