How to merge large data sets?

[Matthias Lendholt]

Dear list,
I have a shapefile with about 6000 multi polygon features (each having
more than 100 vertices) and I need them merged to one feature. (I just
need the outline/border).
What is the best algorithm to realize this with geotools?

regards
Matthias


------------------------------------------------------------------------------
Crystal Reports - New Free Runtime and 30 Day Trial
Check out the new simplified licensing option that enables unlimited
royalty-free distribution of the report engine for externally facing
server and web deployment.
http://p.sf.net/sfu/businessobjects
_______________________________________________
Geotools-gt2-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geotools-gt2-users

[Christian Müller-3]

Open the shape file as described in the user guide, call getBBox to get the
outline and create a new shape file with one record.

Samples are in the users guide.


Matthias Lendholt writes:

> Dear list,
> I have a shapefile with about 6000 multi polygon features (each having
> more than 100 vertices) and I need them merged to one feature. (I just
> need the outline/border).
> What is the best algorithm to realize this with geotools?
>
> regards
> Matthias
>
>
> ------------------------------------------------------------------------------
> Crystal Reports - New Free Runtime and 30 Day Trial
> Check out the new simplified licensing option that enables unlimited
> royalty-free distribution of the report engine for externally facing
> server and web deployment.
> http://p.sf.net/sfu/businessobjects
> _______________________________________________
> Geotools-gt2-users mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/geotools-gt2-users

 


------------------------------------------------------------------------------
Are you an open source citizen? Join us for the Open Source Bridge conference!
Portland, OR, June 17-19. Two days of sessions, one day of unconference: $250.
Need another reason to go? 24-hour hacker lounge. Register today!
http://ad.doubleclick.net/clk;215844324;13503038;v?http://opensourcebridge.org
_______________________________________________
Geotools-gt2-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geotools-gt2-users

[Mauricio Pazos]

On Friday 19 June 2009 10:27:24 am Matthias Lendholt wrote:
> Dear list,
> I have a shapefile with about 6000 multi polygon features (each having
> more than 100 vertices) and I need them merged to one feature. (I just
> need the outline/border).
> What is the best algorithm to realize this with geotools?
I think there is not a direct solution in geotools,but maybe you can use the
udig dissolve operation or the merge tool. More in:

http://udig.refractions.net/confluence/display/EN/Dissolve+Operation

http://udig.refractions.net/confluence/display/EN/Merge+Tool 
>
> regards
> Matthias
cheers
--
Mauricio Pazos
www.axios.es

------------------------------------------------------------------------------
Crystal Reports - New Free Runtime and 30 Day Trial
Check out the new simplified licensing option that enables unlimited
royalty-free distribution of the report engine for externally facing
server and web deployment.
http://p.sf.net/sfu/businessobjects
_______________________________________________
Geotools-gt2-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geotools-gt2-users

[aaime]

Matthias Lendholt ha scritto:
> Dear list,
> I have a shapefile with about 6000 multi polygon features (each having
> more than 100 vertices) and I need them merged to one feature. (I just
> need the outline/border).
> What is the best algorithm to realize this with geotools?

Extract the geometries from the features and use JTS.
Repeated merge or buffer(0) of the collection should be the way.
See
http://www.foss4g2007.org/presentations/viewattachment.php?attachment_id=35
http://lin-ear-th-inking.blogspot.com/2007/11/fast-polygon-merging-in-jts-using.html

Cheers
Andrea


--
Andrea Aime
OpenGeo - http://opengeo.org
Expert service straight from the developers.

------------------------------------------------------------------------------
Crystal Reports - New Free Runtime and 30 Day Trial
Check out the new simplified licensing option that enables unlimited
royalty-free distribution of the report engine for externally facing
server and web deployment.
http://p.sf.net/sfu/businessobjects
_______________________________________________
Geotools-gt2-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geotools-gt2-users

[mbedward]

> Open the shape file as described in the user guide, call getBBox to get the
> outline and create a new shape file with one record.
>

I think Matthias is after the outer-most boundary of his merged
features rather than the bounding box.  Andrea's suggestion sounds
good.

Michael

------------------------------------------------------------------------------
Are you an open source citizen? Join us for the Open Source Bridge conference!
Portland, OR, June 17-19. Two days of sessions, one day of unconference: $250.
Need another reason to go? 24-hour hacker lounge. Register today!
http://ad.doubleclick.net/clk;215844324;13503038;v?http://opensourcebridge.org
_______________________________________________
Geotools-gt2-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geotools-gt2-users

[Matthias Lendholt]

Andrea Aime schrieb:
> Matthias Lendholt ha scritto:
>> Dear list,
>> I have a shapefile with about 6000 multi polygon features (each having
>> more than 100 vertices) and I need them merged to one feature. (I just
>> need the outline/border).
>> What is the best algorithm to realize this with geotools?
>
> Extract the geometries from the features and use JTS.
> Repeated merge or buffer(0) of the collection should be the way.

In both cases
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space.
My next try was to split it up into chunks (each 200 geometries) but
after ~4000 the OutOfMemoryException occured. I guess those features
have to many vertices and there's no chance to get them into the memory.

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
        at
com.vividsolutions.jts.noding.SegmentNodeList.createSplitEdge(SegmentNodeList.java:243)
        at
com.vividsolutions.jts.noding.SegmentNodeList.addSplitEdges(SegmentNodeList.java:193)
        at
com.vividsolutions.jts.noding.NodedSegmentString.getNodedSubstrings(NodedSegmentString.java:66)
        at
com.vividsolutions.jts.noding.NodedSegmentString.getNodedSubstrings(NodedSegmentString.java:58)
        at
com.vividsolutions.jts.noding.MCIndexNoder.getNodedSubstrings(MCIndexNoder.java:70)
        at
com.vividsolutions.jts.operation.buffer.BufferBuilder.computeNodedEdges(BufferBuilder.java:191)
        at
com.vividsolutions.jts.operation.buffer.BufferBuilder.buffer(BufferBuilder.java:152)
        at
com.vividsolutions.jts.operation.buffer.BufferOp.bufferOriginalPrecision(BufferOp.java:309)
        at
com.vividsolutions.jts.operation.buffer.BufferOp.computeGeometry(BufferOp.java:276)
        at
com.vividsolutions.jts.operation.buffer.BufferOp.getResultGeometry(BufferOp.java:270)
        at
com.vividsolutions.jts.operation.buffer.BufferOp.bufferOp(BufferOp.java:146)
        at com.vividsolutions.jts.geom.Geometry.buffer(Geometry.java:990)

Thanks anyway!
Matthias

------------------------------------------------------------------------------
Are you an open source citizen? Join us for the Open Source Bridge conference!
Portland, OR, June 17-19. Two days of sessions, one day of unconference: $250.
Need another reason to go? 24-hour hacker lounge. Register today!
http://ad.doubleclick.net/clk;215844324;13503038;v?http://opensourcebridge.org
_______________________________________________
Geotools-gt2-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geotools-gt2-users

[Christian Müller-3]

Have you tried to increase the heap space.
Start your java app with the -Xmx parameter

java -Xmx1024m ......

This will give you a maximum heap size of 1024 Megabytes ( 1GB).

 

Matthias Lendholt writes:

> Andrea Aime schrieb:
>> Matthias Lendholt ha scritto:
>>> Dear list,
>>> I have a shapefile with about 6000 multi polygon features (each having
>>> more than 100 vertices) and I need them merged to one feature. (I just
>>> need the outline/border).
>>> What is the best algorithm to realize this with geotools?
>>
>> Extract the geometries from the features and use JTS.
>> Repeated merge or buffer(0) of the collection should be the way.
>
> In both cases
> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space.
> My next try was to split it up into chunks (each 200 geometries) but
> after ~4000 the OutOfMemoryException occured. I guess those features
> have to many vertices and there's no chance to get them into the memory.
>
> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
> at
> com.vividsolutions.jts.noding.SegmentNodeList.createSplitEdge(SegmentNodeList.java:243)
> at
> com.vividsolutions.jts.noding.SegmentNodeList.addSplitEdges(SegmentNodeList.java:193)
> at
> com.vividsolutions.jts.noding.NodedSegmentString.getNodedSubstrings(NodedSegmentString.java:66)
> at
> com.vividsolutions.jts.noding.NodedSegmentString.getNodedSubstrings(NodedSegmentString.java:58)
> at
> com.vividsolutions.jts.noding.MCIndexNoder.getNodedSubstrings(MCIndexNoder.java:70)
> at
> com.vividsolutions.jts.operation.buffer.BufferBuilder.computeNodedEdges(BufferBuilder.java:191)
> at
> com.vividsolutions.jts.operation.buffer.BufferBuilder.buffer(BufferBuilder.java:152)
> at
> com.vividsolutions.jts.operation.buffer.BufferOp.bufferOriginalPrecision(BufferOp.java:309)
> at
> com.vividsolutions.jts.operation.buffer.BufferOp.computeGeometry(BufferOp.java:276)
> at
> com.vividsolutions.jts.operation.buffer.BufferOp.getResultGeometry(BufferOp.java:270)
> at
> com.vividsolutions.jts.operation.buffer.BufferOp.bufferOp(BufferOp.java:146)
> at com.vividsolutions.jts.geom.Geometry.buffer(Geometry.java:990)
>
> Thanks anyway!
> Matthias
>
> ------------------------------------------------------------------------------
> Are you an open source citizen? Join us for the Open Source Bridge conference!
> Portland, OR, June 17-19. Two days of sessions, one day of unconference: $250.
> Need another reason to go? 24-hour hacker lounge. Register today!
> http://ad.doubleclick.net/clk;215844324;13503038;v?http://opensourcebridge.org
> _______________________________________________
> Geotools-gt2-users mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/geotools-gt2-users

 


------------------------------------------------------------------------------
Are you an open source citizen? Join us for the Open Source Bridge conference!
Portland, OR, June 17-19. Two days of sessions, one day of unconference: $250.
Need another reason to go? 24-hour hacker lounge. Register today!
http://ad.doubleclick.net/clk;215844324;13503038;v?http://opensourcebridge.org
_______________________________________________
Geotools-gt2-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geotools-gt2-users

[Matthias Lendholt]

Christian Müller schrieb:
> Have you tried to increase the heap space.
> Start your java app with the -Xmx parameter
> java -Xmx1024m ......
> This will give you a maximum heap size of 1024 Megabytes ( 1GB).

This is my default setting. It doesn't help.

Matthias

------------------------------------------------------------------------------
Are you an open source citizen? Join us for the Open Source Bridge conference!
Portland, OR, June 17-19. Two days of sessions, one day of unconference: $250.
Need another reason to go? 24-hour hacker lounge. Register today!
http://ad.doubleclick.net/clk;215844324;13503038;v?http://opensourcebridge.org
_______________________________________________
Geotools-gt2-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geotools-gt2-users

[Christian Müller-3]

Another idea.

Use ogr2ogr from www.gdal.org to split your shape file.

look here for an example
http://lists.xastir.org/pipermail/xastir/2005-January/007610.html 



------------------------------------------------------------------------------
Are you an open source citizen? Join us for the Open Source Bridge conference!
Portland, OR, June 17-19. Two days of sessions, one day of unconference: $250.
Need another reason to go? 24-hour hacker lounge. Register today!
http://ad.doubleclick.net/clk;215844324;13503038;v?http://opensourcebridge.org
_______________________________________________
Geotools-gt2-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geotools-gt2-users