gdalwarp -co COMPRESSION issues

[Maciej Sieczka - old]

Hi

The bug http://trac.osgeo.org/gdal/ticket/1689 is still present in 1.5.0
+ SVN r13865. In that report I mention that gdalwarp creates compressed
TIFFs 45% bigger than gdal_translate. Recently I've noticed it can even
produce TIFFs 13 times (!) bigger (and more?).

Given an input GeoTIFF of size 21802684 bytes, the command:

$ gdal_translate -co "COMPRESS=DEFLATE" slope.tif slope_gt.tif

produces it's clone, of the same size - OK. However, the command:

$ gdalwarp -co "COMPRESS=DEFLATE" slope.tif slope_gw.tif

creates a file 13x bigger - 299550004 bytes!


Moreover, if the input file is big enough for gdalwarp with compression
to try creating 4GB+ raster, it corrupts the output, yielding errors:

0...O10...20...30...40...50...60...70...80...90ERROR 1:
TIFFAppendToStrip:Maximum TIFF file size exceeded
ERROR 1: TIFFAppendToStrip:Maximum TIFF file size exceeded
ERROR 1: TIFFWriteEncodedStrip() failed.

which I guess should not happen, since BigTIFF is enabled - at configure
I have: "LIBTIFF support: internal (BigTIFF=yes)".


For the above reasons, shouldn't output compression for gdalwarp be
disabled to avoid data corruption and wasting space? Or would fixing
another bug, http://trac.osgeo.org/gdal/ticket/1688, fix the compression
in gdalwarp too?

Maciek
_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev

_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev

[Rahkonen Jukka]

Maciej Sieczka <tutey <at> o2.pl> writes:

> Moreover, if the input file is big enough for gdalwarp with compression
> to try creating 4GB+ raster, it corrupts the output, yielding errors:

> TIFFAppendToStrip:Maximum TIFF file size exceeded
> ERROR 1: TIFFAppendToStrip:Maximum TIFF file size exceeded
> ERROR 1: TIFFWriteEncodedStrip() failed.
>
> which I guess should not happen, since BigTIFF is enabled - at configure
> I have: "LIBTIFF support: internal (BigTIFF=yes)".


About BigTIFF, I guess that the error will perhaps not happen if you ask
gdalwarp to use it by adding -co BigTIFF=yes.

-Jukka Rahkonen-

_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev

[Maciej Sieczka - old]

Jukka Rahkonen pisze:
> Maciej Sieczka <tutey <at> o2.pl> writes:

> About BigTIFF, I guess that the error will perhaps not happen if you ask
> gdalwarp to use it by adding -co BigTIFF=yes.

Right. I forgot it.

Anyway, it still holds true that gdalwarp produces compressed rasters
several times bigger than gdal_translate.

For example - fetch this RGB 290 MB ortophoto tile. It's an RGB (3
8-byte bands) GeoTIFF - nothing fancy [1].

It was created with gdal_translate -co "COMPRESS=LZW". It's size is
303714237 bytes.

Process it with gdalwarp -co "COMPRESS=LZW" - it becomes 1443573134
bytes big, ie. nearly 5 times bigger. Even about 3 times bigger than the
same raster without any compression!

Worse ratios happen, depending on your data. When can we expect this
issue to be fixed?

[1]http://www.biol.uni.wroc.pl/sieczka/udostepnione/M-33-21-A-b-4.tif

Using GDAL 1.5.1 + SVN r14009.

Maciek
_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev

[Frank Warmerdam]

Maciej Sieczka wrote:

> Jukka Rahkonen pisze:
>> Maciej Sieczka <tutey <at> o2.pl> writes:
>
>> About BigTIFF, I guess that the error will perhaps not happen if you ask
>> gdalwarp to use it by adding -co BigTIFF=yes.
>
> Right. I forgot it.
>
> Anyway, it still holds true that gdalwarp produces compressed rasters
> several times bigger than gdal_translate.
>
> For example - fetch this RGB 290 MB ortophoto tile. It's an RGB (3
> 8-byte bands) GeoTIFF - nothing fancy [1].
>
> It was created with gdal_translate -co "COMPRESS=LZW". It's size is
> 303714237 bytes.
>
> Process it with gdalwarp -co "COMPRESS=LZW" - it becomes 1443573134
> bytes big, ie. nearly 5 times bigger. Even about 3 times bigger than the
> same raster without any compression!
>
> Worse ratios happen, depending on your data. When can we expect this
> issue to be fixed?

Maciej,

It is not on my critical path, so I don't anticipate working on it in
the near future.  It may well be challenging for others to address.

Basically, there is no particular timeline for it to be fixed.

You could try and fix it yourself, or contract with someone to fix it
if it is very important to you.

BTW, I assume there is a trac ticket for this problem.  I'd suggest noting
it in emails about the topic.  If it were quoted, I might have contemplated
boosting it's priority.

Best regards,
--
---------------------------------------+--------------------------------------
I set the clouds in motion - turn up   | Frank Warmerdam, [hidden email]
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush    | President OSGeo, http://osgeo.org

_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev

[Maciej Sieczka - old]

Frank Warmerdam pisze:

> It is not on my critical path, so I don't anticipate working on it in
> the near future.  It may well be challenging for others to address.
>
> Basically, there is no particular timeline for it to be fixed.
>
> You could try and fix it yourself, or contract with someone to fix it
> if it is very important to you.

It is very important for all gdalwarp users. gdalwarp creates compressed
GeoTIFFs too big; even several times bigger than their uncompressed
versions.

> BTW, I assume there is a trac ticket for this problem.  I'd suggest noting
> it in emails about the topic.  If it were quoted, I might have contemplated
> boosting it's priority.

I've already noticed it in my intial post in this thread. Here they go
again:

http://trac.osgeo.org/gdal/ticket/1688
http://trac.osgeo.org/gdal/ticket/1689

They propably should be merged.

Maciek

_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev

[Adam Nowacki]

Add -co "TILED=YES" to produce tiled tiff (and a lot smaller that way).

Maciej Sieczka wrote:

> Jukka Rahkonen pisze:
>> Maciej Sieczka <tutey <at> o2.pl> writes:
>
>> About BigTIFF, I guess that the error will perhaps not happen if you ask
>> gdalwarp to use it by adding -co BigTIFF=yes.
>
> Right. I forgot it.
>
> Anyway, it still holds true that gdalwarp produces compressed rasters
> several times bigger than gdal_translate.
>
> For example - fetch this RGB 290 MB ortophoto tile. It's an RGB (3
> 8-byte bands) GeoTIFF - nothing fancy [1].
>
> It was created with gdal_translate -co "COMPRESS=LZW". It's size is
> 303714237 bytes.
>
> Process it with gdalwarp -co "COMPRESS=LZW" - it becomes 1443573134
> bytes big, ie. nearly 5 times bigger. Even about 3 times bigger than the
> same raster without any compression!
>
> Worse ratios happen, depending on your data. When can we expect this
> issue to be fixed?
>
> [1]http://www.biol.uni.wroc.pl/sieczka/udostepnione/M-33-21-A-b-4.tif
>
> Using GDAL 1.5.1 + SVN r14009.
>
> Maciek
> _______________________________________________
> gdal-dev mailing list
> [hidden email]
> http://lists.osgeo.org/mailman/listinfo/gdal-dev
>
>

_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev

[Maciej Sieczka - old]

Adam Nowacki pisze:
> Add -co "TILED=YES" to produce tiled tiff (and a lot smaller that way).

1. There are software that perform better with stripped TIFFs, eg.
ArcPad [1], QGIS (latest SVN trunk) too - the tiff at [2] if tiled with
gdal_translate takes about 10 (ten) times longer to render that it's
stripped version in QGIS.

2. There are software which don't support tiled TIFF at all. To stay on
the safe side I prefer to create stripped TIFFs in case I might be
sharing/selling them. I might be not the only one.

2. User will be surprised that gdalwarp -co "TILED=NO" -co
"COMPRESS=LZW" (or other compression alg.) creates tiffs bigger than
"COMPRESS=NONE".

3. Mind that gdal_translate -co "TILED=YES" still outperforms gdalwarp
-co "TILED=YES", any compression. With the sample [2] it creates a TIFF
over 8% smaller. Not *very* much, but with 13 such tiffs gdal_translate
gives you a room on your hard disk for another one, compared to gdalwarp.

[1]http://arcpadteam.blogspot.com/2006/08/raster-tips-1-choosing-raster-format.html
[2]http://www.biol.uni.wroc.pl/sieczka/udostepnione/M-33-21-A-b-4.tif

Maciek
_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev

[Maciej Sieczka - old]

Frank Warmerdam pisze:
> Maciej Sieczka wrote:

>> It is very important for all gdalwarp users.

> I disagree with the above statement.  I think it is a small
> segment of people who create compressed files with gdalwarp, and
> there is a fairly straightforward workaround (warp then compress).

Since gdalwarp supports compressing tiffs, who would guess he has to do
something extra? gdalwarp makes an impression of being able to warp and
compress in one step. Not many users will expect that gdalwarp writes a
compressed TIFF bigger than not compressed, and that he has to fix this
with gdal_translate afterwards.

Moreover, I wonder if the guts of the issue that gdalwarp handles
compression wrong, might also affect GDAL-dependent software. Eg.
r.out.gdal in GRASS - I took the sample [1], imported into GRASS and
re-exported with:

r.in.gdal in=M-33-21-A-b-4.tif out=M-33-21-A-b-4 -o
g.region rast=M-33-21-A-b-4.red -a
r.out.gdal in=M-33-21-A-b-4 out=M-33-21-A-b-4_gr_lzw.tif
-createopt="COMPRESS=LZW" type=Byte

which created an LZW-compressed TIFF 2.33 times bigger than the
original, also LZW-compressed. Could this r.out.gdal issue be related to
gdalwarp issue?

Maciek

_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev

[Maciej Sieczka - old]

Maciej Sieczka pisze:
> I took the sample [1]

And here's the link to the sample:

[1]http://www.biol.uni.wroc.pl/sieczka/udostepnione/M-33-21-A-b-4.tif

Maciek
_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev

[Even Rouault]

Maciek,

we are well aware of the problem but I'm afraid that complaining more will not
help to fix it quicker.

I've analyzed the problem for my own needs a few months ago and I came with
the conclusion that it looks like a complex interaction between GDAL and
libtiff. The strip/tile allocation strategy in libtiff is surely at stake,
but it could also involve the GeoTIFF driver in GDAL, or the way that some
algorithms such as gdalwarp work.

Changing libtiff strip/tile allocation strategy to enable it to recycle wasted
space or to reserve space to anticipated partial writes is more a (complex)
feature development than a bug fix...

Let imagine the following scenario :
- Create a LZW compress GTiff with GDALCreate
- Write sequentially each strip with zeros. You'll get a small image
- Then write sequentially each strip with first half of the line being zeros
and second half random data. Libtiff will try to reuse the previously written
strips, but, of course, strips with random data compressed will not fit into
the very small strips produced. So libtiff must allocate new space at the end
of the file to put the new data.
- Let do it again but with only random data. Again, the last allocated strip
space will not be enough.

And I'm not mentionning that I/O in GDAL are cached, that gdalwrap doesn't
write sequentially strips...

So, we could now imagine to recycle wasted space, but that wouldn't help a lot
as you're going to have fragmentation and strips not organized in the file as
they are in the image. Thus frequent read seeks forth and back, leading to
poor read performance. So you need some sort of defragmentation utility. And
guess what ? I know one that already does more or less that job : it's called
gdal_translate....

Of course I can be wrong and over-exaggerating the difficulties...

Best regards,
Even

Le Friday 14 March 2008 22:17:06 Maciej Sieczka, vous avez écrit :

> Adam Nowacki pisze:
> > Add -co "TILED=YES" to produce tiled tiff (and a lot smaller that way).
>
> 1. There are software that perform better with stripped TIFFs, eg.
> ArcPad [1], QGIS (latest SVN trunk) too - the tiff at [2] if tiled with
> gdal_translate takes about 10 (ten) times longer to render that it's
> stripped version in QGIS.
>
> 2. There are software which don't support tiled TIFF at all. To stay on
> the safe side I prefer to create stripped TIFFs in case I might be
> sharing/selling them. I might be not the only one.
>
> 2. User will be surprised that gdalwarp -co "TILED=NO" -co
> "COMPRESS=LZW" (or other compression alg.) creates tiffs bigger than
> "COMPRESS=NONE".
>
> 3. Mind that gdal_translate -co "TILED=YES" still outperforms gdalwarp
> -co "TILED=YES", any compression. With the sample [2] it creates a TIFF
> over 8% smaller. Not *very* much, but with 13 such tiffs gdal_translate
> gives you a room on your hard disk for another one, compared to gdalwarp.
>
> [1]http://arcpadteam.blogspot.com/2006/08/raster-tips-1-choosing-raster-for
>mat.html
> [2]http://www.biol.uni.wroc.pl/sieczka/udostepnione/M-33-21-A-b-4.tif
>
> Maciek
> _______________________________________________
> gdal-dev mailing list
> [hidden email]
> http://lists.osgeo.org/mailman/listinfo/gdal-dev

_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev

[Craig Miller-6]

I'm interested now, as it sounds like the GDAL API calls that gdalwarp uses
are dictating how good the compression is which in turn is affecting how
good the compression is of an in-house utility we are using.  I don't have
time to look at it for a couple of weeks, but if nobody has looked into it
by then I will.

In the meantime, you have a good workaround.  Use gdalwarp followed by a
gdal_translate to get the compression.  A bit of a PITA, but it doesn't
sound like it is a showstopper.

Craig
 
---
http://www.overlandnavigator.com
Touchscreen Friendly GPS Mapping

-----Original Message-----
From: [hidden email]
[mailto:[hidden email]] On Behalf Of Maciej Sieczka
Sent: Friday, March 14, 2008 2:50 PM
To: Frank Warmerdam
Cc: [hidden email]; GRASS developers list
Subject: [gdal-dev] Re: gdalwarp -co COMPRESS issues

Frank Warmerdam pisze:
> Maciej Sieczka wrote:

>> It is very important for all gdalwarp users.

> I disagree with the above statement.  I think it is a small
> segment of people who create compressed files with gdalwarp, and
> there is a fairly straightforward workaround (warp then compress).

Since gdalwarp supports compressing tiffs, who would guess he has to do
something extra? gdalwarp makes an impression of being able to warp and
compress in one step. Not many users will expect that gdalwarp writes a
compressed TIFF bigger than not compressed, and that he has to fix this
with gdal_translate afterwards.

Moreover, I wonder if the guts of the issue that gdalwarp handles
compression wrong, might also affect GDAL-dependent software. Eg.
r.out.gdal in GRASS - I took the sample [1], imported into GRASS and
re-exported with:

r.in.gdal in=M-33-21-A-b-4.tif out=M-33-21-A-b-4 -o
g.region rast=M-33-21-A-b-4.red -a
r.out.gdal in=M-33-21-A-b-4 out=M-33-21-A-b-4_gr_lzw.tif
-createopt="COMPRESS=LZW" type=Byte

which created an LZW-compressed TIFF 2.33 times bigger than the
original, also LZW-compressed. Could this r.out.gdal issue be related to
gdalwarp issue?

Maciek

_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev

No virus found in this incoming message.
Checked by AVG.
Version: 7.5.519 / Virus Database: 269.21.7/1329 - Release Date: 3/14/2008
12:33 PM
 

No virus found in this outgoing message.
Checked by AVG.
Version: 7.5.519 / Virus Database: 269.21.7/1329 - Release Date: 3/14/2008
12:33 PM
 

_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev

[Frank Warmerdam]

Maciej Sieczka wrote:
> Since gdalwarp supports compressing tiffs, who would guess he has to do
> something extra? gdalwarp makes an impression of being able to warp and
> compress in one step. Not many users will expect that gdalwarp writes a
> compressed TIFF bigger than not compressed, and that he has to fix this
> with gdal_translate afterwards.

Maciej,

While I agree with the above, it does not support your claim that
this affects all gdalwarp users and is somehow an important problem.

> Moreover, I wonder if the guts of the issue that gdalwarp handles
> compression wrong, might also affect GDAL-dependent software. Eg.
> r.out.gdal in GRASS - I took the sample [1], imported into GRASS and
> re-exported with:
>
> r.in.gdal in=M-33-21-A-b-4.tif out=M-33-21-A-b-4 -o
> g.region rast=M-33-21-A-b-4.red -a
> r.out.gdal in=M-33-21-A-b-4 out=M-33-21-A-b-4_gr_lzw.tif
> -createopt="COMPRESS=LZW" type=Byte
>
> which created an LZW-compressed TIFF 2.33 times bigger than the
> original, also LZW-compressed. Could this r.out.gdal issue be related to
> gdalwarp issue?

It might, or might not be. It is hard to know without analysis.

All in all, I believe I will discover that if I spend several hours
digging into this problem it will turn out to be the normal result
of losing track of space in the TIFF file each time a tile is
recompressed and gets bigger and thus has to be moved to the end of
the file.  There isn't much I can do about this without spending a
huge effort to have libtiff keep track of all unused space in the file
and manage it on the fly.

So I'm basically not planning to spend any more time on it, and will
likely ignore questions about the topic.

Regards,
--
---------------------------------------+--------------------------------------
I set the clouds in motion - turn up   | Frank Warmerdam, [hidden email]
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush    | President OSGeo, http://osgeo.org

_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev

[Craig Miller-6]

Could you post:
- gdalinfo of the source image
- gdalwarp command line + gdalinfo of resultant image
- gdal_translate command line + gdalinfo of resultant image

No need to include the entire color palette, just the lines that have colors
defined.

Thanks,
Craig

Note:  Removed Frank and GRASS mailing list from reply.

---
Spatial Minds, LLC
GeoSpatial Software Engineering
& Touchscreen Friendly GPS Mapping

-----Original Message-----
From: [hidden email]
[mailto:[hidden email]] On Behalf Of Maciej Sieczka
Sent: Friday, March 14, 2008 2:52 PM
To: Frank Warmerdam
Cc: [hidden email]; GRASS developers list
Subject: Re: [gdal-dev] Re: gdalwarp -co COMPRESS issues

Maciej Sieczka pisze:
> I took the sample [1]

And here's the link to the sample:

[1]http://www.biol.uni.wroc.pl/sieczka/udostepnione/M-33-21-A-b-4.tif

Maciek
_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev

No virus found in this incoming message.
Checked by AVG.
Version: 7.5.519 / Virus Database: 269.21.7/1329 - Release Date: 3/14/2008
12:33 PM
 

No virus found in this outgoing message.
Checked by AVG.
Version: 7.5.519 / Virus Database: 269.21.7/1329 - Release Date: 3/14/2008
12:33 PM
 

_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev

[Maciej Sieczka - old]

Craig Miller pisze:
> Could you post:
> - gdalinfo of the source image

$ gdalinfo M-33-21-A-b-4.tif
Driver: GTiff/GeoTIFF
Files: tmp/M-33-21-A-b-4.tif
Size is 12123, 12914
Coordinate System is:
PROJCS["WGS 84 / UTM zone 33N",
     GEOGCS["WGS 84",
         DATUM["WGS_1984",
             SPHEROID["WGS 84",6378137,298.2572235629972,
                 AUTHORITY["EPSG","7030"]],
             AUTHORITY["EPSG","6326"]],
         PRIMEM["Greenwich",0],
         UNIT["degree",0.0174532925199433],
         AUTHORITY["EPSG","4326"]],
     PROJECTION["Transverse_Mercator"],
     PARAMETER["latitude_of_origin",0],
     PARAMETER["central_meridian",15],
     PARAMETER["scale_factor",0.9996],
     PARAMETER["false_easting",500000],
     PARAMETER["false_northing",0],
     UNIT["metre",1,
         AUTHORITY["EPSG","9001"]],
     AUTHORITY["EPSG","32633"]]
Origin = (582099.959999999962747,5720171.540000000037253)
Pixel Size = (0.379999999999999,-0.380000000000023)
Metadata:
   AREA_OR_POINT=Area
Image Structure Metadata:
   COMPRESSION=LZW
   INTERLEAVE=PIXEL
Corner Coordinates:
Upper Left  (  582099.960, 5720171.540) ( 16d11'9.91"E, 51d37'35.64"N)
Lower Left  (  582099.960, 5715264.220) ( 16d11'5.78"E, 51d34'56.83"N)
Upper Right (  586706.700, 5720171.540) ( 16d15'9.44"E, 51d37'33.15"N)
Lower Right (  586706.700, 5715264.220) ( 16d15'5.08"E, 51d34'54.34"N)
Center      (  584403.330, 5717717.880) ( 16d13'7.55"E, 51d36'15.01"N)
Band 1 Block=12123x1 Type=Byte, ColorInterp=Red
   Metadata:
     COLOR_TABLE_RULES_COUNT=1
     COLOR_TABLE_RULE_RGB_0=0.000000e+00 2.550000e+02 0 0 0 255 255 255
Band 2 Block=12123x1 Type=Byte, ColorInterp=Green
   Metadata:
     COLOR_TABLE_RULES_COUNT=1
     COLOR_TABLE_RULE_RGB_0=0.000000e+00 2.550000e+02 0 0 0 255 255 255
Band 3 Block=12123x1 Type=Byte, ColorInterp=Blue
   Metadata:
     COLOR_TABLE_RULES_COUNT=1
     COLOR_TABLE_RULE_RGB_0=0.000000e+00 2.550000e+02 0 0 0 255 255 255

> - gdalwarp command line + gdalinfo of resultant image

$ gdalwarp -co "COMPRESS=LZW" M-33-21-A-b-4.tif M-33-21-A-b-4_gw.tif

$ gdalinfo M-33-21-A-b-4_gw.tif
Driver: GTiff/GeoTIFF
Files: M-33-21-A-b-4_gw.tif
Size is 12123, 12914
Coordinate System is:
PROJCS["WGS 84 / UTM zone 33N",
     GEOGCS["WGS 84",
         DATUM["WGS_1984",
             SPHEROID["WGS 84",6378137,298.2572235629972,
                 AUTHORITY["EPSG","7030"]],
             AUTHORITY["EPSG","6326"]],
         PRIMEM["Greenwich",0],
         UNIT["degree",0.0174532925199433],
         AUTHORITY["EPSG","4326"]],
     PROJECTION["Transverse_Mercator"],
     PARAMETER["latitude_of_origin",0],
     PARAMETER["central_meridian",15],
     PARAMETER["scale_factor",0.9996],
     PARAMETER["false_easting",500000],
     PARAMETER["false_northing",0],
     UNIT["metre",1,
         AUTHORITY["EPSG","9001"]],
     AUTHORITY["EPSG","32633"]]
Origin = (582099.959999999962747,5720171.540000000037253)
Pixel Size = (0.380000000000012,-0.380000000000012)
Metadata:
   AREA_OR_POINT=Area
Image Structure Metadata:
   COMPRESSION=LZW
   INTERLEAVE=PIXEL
Corner Coordinates:
Upper Left  (  582099.960, 5720171.540) ( 16d11'9.91"E, 51d37'35.64"N)
Lower Left  (  582099.960, 5715264.220) ( 16d11'5.78"E, 51d34'56.83"N)
Upper Right (  586706.700, 5720171.540) ( 16d15'9.44"E, 51d37'33.15"N)
Lower Right (  586706.700, 5715264.220) ( 16d15'5.08"E, 51d34'54.34"N)
Center      (  584403.330, 5717717.880) ( 16d13'7.55"E, 51d36'15.01"N)
Band 1 Block=12123x1 Type=Byte, ColorInterp=Red
Band 2 Block=12123x1 Type=Byte, ColorInterp=Green
Band 3 Block=12123x1 Type=Byte, ColorInterp=Blue

> - gdal_translate command line + gdalinfo of resultant image

The original M-33-21-A-b-4.tif is an output of gdal_translate -co
"COMPRESS=LZW".

Maciek
_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev

[Rahkonen Jukka]

Maciej Sieczka <tutey <at> o2.pl> writes:

> Since gdalwarp supports compressing tiffs, who would guess he has to do
> something extra? gdalwarp makes an impression of being able to warp and
> compress in one step. Not many users will expect that gdalwarp writes a
> compressed TIFF bigger than not compressed, and that he has to fix this
> with gdal_translate afterwards.

For my mind this gdalwarp - LZW compression bug could be well enough corrected
by adding a line to gdalwarp documantation saying that it is not recommended
without preliminary test with actual image data.

I don't know if an effective LZW compression is possible to perform in warping
situation, when the compressor should be able to compress a stream of warped
data without knowing before hand what will follow.  Perhaps it is, for example
line by line, but with some other compressing methods it is for sure not.  I
believe that having two runs, first warp to uncompressed file and then
compressing it will not take very much longer time.  LZW compression is rather a
a fast process compared to warping. I do understand that this is mostly a
question of principle: if it is possible to do then it should do it right and
you would be happy if it is deactivated.  For example one cannot warp directly
with gdalwarp into ecw format but people are tolerating this limitation. Even it
is possible to do with ER Mapper products.

By the way, have you ever tried to create a tiled tiff as output?  It might give
the compressor some nice intermediate points to decide when to compress a bunch
of image data and then, perhaps, the final file size would not increase so much.

-Jukka Rahkonen-


_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev

[Matt Wilkie-2]

 > ...Here they go again:
 >
 > http://trac.osgeo.org/gdal/ticket/1688
 > http://trac.osgeo.org/gdal/ticket/1689

I first noticed compression problems with gdalwarp about gdal 1.4.0, and
filed ticket http://trac.osgeo.org/gdal/ticket/1470, which was fixed.
Perhaps there is something in that change which could help here.

If not and fixing the problem is as difficult as Even and Frank surmise
(more likely), then I suggest that to avoid confusing people the
compress option be dropped from gdalwarp for now, or at least issue a
warning when used.

It might also be fruitful if Even's extended description for why this
bug won't be addressed soon is posted to one of the ticket numbers above.

best regards,

matt wilkie
--------------------------------------------
Geographic Information,
Information Management and Technology,
Yukon Department of Environment
10 Burns Road * Whitehorse, Yukon * Y1A 4Y9
867-667-8133 Tel * 867-393-7003 Fax
http://environmentyukon.gov.yk.ca/geomatics/
--------------------------------------------

_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev

[Maciej Sieczka - old]

Jukka Rahkonen pisze:
> Maciej Sieczka <tutey <at> o2.pl> writes:

>> Since gdalwarp supports compressing tiffs, who would guess he has
>> to do something extra? gdalwarp makes an impression of being able
>> to warp and compress in one step. Not many users will expect that
>> gdalwarp writes a compressed TIFF bigger than not compressed, and
>> that he has to fix this with gdal_translate afterwards.

> For my mind this gdalwarp - LZW compression bug could be well enough
> corrected by adding a line to gdalwarp documantation saying that it
> is not recommended without preliminary test with actual image data.

IMO better drop the buggy feature alltogether if it is not going to be
fixed.

> By the way, have you ever tried to create a tiled tiff as output?  It
>  might give the compressor some nice intermediate points to decide
> when to compress a bunch of image data and then, perhaps, the final
> file size would not increase so much.

Tiled tiff is not always an option. As I wrote earlier in the thread:

1. There are software that perform better with stripped TIFFs, eg.
ArcPad [1], QGIS - the tiff at [2] if tiled with gdal_translate takes
about 10 (ten) times longer to render in QGIS than it's stripped version.

2. There are software which don't support tiled TIFF at all. To stay
on the safe side I prefer to create stripped TIFFs in case I might be
sharing/selling them. I might be not the only one.

3. User will be surprised that gdalwarp -co "TILED=NO" -co
"COMPRESS=LZW" (or other compression alg.) creates tiffs bigger than
"COMPRESS=NONE".

4. Mind that gdal_translate -co "TILED=YES" still outperforms
gdalwarp -co "TILED=YES", any compression, file size - wise. With the
sample [2] it creates a TIFF over 8% smaller. Not *very* much, but with
13 such tiffs gdal_translate gives you a room on your hard disk for
another one, compared to gdalwarp.

[1]http://arcpadteam.blogspot.com/2006/08/raster-tips-1-choosing-raster-format.html
[2]http://www.biol.uni.wroc.pl/sieczka/udostepnione/M-33-21-A-b-4.tif
_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev

[Frank Warmerdam]

Matt Wilkie wrote:
> If not and fixing the problem is as difficult as Even and Frank surmise
> (more likely), then I suggest that to avoid confusing people the
> compress option be dropped from gdalwarp for now, or at least issue a
> warning when used.

Matt,

1) It does not appear to be a bug.  It is the natural behavior of the
driver when rewriting in place.

2) It is not possible to drop driver specific creation options from gdalwarp.
Gdalwarp passes driver options to the drivers and has no gdalwarp specific
knowledge about them.

If there is anything to be done it would be to explain the issues of
lost file space with "multiple update" applications like gdalwarp in
the GTiff driver.

Grr.  I said I wasn't going to respond anymore!

Best regards,
--
---------------------------------------+--------------------------------------
I set the clouds in motion - turn up   | Frank Warmerdam, [hidden email]
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush    | President OSGeo, http://osgeo.org

_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev

[Matt Wilkie-2]

> 1) It does not appear to be a bug.  It is the natural behavior of the
> driver when rewriting in place.

It may be the program is doing exactly what it has been instructed to
do, but it is contrary what users reasonably expect: that asking for
compressed output will result in a smaller file. However since it does
not appear to be possible to change the behaviour at present I've added
a note to http://trac.osgeo.org/gdal/wiki/UserDocs/GdalWarp about it.

> Grr.  I said I wasn't going to respond anymore!

I guess there is just no way to keep a good man down :)


matt wilkie
--------------------------------------------
Geographic Information,
Information Management and Technology,
Yukon Department of Environment
10 Burns Road * Whitehorse, Yukon * Y1A 4Y9
867-667-8133 Tel * 867-393-7003 Fax
http://environmentyukon.gov.yk.ca/geomatics/
--------------------------------------------

_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev