Patch for WMTS in TileCache

[Jim Groffen]

Some javascript/style in this post has been disabled (why?)

Hello,

 

Attached is a patch file that, if I made it correctly, will update TileCache revision 391 with the WMTS capabilities. I’ve added a guide (WMTSGuide.txt) and sample configuration file (wmts.cfg) to the docs folder that describes the new features.

 

The implementation got a fair bit of testing in the OWS-6 testbed, but since then I’ve implemented support for MetaTiling, which only I’ve tested. A couple of things I came up with while merging with the latest revision and writing the documentation were:

 

-          I avoided changes to existing TileCache files as much as possible. The goal was to keep the WMTS functionality contained to its Service, Layer and Disk module files as much as possible. When it comes to the changes to support reading TileMatrixSet configurations though I had a hard time separating this out from Service.py. I wasn’t keen to subclass Service.py, but I don’t like how much complexity has been added to the configuration reading. If anyone can suggest something that’d be great.

-          MetaTile was subclassed with WMTSTile so it could keep track of tile_matrix (used instead of resolution to go from tile co-ords to bounds). After implementing meta tiling for WMTS I think It may have been better to add tile_matrix to the Tile class instead, and WMTS specific code could look for it in the class if they wanted to use it.

-          top_left configuration setting – I don’t like this name because it causes a bit of confusion. The setting wants a coordinate which is usually ordered left,top and not top,left. The top_left matches the wording as it comes out in the WMTS capabilities document, but perhaps this should be something like ‘origin’? The problem with origin is it doesn’t state what corner of the bounds is used as the origin. I don’t like left_top either – it may be more accurate than top_left most of the time but it still implies a specific order and now doesn’t match WMTS (which is TopLeftCorner), so after all that I’m not sure what it should be called. Maybe top_left_corner?

-          There is a pending change request that was popular to make style always be specified in the path. When this change occurs, If no style is set in configuration then a default style name should be set and used in the paths which won’t affect the WMS getMap requests. This would be easy enough to do but isn’t *quite* yet standard WMTS behavior . Should it get done now?

 

If there are any questions, problems, feedback or anything else you need me to do let me know and I’ll do what I can to sort them out. I’m a fledgling pythoneer but love the language and philosophy so I’m keen for feedback.

 

Thanks,

 

Jim Groffen

Senior Software Engineer, LISAsoft

PH: [phone]

MOB: [phone]

 

---

The contents of this email are confidential and may be subject to legal or professional privilege and copyright. No representation is made that this email is free of viruses or other defects. If you have received this communication in error, you may not copy or distribute any part of it or otherwise disclose its contents to anyone. Please advise the sender of your incorrect receipt of this correspondence.


_______________________________________________
Tilecache mailing list
[hidden email]
http://openlayers.org/mailman/listinfo/tilecache

[Christopher Schmidt-2]

On Thu, Jun 25, 2009 at 07:38:26PM +0930, Jim Groffen wrote:
> Hello,

Sorry this has taken so long.

> Attached is a patch file that, if I made it correctly, will update TileCache revision 391 with the WMTS capabilities. I've added a guide (WMTSGuide.txt) and sample configuration file (wmts.cfg) to the docs folder that describes the new features.

Having briefly reviewed this patch, I have the following questions:

 * Currently, TileCache is designed to support accessing any tile set
   with any cache backend. Since all tiling schemes essentially work the
   same -- specifically, they all work on a fixed grid, with a set of
   scales and reslutions that allows a multilevel fixed grid, typically
   with the same corners at all levels of the grid -- it is easiest to
   transform this grid into a fixed set.

   It seems like the WMTS layer does not do this, and invents its own
   cache instead.

   Is the cache set up by WMTS different enough that this is *really*
   neccesary?

 * It appears that WMTS has been developed as a 'Layer'. Layers are ways
   of accessing backend services -- so the MapServer layer accesses a
   MapServer service, the WMS layer accesses a WMS service, etc. WMTS
   does not appear to be this (since you're pointing at my WMS for your
   demo config). Instead, it appears that WMTS should be a *service*
   (TileCache/Services/ directory); the configuration parameters that
   you are using to define tilesets should potentially be added to *any*
   layer, and these configuration parameters are used to return metadata
   and tile requests whenever a request comes in in the WMTS service
   format.

   As it is now, presumably WMTS doesn't work with Mapnik, MapServer,
   GDAL, etc. backends. This ireally counter to the design of TileCache,
   and I'm sorry that was unclear.

   Note that a WMTS layer -- that is, a layer class which requested
   tiles fro ma WMTS service -- is not out of the question, and could be
   implemented -- as could a TMS layer, for example, but I don't think
   that either of those is what has been built here.

   
 * top_left=-180,90
   Generally, this can simply be extracted from the bbox, though I see
   here that different levels can have different 'top left' expectations
   (hooray for setting up a standard that is completely different from
   everything else). I guess that this means that WMTS really isn't the
   same, though I still feel that it's probably better to not go bother
   writing another cache, and simply document that you can't use WMS and
   WMTS on the same cache. Porting each of the Layer and Cache classes
   to WMTS would mean duplicating tons of code.

> The implementation got a fair bit of testing in the OWS-6 testbed, but
> since then I've implemented support for MetaTiling, which only I've
> tested. A couple of things I came up with while merging with the
> latest revision and writing the documentation were:
>
>
> -          I avoided changes to existing TileCache files as much as
> possible. The goal was to keep the WMTS functionality contained to its
> Service, Layer and Disk module files as much as possible. When it
> comes to the changes to support reading TileMatrixSet configurations
> though I had a hard time separating this out from Service.py. I wasn't
> keen to subclass Service.py, but I don't like how much complexity has
> been added to the configuration reading. If anyone can suggest
> something that'd be great.

I really think that -- althogh your effort was admirable -- I feel like
a lot of this work was simply done in the wrong place/way.

>
> -          MetaTile was subclassed with WMTSTile so it could keep
> track of tile_matrix (used instead of resolution to go from tile
> co-ords to bounds). After implementing meta tiling for WMTS I think It
> may have been better to add tile_matrix to the Tile class instead, and
> WMTS specific code could look for it in the class if they wanted to
> use it.

Describe tile_matrix more, and I can probably comment on it. What is a
tile_matrix? What does it map to most closely in existing code?

>
> -          top_left configuration setting - I don't like this name
> because it causes a bit of confusion. The setting wants a coordinate
> which is usually ordered left,top and not top,left. The top_left
> matches the wording as it comes out in the WMTS capabilities document,
> but perhaps this should be something like 'origin'? The problem with
> origin is it doesn't state what corner of the bounds is used as the
> origin. I don't like left_top either - it may be more accurate than
> top_left most of the time but it still implies a specific order and
> now doesn't match WMTS (which is TopLeftCorner), so after all that I'm
> not sure what it should be called. Maybe top_left_corner?

If the ordering is always 'x,y' (or lon,lat) and not some freaky changed
thing with projections, then I'd preer to just call it left_top and be
done with it.

>
> -          There is a pending change request that was popular to make
> style always be specified in the path. When this change occurs, If no
> style is set in configuration then a default style name should be set
> and used in the paths which won't affect the WMS getMap requests. This
> would be easy enough to do but isn't *quite* yet standard WMTS
> behavior . Should it get done now?

Not enough information to understand the question.

Regards,
--
Christopher Schmidt
MetaCarta
_______________________________________________
Tilecache mailing list
[hidden email]
http://openlayers.org/mailman/listinfo/tilecache

[Jim Groffen]

Hello Chris,

I apologize for the size of this email, but I'm trying to be complete. I've responded to your questions below and further on talk about an idea for incorporating your feedback.

1 Currently, TileCache is designed to support accessing any tile set
  with any cache backend. Since all tiling schemes essentially work the
  same -- specifically, they all work on a fixed grid, with a set of
  scales and reslutions that allows a multilevel fixed grid, typically
  with the same corners at all levels of the grid -- it is easiest to
  transform this grid into a fixed set.

  It seems like the WMTS layer does not do this, and invents its own
  cache instead.

  Is the cache set up by WMTS different enough that this is *really*
  neccesary?

TileCache does work with WMTS layers using disk cache and should work with any other cache too (I didn't test all of them). It does this by turning a WMTS layer definition into many TileCache compatible layers. Each TileCache layer is a unique combination of the style, dimensions and tile matrix sets for a layer. Before WMTS came along if you wanted the same layer with two different styles in TileCache you would make two different layers. That's how the WMTS implementation deals with it. Around line 118 of Service.py is where, during configuration file reading, a WMTS layer is found and a call to TileCache.Layers.WMTS.load_wmts_layer is made.

Given the following WMTS layer in the config file:

[basic]
type=WMTS
...
...
title=Basic
tile_matrix_sets=WholeWorld_CRS_84,World84-90_CRS_84

If was ask for a TMS capabilities document we see two layers, one for each tile matrix set. The name for each layer is a string join("_") of layer name, style if present, dimensions if present and tile matrix set. Every possible combination is treated as its own layer because they will be cached separately on disk. An example TMS capabilities file is attached.

It is true that when using a cache of type WMTS, unless passed a WMTSTile instance it will raise a TileCacheException. WMTSTile instances are at the moment only coming from a WMTS layer, because they need to have a tile matrix set. If the layer associated to the tile has style or dimension information then that gets used too.

2 It appears that WMTS has been developed as a 'Layer'. Layers are ways
  of accessing backend services -- so the MapServer layer accesses a
  MapServer service, the WMS layer accesses a WMS service, etc. WMTS
  does not appear to be this (since you're pointing at my WMS for your
  demo config). Instead, it appears that WMTS should be a *service*
  (TileCache/Services/ directory); the configuration parameters that
  you are using to define tilesets should potentially be added to *any*
  layer, and these configuration parameters are used to return metadata
  and tile requests whenever a request comes in in the WMTS service
  format.

  As it is now, presumably WMTS doesn't work with Mapnik, MapServer,
  GDAL, etc. backends. This ireally counter to the design of TileCache,
  and I'm sorry that was unclear.

  Note that a WMTS layer -- that is, a layer class which requested
  tiles from a WMTS service -- is not out of the question, and could be
  implemented -- as could a TMS layer, for example, but I don't think
  that either of those is what has been built here.

In my current implementation when a WMTS layer is seen in the config, TileCache.Layers.WMTS interprets it into many WMS layers. In other words TileCache.Layers.WMTS provides support for WMTS layer definitions in the config. This is as you say supporting WMTS for only one backend service and not all backend services.

Also: when a WMTS request comes in it goes to the WMTS Service, which in turn will hand it on to the WMTS Layer if it needs to get the tile from the backend. It would be possible to have the WMTS Service go to any layer to handle the backend as other Service implementations do, but it would mean pushing WMTS specific implementation bits down to the backends. The reason
I didn't do this is because right now "Layers.WMS" knows how to handle tiles the TileCache way where Layers.WMTS knows how to handle tiles the WMTS way. They are different and I can't consolidate them easily. I'll get into that after I've explained TileMatrixSets

3 I really think that -- althogh your effort was admirable -- I feel like
  a lot of this work was simply done in the wrong place/way.

Having as little impact on the existing TileCache implementation as possible and isolating the WMTS code has lead me to not implement WMTS the way it should have been?. When put that way it does seem like I put that goal over a correct implementation but that's not the case. I'll be honest - I didn't know TileCache all that well when I started but I know it much better now and these emails are improving my understanding even more. This is why the implementation is not exactly how you would like it to be. I'm aware that most users and contributors to TileCache won't be interested in WMTS and didn't want to be the cause of added complexity for them but I wouldn't sacrifice the quality of the implementation for it. Looking back, I should have badgered you earlier and more often than I did to get your feedback on my implementation plans.

4 Describe tile_matrix more, and I can probably comment on it. What is a
  tile_matrix? What does it map to most closely in existing code?

A tile matrix set controls what tiles are available for a layer. In TileCache terms a Tile Matrix Set replaces the bounding box, resolutions list, image size and CRS. A tile matrix set is made up of multiple "TileMatrix", one for each zoom level. This means a Tile Matrix maps to one element of the resolution array, but because WMTS supports different bounds and image sizes per zoom level, it also maps to bounding box and image size. In TileCache configuration I've made it so that you can set things like top left point and image width / height at the Tile Matrix Set level and all tile matrices for that level will inherit the same settings. The WMTSGuide-excerpt.txt attachment has the documentation I wrote on WMTS configuration of Tile Matrix Sets in it. It's from docs/WMTSGuide.txt This is why Layers.WMTS handles asking for tiles from the backend - why WMTS is its own backend.



Based on your feedback, let's think of an alternative implementation:

First let's get rid of Layers.WMTS and move the config handling part somewhere else (say the Service class, that's where other layers are created from config). Now when config sees a layer of any type with WMTS properties (styles, dimensions, tilematrixsets) it will instead create many layer instances to handle each style/dimension/tilematrixset combination. When a WMTS request comes in, the WMTS service will find and use the right layer for the unique style/dimension/tilematrixset combination of that request.

Now we are left with the problem that the way TileCache addresses tiles is different to how WMTS does it. I'd say they are different, but not incompatible. The way TileCache does it with resolutions, bbox, image size and crs can be represented in a TileMatrixSet, we could support the existing config files and generate a TileMatrixSet instead. Then each backend would use the tile matrix set instead of resolution, image width / height, bbox, crs. This would work without the need for code duplication but it would impact every layer and service. Also, not all of the capabilities of WMTS are supported by every layer (such as dimensions and multiple styles) - Of course they don't have to be supported right away or even at all in some cases.

Is this headed in the right direction? Is it better than what we have at the moment where WMTS is only supported for the WMS backend? While much of the code needed (at least for WMS service and back end, and the WMTS service) is already done I am daunted by the amount of work involved, any suggestions?



These last two points are less contentious.

5 If the ordering is always 'x,y' (or lon,lat) and not some freaky changed
  thing with projections, then I'd preer to just call it left_top and be
  done with it.

It will pretty much always be x, y. It's meant to be in the projection that the tile matrix set is using. Based on your feedback though I'm going to switch it to left_top and tone down the warning I have in the WMTSGuide.txt about it (that's in the attached excerpt too).

6 There is a pending change request...

The OGC recently have a motion to "Approve Web Map Tiling Service (WMTS)" [07-057r7] as an OGC implementation standard". The vote doesn't close until 2009-11-01 but if the change request I'm talking about is in there then I'll sort it out.

Regards,

Jim Groffen.


-----Original Message-----
From: Christopher Schmidt [mailto:[hidden email]]
Sent: Thursday, 3 September 2009 9:41 PM
To: Jim Groffen
Cc: [hidden email]; Stefan Hansen; John Hudson
Subject: Re: [Tilecache] Patch for WMTS in TileCache

On Thu, Jun 25, 2009 at 07:38:26PM +0930, Jim Groffen wrote:
> Hello,

Sorry this has taken so long.

> Attached is a patch file that, if I made it correctly, will update TileCache revision 391 with the WMTS capabilities. I've added a guide (WMTSGuide.txt) and sample configuration file (wmts.cfg) to the docs folder that describes the new features.

Having briefly reviewed this patch, I have the following questions:

 * Currently, TileCache is designed to support accessing any tile set
   with any cache backend. Since all tiling schemes essentially work the
   same -- specifically, they all work on a fixed grid, with a set of
   scales and reslutions that allows a multilevel fixed grid, typically
   with the same corners at all levels of the grid -- it is easiest to
   transform this grid into a fixed set.

   It seems like the WMTS layer does not do this, and invents its own
   cache instead.

   Is the cache set up by WMTS different enough that this is *really*
   neccesary?

 * It appears that WMTS has been developed as a 'Layer'. Layers are ways
   of accessing backend services -- so the MapServer layer accesses a
   MapServer service, the WMS layer accesses a WMS service, etc. WMTS
   does not appear to be this (since you're pointing at my WMS for your
   demo config). Instead, it appears that WMTS should be a *service*
   (TileCache/Services/ directory); the configuration parameters that
   you are using to define tilesets should potentially be added to *any*
   layer, and these configuration parameters are used to return metadata
   and tile requests whenever a request comes in in the WMTS service
   format.

   As it is now, presumably WMTS doesn't work with Mapnik, MapServer,
   GDAL, etc. backends. This ireally counter to the design of TileCache,
   and I'm sorry that was unclear.

   Note that a WMTS layer -- that is, a layer class which requested
   tiles fro ma WMTS service -- is not out of the question, and could be
   implemented -- as could a TMS layer, for example, but I don't think
   that either of those is what has been built here.


 * top_left=-180,90
   Generally, this can simply be extracted from the bbox, though I see
   here that different levels can have different 'top left' expectations
   (hooray for setting up a standard that is completely different from
   everything else). I guess that this means that WMTS really isn't the
   same, though I still feel that it's probably better to not go bother
   writing another cache, and simply document that you can't use WMS and
   WMTS on the same cache. Porting each of the Layer and Cache classes
   to WMTS would mean duplicating tons of code.

> The implementation got a fair bit of testing in the OWS-6 testbed, but
> since then I've implemented support for MetaTiling, which only I've
> tested. A couple of things I came up with while merging with the
> latest revision and writing the documentation were:
>
>
> -          I avoided changes to existing TileCache files as much as
> possible. The goal was to keep the WMTS functionality contained to its
> Service, Layer and Disk module files as much as possible. When it
> comes to the changes to support reading TileMatrixSet configurations
> though I had a hard time separating this out from Service.py. I wasn't
> keen to subclass Service.py, but I don't like how much complexity has
> been added to the configuration reading. If anyone can suggest
> something that'd be great.

I really think that -- althogh your effort was admirable -- I feel like
a lot of this work was simply done in the wrong place/way.

>
> -          MetaTile was subclassed with WMTSTile so it could keep
> track of tile_matrix (used instead of resolution to go from tile
> co-ords to bounds). After implementing meta tiling for WMTS I think It
> may have been better to add tile_matrix to the Tile class instead, and
> WMTS specific code could look for it in the class if they wanted to
> use it.

Describe tile_matrix more, and I can probably comment on it. What is a
tile_matrix? What does it map to most closely in existing code?

>
> -          top_left configuration setting - I don't like this name
> because it causes a bit of confusion. The setting wants a coordinate
> which is usually ordered left,top and not top,left. The top_left
> matches the wording as it comes out in the WMTS capabilities document,
> but perhaps this should be something like 'origin'? The problem with
> origin is it doesn't state what corner of the bounds is used as the
> origin. I don't like left_top either - it may be more accurate than
> top_left most of the time but it still implies a specific order and
> now doesn't match WMTS (which is TopLeftCorner), so after all that I'm
> not sure what it should be called. Maybe top_left_corner?

If the ordering is always 'x,y' (or lon,lat) and not some freaky changed
thing with projections, then I'd preer to just call it left_top and be
done with it.

>
> -          There is a pending change request that was popular to make
> style always be specified in the path. When this change occurs, If no
> style is set in configuration then a default style name should be set
> and used in the paths which won't affect the WMS getMap requests. This
> would be easy enough to do but isn't *quite* yet standard WMTS
> behavior . Should it get done now?

Not enough information to understand the question.

Regards,
--
Christopher Schmidt
MetaCarta

The contents of this email are confidential and may be subject to legal or professional privilege and copyright. No representation is made that this email is free of viruses or other defects. If you have received this communication in error, you may not copy or distribute any part of it or otherwise disclose its contents to anyone. Please advise the sender of your incorrect receipt of this correspondence.


Tile Matrix Sets
================
WMTS requires the ability to define Tile Matrix Sets. A tile
matrix set controls what tiles are available for a layer.
In TileCache terms a Tile Matrix Set replaces the layer bounding
box, resolutions list, image size and CRS.

In the TileCache configuration file, each section represents a
TileCache layer except for two special sections - the [cache] and
[tilecache_options] sections. To define a Tile Matrix Set, add
the following option to the [tilecache_options] section:

  tile_matrix_sets=WholeWorld_CRS_84,World84-90_CRS_84

This is a comma separated list of configuration file section names
to treat as TileMatrixSets instead of a TileCache layer.

A Tile Matrix Set consists of a series of Tile Matrices and some
common parameters. As per TileCache layers, the section name becomes
the name of the tile matrix set.

TileMatrixSet Configuration
=========================
crs
        CRS of the tile matrix set. Default is EPSG:4326. The tile matrix
        set CRS and the layer CRS don't need to match.
identifiers
        Comma separated list of tile matrix identifiers.
top_left
        Upper left point of the matrix set in the CRS specified.
        Applied to every tile matrix. EG: -180,90
        A warning: Being named top_left causes confusion as co-ordinates  
        must be in the order specified by the CRS, which for the
        default EPSG:4326 is LEFT,TOP.
tile_size
        Width and height of each tile in the tile matrix set. EG: 256,256
wellknown_scale_set
        If set, the WMTS capabilities document will verify that the  
        tile matrix set conforms to (and can be overlayed with) tiles from
        a tile matrix set that uses the same well known scale set. Does not
        enforce conformity. Example: GlobalCRS84Pixel
<identifier>__matrix_size
        Width and height of the tile matrix in tiles.  
<identifier>__scale_denominator
        The scale denominator for the tile matrix identified by <identifier>  
<identifier>__tile_size
        You can set the tile size per tile matrix if you need to.  
<identifier>__top_left
        You can set the top left coordinate per tile matrix if you need to.
        This overrides the top_left setting for the tile matrix set if
        there is one. Remember that this is a point using the CRS of this
        tile matrix set, so not necessarily top,left (in fact by default
        it's left,top).

_______________________________________________
Tilecache mailing list
[hidden email]
http://openlayers.org/mailman/listinfo/tilecache

[Christopher Schmidt-2]

On Fri, Sep 11, 2009 at 04:30:12PM +0930, Jim Groffen wrote:

> Hello Chris,
>
> I apologize for the size of this email, but I'm trying to be complete. I've responded to your questions below and further on talk about an idea for incorporating your feedback.
>
> 1 Currently, TileCache is designed to support accessing any tile set
>   with any cache backend. Since all tiling schemes essentially work the
>   same -- specifically, they all work on a fixed grid, with a set of
>   scales and reslutions that allows a multilevel fixed grid, typically
>   with the same corners at all levels of the grid -- it is easiest to
>   transform this grid into a fixed set.
>
>   It seems like the WMTS layer does not do this, and invents its own
>   cache instead.
>
>   Is the cache set up by WMTS different enough that this is *really*
>   neccesary?
>
> TileCache does work with WMTS layers using disk cache and should work with any other cache too (I didn't test all of them). It does this by turning a WMTS layer definition into many TileCache compatible layers. Each TileCache layer is a unique combination of the style, dimensions and tile matrix sets for a layer. Before WMTS came along if you wanted the same layer with two different styles in TileCache you would make two different layers. That's how the WMTS implementation deals with it. Around line 118 of Service.py is where, during configuration file reading, a WMTS layer is found and a call to TileCache.Layers.WMTS.load_wmts_layer is made.
>
> Given the following WMTS layer in the config file:
>
> [basic]
> type=WMTS
> ...
> ...
> title=Basic
> tile_matrix_sets=WholeWorld_CRS_84,World84-90_CRS_84
>
> If was ask for a TMS capabilities document we see two layers, one for each
> tile matrix set. The name for each layer is a string join("_") of layer name,
> style if present, dimensions if present and tile matrix set. Every possible
> combination is treated as its own layer because they will be cached
> separately on disk. An example TMS capabilities file is attached.

Okay, this is a valuable piece of information. This might lead to a
better implementation suggestion; thinking through it.

> It is true that when using a cache of type WMTS, unless passed a WMTSTile
> instance it will raise a TileCacheException. WMTSTile instances are at the
> moment only coming from a WMTS layer, because they need to have a tile matrix
> set. If the layer associated to the tile has style or dimension information
> then that gets used too.

I see (upon reading more carefully) that the WMTS disk cache is
apparently designed to support some sort of WMTS behavior where files
are read directly from disk, and that's why it exists. I was concerned
on reading it, but am no longer; objections to this are withdrawn.

> 2 It appears that WMTS has been developed as a 'Layer'. Layers are ways
>   of accessing backend services -- so the MapServer layer accesses a
>   MapServer service, the WMS layer accesses a WMS service, etc. WMTS
>   does not appear to be this (since you're pointing at my WMS for your
>   demo config). Instead, it appears that WMTS should be a *service*
>   (TileCache/Services/ directory); the configuration parameters that
>   you are using to define tilesets should potentially be added to *any*
>   layer, and these configuration parameters are used to return metadata
>   and tile requests whenever a request comes in in the WMTS service
>   format.
>
>   As it is now, presumably WMTS doesn't work with Mapnik, MapServer,
>   GDAL, etc. backends. This ireally counter to the design of TileCache,
>   and I'm sorry that was unclear.
>
>   Note that a WMTS layer -- that is, a layer class which requested
>   tiles from a WMTS service -- is not out of the question, and could be
>   implemented -- as could a TMS layer, for example, but I don't think
>   that either of those is what has been built here.
>
> In my current implementation when a WMTS layer is seen in the config,
> TileCache.Layers.WMTS interprets it into many WMS layers. In other
> words TileCache.Layers.WMTS provides support for WMTS layer
> definitions in the config. This is as you say supporting WMTS for only
> one backend service and not all backend services.

So, what The WMTS layer object actually does, as I understand it, is
create a *set* of TileCache layers; each of those TileCache layers is
(currently) a WMS layer. Is that correct? This makes me wonder if
something that could (Thought not neccesarily 'should') be done is to
change the code to instead be a utility which takes a set of WMTS
configuration and turns it into a TileCache config.

My expectations for WMTS was that the only thing that would be needed
would be a 'Service' class. This Service class would take in requests,
and return the correct tile. That class appears to exist; thank you for
writing it. It returns the MEtaData, maps tile requests to internal
tiles, etc. (Somehow, in my first readthrough, I totally missed this.)

It appears there is additional code in the form of the Tile and Layer,
as well as the TileMatrix Layer object. I am less certain how these are
meant to work, and concerned that people might not want to use them the
way that they are currently designed. (For example, wanting to use WMTS
against a Mapnik backend.) If, instead of a class which 'magically'
creates multiple layers, there was a pre-processing step which generated
the TileCache configs, it seems like it's possible that if someone
wanted to (as I suggest) use Mapnik instead of WMS, they could tweak the
output config.

In order to accomodate for the WMTS specific layer properties, the base
layer could be extended; for example, you could add a "wmts_layerset"
key/value param. Then all layers which had the same wmts_layerset would
get put together in the output capabilities.

> Also: when a WMTS request comes in it goes to the WMTS Service, which
> in turn will hand it on to the WMTS Layer if it needs to get the tile
> from the backend. It would be possible to have the WMTS Service go to
> any layer to handle the backend as other Service implementations do,
> but it would mean pushing WMTS specific implementation bits down to
> the backends. The reason I didn't do this is because right now
> "Layers.WMS" knows how to handle tiles the TileCache way where
> Layers.WMTS knows how to handle tiles the WMTS way. They are different
> and I can't consolidate them easily. I'll get into that after I've
> explained TileMatrixSets

Layers.WMS takes a tile (layername, x, y, z) and fetches it from a
server. This is the part of the Layers.WMTS in 'renderTile'. This method
looks pretty much the same as the WMS layer to me. The additional
configuration properties on the layer look like something that could
potentially be pushed as options into the base Layer.

This leaves a couple things in the Layer.WMTS:
 * GetFeatureInfo. In my opinion, this should be removed. TileCache
   is really not designed to be fetching and caching feature data,
   in my opinion; or if it is, then it should be something we work on
   more thoroughly than just pushing it into this one layer type.
 * Dimensions? Not entirely sure about this, but they just look like
   extra KV Pairs, right?
 * The "WMTSTile" stuff, which could be a Layer-level option -- "When
   creating Tiles, use this subclass."

> 3 I really think that -- althogh your effort was admirable -- I feel like
>   a lot of this work was simply done in the wrong place/way.
>
> Having as little impact on the existing TileCache implementation as
> possible and isolating the WMTS code has lead me to not implement WMTS
> the way it should have been?

While that's certainly a possibility, I don't think that's actually the
case; but even if it was, I think we can both admit that 'changing as
little code as possible when adding a major feature' is likely to
hamstring any development through no fault of your own (and only the
fault of a mostly absent maintainer, imho).

> When put that way it does seem like I put that goal over a correct
> implementation but that's not the case.
> I'll be honest - I didn't know TileCache all that well when I started
> but I know it much better now and these emails are improving my
> understanding even more. This is why the implementation is not exactly
> how you would like it to be. I'm aware that most users and
> contributors to TileCache won't be interested in WMTS and didn't want
> to be the cause of added complexity for them but I wouldn't sacrifice
> the quality of the implementation for it. Looking back, I should have
> badgered you earlier and more often than I did to get your feedback on
> my implementation plans.

That's okay. I should have badgered you more on what you were working
on; working in a vaccum was definitely not the result of your lack of
trying to get me interested. (My fundamental dislike of the spec,
especially as it seems to have gotten so much more ridiculous in more
recent revisions, plays a significant role, obviously.)

> 4 Describe tile_matrix more, and I can probably comment on it. What is a
>   tile_matrix? What does it map to most closely in existing code?
>
> A tile matrix set controls what tiles are available for a layer. In
> TileCache terms a Tile Matrix Set replaces the bounding box,
> resolutions list, image size and CRS. A tile matrix set is made up of
> multiple "TileMatrix", one for each zoom level. This means a Tile
> Matrix maps to one element of the resolution array, but because WMTS
> supports different bounds and image sizes per zoom level, it also maps
> to bounding box and image size. In TileCache configuration I've made
> it so that you can set things like top left point and image width /
> height at the Tile Matrix Set level and all tile matrices for that
> level will inherit the same settings. The WMTSGuide-excerpt.txt
> attachment has the documentation I wrote on WMTS configuration of Tile
> Matrix Sets in it. It's from docs/WMTSGuide.txt This is why
> Layers.WMTS handles asking for tiles from the backend - why WMTS is
> its own backend.

>
>
> Based on your feedback, let's think of an alternative implementation:
>
> First let's get rid of Layers.WMTS and move the config handling part
> somewhere else (say the Service class, that's where other layers are
> created from config). Now when config sees a layer of any type with
> WMTS properties (styles, dimensions, tilematrixsets) it will instead
> create many layer instances to handle each
> style/dimension/tilematrixset combination. When a WMTS request comes
> in, the WMTS service will find and use the right layer for the unique
> style/dimension/tilematrixset combination of that request.

Yes, I think this is how I would have it done. The only question in my
mind, given that, is where the first part lives. My suggestion earlier
of a config file 'preprocessor' which takes these config options and
writes out a 'regular' tilecache config was designed to address this.

> Now we are left with the problem that the way TileCache addresses
> tiles is different to how WMTS does it. I'd say they are different,
> but not incompatible. The way TileCache does it with resolutions,
> bbox, image size and crs can be represented in a TileMatrixSet, we
> could support the existing config files and generate a TileMatrixSet
> instead. Then each backend would use the tile matrix set instead of
> resolution, image width / height, bbox, crs. This would work without
> the need for code duplication but it would impact every layer and
> service. Also, not all of the capabilities of WMTS are supported by
> every layer (such as dimensions and multiple styles) - Of course they
> don't have to be supported right away or even at all in some cases.

It seems to me that 'dimensions' and 'multiple styles' are likely to be
a WMS-only option. If we went with the 'preprocessing' method I
suggested above, and someone said "Okay, here is my WMTS config, and I
want you to write out a bunch of Mapnik layers, with these
'dimensions'", the script might throw an error saying "only WMS supports
dimensions."

> Is this headed in the right direction? Is it better than what we have
> at the moment where WMTS is only supported for the WMS backend? While
> much of the code needed (at least for WMS service and back end, and
> the WMTS service) is already done I am daunted by the amount of work
> involved, any suggestions?

So, first of all, let's put the work you have in a branch, and get you
commit access to TileCache SVN. http://svn.tilecache.org/branches/wmts/
, and I'll email you a password.

Next, let's think about what the right implementation is. *My* desire is
to continue to use bbox/resolution/etc. for all layers, and if we need
to give the set of parameters that is a name, we can add additional data
to do that. So for example, we might extend the TileCache config to
support any sections which start with [tilematrix_blah] to describe a
tilematrix; then, you have two options when setting up a layer:

bbox=
resolution=
etc.=

*or*

tilematrix=blah

The Service init can then take those and 'look up' the tile matrix and
match it to the bbox/resolution.

The difficulties I see with this are that I'm not sure that a layer in
TileCache actually has all the config info you need: you mentioned that
each 'zoom level' can basically use a different tileset, and TileCache
doesn't really have a good way of doing that, and I"m not sure that I
want to extend it to do so. So I guess then the question is: "how
important is that?" Is it something that the spec absolutely requires?
is it something that we can avoid? (And who the hell is ever going to
use it?) Honestly, when there's a 'feature' in a spec that even ESRI
hasn't seen fit to implement, I Get a little bit concerned at the idea that
it is the Most Important Feature Ever; it seems an unlikely-to-be-used
feature to me.

If it really is important, then I guess we'll have to think about the
right way to do it.

So, here's what I think I would like, and you can tell me if this is
going to be useful:

 1. We keep the WMTS service. It looks good.
 2. We move the properties we need for WMTS "layers" into the base
    layer.
 3. We figure out whether we should be:
    * Creating Layers on the fly when the config is loaded
  or
    * Creating layers via a seperate tool that uses a WMTS-like config
      using the library functions you've already built. (The latter
      allows users to tweak the resulting configs more easily; the
      former allows the 'write and forget', but at the cost of the
      config now being 'magical' in that it creates additional layers.)

 4. We extend TileCache to support named descriptions of layer extents
    and resolutions (and whatever else). (These are 'tilematrixes', I believe.)

 5. For now, we punt on the "WMTSTile" and "WMTSCache" until we get the
     basics working. At that point, we can establish how to get the
     tiles written on disk in a wya that you would access them through
     the WMTS 'REST' service.

Does that sound reasonable?      


Regards,
--
Christopher Schmidt
MetaCarta
_______________________________________________
Tilecache mailing list
[hidden email]
http://openlayers.org/mailman/listinfo/tilecache

[Jim Groffen]

Hello again Chris,

> So, what The WMTS layer object actually does, as I understand it, is
> create a *set* of TileCache layers; each of those TileCache layers is
> (currently) a WMS layer. Is that correct? This makes me wonder if
> something that could (Thought not neccesarily 'should') be done is to
> change the code to instead be a utility which takes a set of WMTS
> configuration and turns it into a TileCache config.

That's right, it turns one WMTS definition in the config into many layers. It does do WMTS layers though as opposed to WMS layers because WMTS layers knows about WMTS specific stuff like per-zoom level TileMatrixSet stuff and supports getFeatureInfo. The alternative would be to make this generate any TileCache layer.

> It appears there is additional code in the form of the Tile and Layer,
> as well as the TileMatrix Layer object. I am less certain how these are
> meant to work, and concerned that people might not want to use them the
> way that they are currently designed. (For example, wanting to use WMTS
> against a Mapnik backend.) If, instead of a class which 'magically'
> creates multiple layers, there was a pre-processing step which generated
> the TileCache configs, it seems like it's possible that if someone
> wanted to (as I suggest) use Mapnik instead of WMS, they could tweak the
> output config.

TileMatrixSet contains the definition of a tile matrix set and holds a list of TileMatrix instances. The WMTS layers point to a particular tile matrix set and use that for tile addressing instead of crs, bbox and resolution.

Much of the complexity that caused the need for TileMatrix, TileMatrixSet, and a WMTS layer classes is due to the TileMatrixSet concept not being fully compatible with TileCache tile addressing. If we instead restrict the tile addressing capabilities to what TileCache already supports then much of this complexity will go away, and all layers will work without changes.

> In order to accomodate for the WMTS specific layer properties, the base
> layer could be extended; for example, you could add a "wmts_layerset"
> key/value param. Then all layers which had the same wmts_layerset would
> get put together in the output capabilities.

Cool! wmts_layerset is an exact example of a property that automagically happens now but would need to be pushed down into the regular layers (it's called wmts_layer at the moment, calling it wmts_layerset is better though). Other properties currently in the WMTS layer class that need to go somewhere else are:

- title (used in capabilities, common to all layers that share the same 'wmts_layerset')
- keywords (same as title)
- style (particular to this layer)
- dimensions (particular to this layer)
- units (used to go from a scale to a resolution)
- tile_matrix_set (particular to this layer)
- query_layers (to support getFeatureInfo)
- feature_count (to support getFeatureInfo)
- info_formats (to support getFeatureInfo)
- pixel_coord_params (to support getFeatureInfo, so you can override the i and j param names with something else if your WMS isn't using i and j).

Title and keyword could be stored for every layer, though they are unique to the wmts_layerset.

The tile_matrix_set would be needed for the layer to support WMTS cache format. When generating capabilities the WMTS Service uses this information in the same way dimension and style are used.

> Layers.WMS takes a tile (layername, x, y, z) and fetches it from a
> server. This is the part of the Layers.WMTS in 'renderTile'. This method
> looks pretty much the same as the WMS layer to me. The additional
> configuration properties on the layer look like something that could
> potentially be pushed as options into the base Layer.

The only difference is WMTS renderTile deals with style and dimension. Support for style and dimension for layers would need to be honored or ignored by each Layer, maybe throwing an exception stating style or dimension info isn't supported for that layer type. A while ago I saw a patch was added to support additional parameters to be passed through to the backend. Style and Dimension could be done with that instead. That way we'd inherit support for extra params implementation in other layers (if there is any, maybe that was WMS only).

> This leaves a couple things in the Layer.WMTS:
>  * GetFeatureInfo. In my opinion, this should be removed. TileCache
>    is really not designed to be fetching and caching feature data,
>    in my opinion; or if it is, then it should be something we work on
>    more thoroughly than just pushing it into this one layer type.

getFeatureInfo is an optional WMTS requirement, not a mandatory. The supporter in OWS-6 was keen for it to be supported.

> * Dimensions? Not entirely sure about this, but they just look like
>   extra KV Pairs, right?

Yep they are. WMS supports TIME and ELEVATION as named dimensions, plus any other named dimensions as well, but they get prefixed with "dim_". I was dealing with the "dim_" thing automagically. One layer can for example support a time dimension with any number of (say three) different values, an elevation dimension with say two different values and a wavelength with four different values. This does mean that we'd generate 3x2x4=24 unique TileCache layers for every possible combination of dimension value that results in a unique tile. It's rare for a layer to have more than one dimension if it has any, but it's easy to support as they are just extra key value pairs. Styles and tile matrix sets also multiply the number of possible visualizations and hence layers. It's like asking the question "Show me a map of this area at this point in time with this wavelength setting at this elevation in this style".

> * The "WMTSTile" stuff, which could be a Layer-level option -- "When
>   creating Tiles, use this subclass."

I think this problem is going to go away based on the solution.

>> Based on your feedback, let's think of an alternative implementation:
>>
>> First let's get rid of Layers.WMTS and move the config handling part
>> somewhere else (say the Service class, that's where other layers are
>> created from config). Now when config sees a layer of any type with
>> WMTS properties (styles, dimensions, tilematrixsets) it will instead
>> create many layer instances to handle each
>> style/dimension/tilematrixset combination. When a WMTS request comes
>> in, the WMTS service will find and use the right layer for the unique
>> style/dimension/tilematrixset combination of that request.
>
> Yes, I think this is how I would have it done. The only question in my
> mind, given that, is where the first part lives. My suggestion earlier
> of a config file 'preprocessor' which takes these config options and
> writes out a 'regular' tilecache config was designed to address this.

I like the automagic stuff myself. The existing implementation is pretty much done and it keeps TileCache configuration as a one step thing. I can see benefits to a config file generator though - gets rid of the config file reading complexity I added to Service.py by keeping it in a separate utility script - a script that could be run when TileCache is about to load
the config if the tilecache config file is not there. Which is going to be better for users of TileCache? there are benefits and drawbacks to both. I'll think about this one a bit more.

>> ...... Also, not all of the capabilities of WMTS are supported by
>> every layer (such as dimensions and multiple styles) - Of course they
>> don't have to be supported right away or even at all in some cases.
>
> It seems to me that 'dimensions' and 'multiple styles' are likely to be
> a WMS-only option. If we went with the 'preprocessing' method I
> suggested above, and someone said "Okay, here is my WMTS config, and I
> want you to write out a bunch of Mapnik layers, with these
> 'dimensions'", the script might throw an error saying "only WMS supports
> dimensions."

Agreed.

> Next, let's think about what the right implementation is. *My* desire is
> to continue to use bbox/resolution/etc. for all layers, and if we need
> to give the set of parameters that is a name, we can add additional data
> to do that. So for example, we might extend the TileCache config to
> support any sections which start with [tilematrix_blah] to describe a
> tilematrix; then, you have two options when setting up a layer:
>
> bbox=
> resolution=
> etc.=
>
> *or*
>
> tilematrix=blah
>
> The Service init can then take those and 'look up' the tile matrix and
> match it to the bbox/resolution.
>
> The difficulties I see with this are that I'm not sure that a layer in
> TileCache actually has all the config info you need: you mentioned that
> each 'zoom level' can basically use a different tileset, and TileCache
> doesn't really have a good way of doing that, and I"m not sure that I
> want to extend it to do so. So I guess then the question is: "how
> important is that?" Is it something that the spec absolutely requires?
> is it something that we can avoid? (And who the hell is ever going to
> use it?) Honestly, when there's a 'feature' in a spec that even ESRI
> hasn't seen fit to implement, I Get a little bit concerned at the idea that
> it is the Most Important Feature Ever; it seems an unlikely-to-be-used
> feature to me.
>
> If it really is important, then I guess we'll have to think about the
> right way to do it.

The importance of implementing as much of the spec as possible is
difficult for me to answer. My motivation is what's best for both TileCache
AND what's best for the WMTS spec. I agree that it's better to have the WMTS
service supported by all layers and I'm hoping when the dust settles a
nice way to support additional capabilities in the TileMatrixSet will
become apparent.

> So, here's what I think I would like, and you can tell me if this is
> going to be useful:
>
>  1. We keep the WMTS service. It looks good.

Great

>  2. We move the properties we need for WMTS "layers" into the base
>     layer.

Yep

>  3. We figure out whether we should be:
>     * Creating Layers on the fly when the config is loaded
>   or
>     * Creating layers via a seperate tool that uses a WMTS-like config
>       using the library functions you've already built. (The latter
>       allows users to tweak the resulting configs more easily; the
>       former allows the 'write and forget', but at the cost of the
>       config now being 'magical' in that it creates additional layers.)

Yep, I'm not sure which is best yet either.

>  4. We extend TileCache to support named descriptions of layer extents
>     and resolutions (and whatever else). (These are 'tilematrixes', I believe.)

Yep, that's exactly what a TileMatrixSet is. I wonder what the level of effort would be to move Layer properties like resolutions, crs, bbox etc into a LayerExtents class? Then we could have different LayerExtent implementations for different layer addressing schemes.

>  5. For now, we punt on the "WMTSTile" and "WMTSCache" until we get the
>      basics working. At that point, we can establish how to get the
>      tiles written on disk in a wya that you would access them through
>      the WMTS 'REST' service.

Agreed. My feeling is we won't need a WMTSTile anymore to support a WMTSCache. Instead as Tile includes the Layer it should get things like LayerExtent name (tile matrix set name), dimensions, styles from that now.

> Does that sound reasonable?

Coming along nicely Chris. Thanks!

I'm off to QLD for a week of fun in the sun but I'll still be listening to emails.

Regards,

Jim Groffen

The contents of this email are confidential and may be subject to legal or professional privilege and copyright. No representation is made that this email is free of viruses or other defects. If you have received this communication in error, you may not copy or distribute any part of it or otherwise disclose its contents to anyone. Please advise the sender of your incorrect receipt of this correspondence.
_______________________________________________
Tilecache mailing list
[hidden email]
http://openlayers.org/mailman/listinfo/tilecache