PATCH: X-SendFile and Expires headers

[Mark Deneen-2]

This patch adds X-SendFile and the ability to set an Expires header.  It is a patch against tilecache 2.10.

Once applied, you can specify two new options under [cache] in the tilecache.cfg:

sendfile=yes
    This instructs tilecache to not read the tile image and send the fully qualified file name back to the web server in a X-SendFile header.  This avoids both python and the web server from holding the tile in memory at the same time.  The web server serves the tile directly to the client.
expires=n
    ...where nnnn is the number of hours from access time you wish to have the tiles possibly cached in the client browser.

I've tested this with lighttpd and apache (with mod_xsendfile http://tn123.ath.cx/mod_xsendfile/), and it works perfectly.  I've not tried seeding the cache with this.  You might have to disable sendfile to do that.  I'll test tomorrow.

Best Regards,
Mark Deneen
Saucon Technologies
[sendfile_expires.diff]

---

diff -ru tilecache-2.10/TileCache/Caches/Disk.py ../tilecache.mod/TileCache/Caches/Disk.py
--- tilecache-2.10/TileCache/Caches/Disk.py 2008-12-14 09:32:29.000000000 -0500
+++ ../tilecache.mod/TileCache/Caches/Disk.py 2009-10-05 15:50:57.000000000 -0400
@@ -4,11 +4,14 @@
 import sys, os, time, warnings
 
 class Disk (Cache):
-    def __init__ (self, base = None, umask = '002', **kwargs):
+    def __init__ (self, base = None, sendfile = False, expire = False,  umask = '002', **kwargs):
         Cache.__init__(self, **kwargs)
         self.basedir = base
         self.umask = int(umask, 0)
-        
+        self.sendfile = sendfile and sendfile.lower() in ["yes", "y", "t", "true"]
+        self.expire = expire
+        if expire != False:
+            self.expire = long(expire) * 3600
         if sys.platform.startswith("java"):
             from java.io import File
             self.file_module = File
@@ -64,8 +67,11 @@
     def get (self, tile):
         filename = self.getKey(tile)
         if self.access(filename, 'read'):
-            tile.data = file(filename, "rb").read()
-            return tile.data
+            if self.sendfile:
+                return filename
+            else:
+                tile.data = file(filename, "rb").read()
+                return tile.data
         else:
             return None
 
diff -ru tilecache-2.10/TileCache/Service.py ../tilecache.mod/TileCache/Service.py
--- tilecache-2.10/TileCache/Service.py 2009-01-18 21:11:52.000000000 -0500
+++ ../tilecache.mod/TileCache/Service.py 2009-10-05 16:12:34.000000000 -0400
@@ -4,7 +4,7 @@
 
 class TileCacheException(Exception): pass
 
-import sys, cgi, time, os, traceback, ConfigParser
+import sys, cgi, time, os, traceback, email, ConfigParser
 import Cache, Caches
 import Layer, Layers
 
@@ -291,8 +291,17 @@
         fields = parse_formvars(environ)
 
         format, image = service.dispatchRequest( fields, path_info, req_method, host )
-        start_response("200 OK", [('Content-Type',format)])
-        return [image]
+        headers = [('Content-Type',format)]
+        if service.cache.sendfile:
+            headers.append(('X-SendFile', image))
+        if service.cache.expire:
+            headers.append(('Expires', email.Utils.formatdate(time.time() + service.cache.expire, False, True)))
+
+        start_response("200 OK", headers)
+        if service.cache.sendfile:
+            return []
+        else:
+            return [image]
 
     except TileCacheException, E:
         start_response("404 Tile Not Found", [('Content-Type','text/plain')])

---

_______________________________________________
Tilecache mailing list
[hidden email]
http://openlayers.org/mailman/listinfo/tilecache

[Christopher Schmidt-2]

On Mon, Oct 05, 2009 at 05:24:25PM -0400, [hidden email] wrote:
>
> This patch adds X-SendFile and the ability to set an Expires header.  It is a patch against tilecache 2.10.

Looks like the patch only patches the WSGI handler. It is possible to
add this to the rest of the handlers in time, right? I don't mind applying
it as is, with a caveat that I'd like to see a documentation option for each
of these, and specifying that they only work with wsgi at the oment,
with possible instructions on how difficult it would be to add support to
the cgi and mod python handlers.

-- Chris

> Once applied, you can specify two new options under [cache] in the tilecache.cfg:
>
> sendfile=yes
>     This instructs tilecache to not read the tile image and send the fully qualified file name back to the web server in a X-SendFile header.  This avoids both python and the web server from holding the tile in memory at the same time.  The web server serves the tile directly to the client.
> expires=n
>     ...where nnnn is the number of hours from access time you wish to have the tiles possibly cached in the client browser.
>
> I've tested this with lighttpd and apache (with mod_xsendfile http://tn123.ath.cx/mod_xsendfile/), and it works perfectly.  I've not tried seeding the cache with this.  You might have to disable sendfile to do that.  I'll test tomorrow.
>
> Best Regards,
> Mark Deneen
> Saucon Technologies
> diff -ru tilecache-2.10/TileCache/Caches/Disk.py ../tilecache.mod/TileCache/Caches/Disk.py
> --- tilecache-2.10/TileCache/Caches/Disk.py 2008-12-14 09:32:29.000000000 -0500
> +++ ../tilecache.mod/TileCache/Caches/Disk.py 2009-10-05 15:50:57.000000000 -0400
> @@ -4,11 +4,14 @@
>  import sys, os, time, warnings
>  
>  class Disk (Cache):
> -    def __init__ (self, base = None, umask = '002', **kwargs):
> +    def __init__ (self, base = None, sendfile = False, expire = False,  umask = '002', **kwargs):
>          Cache.__init__(self, **kwargs)
>          self.basedir = base
>          self.umask = int(umask, 0)
> -        
> +        self.sendfile = sendfile and sendfile.lower() in ["yes", "y", "t", "true"]
> +        self.expire = expire
> +        if expire != False:
> +            self.expire = long(expire) * 3600
>          if sys.platform.startswith("java"):
>              from java.io import File
>              self.file_module = File
> @@ -64,8 +67,11 @@
>      def get (self, tile):
>          filename = self.getKey(tile)
>          if self.access(filename, 'read'):
> -            tile.data = file(filename, "rb").read()
> -            return tile.data
> +            if self.sendfile:
> +                return filename
> +            else:
> +                tile.data = file(filename, "rb").read()
> +                return tile.data
>          else:
>              return None
>  
> diff -ru tilecache-2.10/TileCache/Service.py ../tilecache.mod/TileCache/Service.py
> --- tilecache-2.10/TileCache/Service.py 2009-01-18 21:11:52.000000000 -0500
> +++ ../tilecache.mod/TileCache/Service.py 2009-10-05 16:12:34.000000000 -0400
> @@ -4,7 +4,7 @@
>  
>  class TileCacheException(Exception): pass
>  
> -import sys, cgi, time, os, traceback, ConfigParser
> +import sys, cgi, time, os, traceback, email, ConfigParser
>  import Cache, Caches
>  import Layer, Layers
>  
> @@ -291,8 +291,17 @@
>          fields = parse_formvars(environ)
>  
>          format, image = service.dispatchRequest( fields, path_info, req_method, host )
> -        start_response("200 OK", [('Content-Type',format)])
> -        return [image]
> +        headers = [('Content-Type',format)]
> +        if service.cache.sendfile:
> +            headers.append(('X-SendFile', image))
> +        if service.cache.expire:
> +            headers.append(('Expires', email.Utils.formatdate(time.time() + service.cache.expire, False, True)))
> +
> +        start_response("200 OK", headers)
> +        if service.cache.sendfile:
> +            return []
> +        else:
> +            return [image]
>  
>      except TileCacheException, E:
>          start_response("404 Tile Not Found", [('Content-Type','text/plain')])

> _______________________________________________
> Tilecache mailing list
> [hidden email]
> http://openlayers.org/mailman/listinfo/tilecache

_______________________________________________
Tilecache mailing list
[hidden email]
http://openlayers.org/mailman/listinfo/tilecache

[Christopher Schmidt-2]

On Tue, Oct 06, 2009 at 11:02:56AM -0400, [hidden email] wrote:
> Chris,
>
> I've attached another patch.  This one adds a bit of documentation, support for the cgi handler and a bug fix to my previous patch.  It will only send the X-SendFile and Expires if the content type is not text/xml.  If text/xml is returned, the output should be identical to what it would be if one were not using X-SendFile.
>
> The patch is against TileCache 2.10, so you don't need to apply the previous patch first.

Applied, with a couple changes:
 * expires is in seconds
 * It checks for image/, rather than for text/xml, since KML is also
   not-cached and served via the KML service.
 * mod_python support added.

Commit was r396. I'll try and push a release with this and other bugfixes
relatively soon.

Thanks for the patch.

-- Chris

> Best Regards,
> Mark Deneen
> Saucon Technologies
>
> ----- "Christopher Schmidt" <[hidden email]> wrote:
>
> > On Tue, Oct 06, 2009 at 10:02:56AM -0400, Mark Deneen wrote:
> > > Where do you want the documentation for the new options?  They are
> > options for the Disk cache and the only documented options I see are
> > for the layers.  Should I just write up a description in
> > docs/Caches.txt?
> >
> > That'd be great. And no need to write the patch for the other
> > hanlders;
> > just mention in the docs what handlers the options work for.
> >
> > Don't worry too much about formatting; just get the text together, and
> >
> > I can fix any formatting issues.
> >
> > Regards,
> > --
> > Christopher Schmidt
> > MetaCarta
> diff -ru tilecache-2.10/docs/Caches.txt ../tilecache.mod/docs/Caches.txt
> --- tilecache-2.10/docs/Caches.txt 2008-12-14 09:59:49.000000000 -0500
> +++ ../tilecache.mod/docs/Caches.txt 2009-10-06 10:55:08.000000000 -0400
> @@ -12,9 +12,24 @@
>    type=Disk
>    base=/var/lib/tilecache
>    umask=002
> +  sendfile=yes
> +  expire=24
>  
>  Dependencies: None
>  
> +The sendfile=yes option instructs TileCache to send an X-SendFile header to
> +the web server rather than the contents of the image.
> +
> +See http://blog.lighttpd.net/articles/2006/07/02/x-sendfile and
> +http://tn123.ath.cx/mod_xsendfile/
> +
> +The expire=n option instructs TileCache to send an Expires header, with the
> +date set 'n' hours into the future.
> +
> +Both the sendfile and expire options are currently only supported with the cgi
> +and wsgi handlers. With minor changes, other handlers could easily be
> +supported.
> +
>  GoogleDisk
>  ----
>  Example Configuration::
> diff -ru tilecache-2.10/TileCache/Caches/Disk.py ../tilecache.mod/TileCache/Caches/Disk.py
> --- tilecache-2.10/TileCache/Caches/Disk.py 2008-12-14 09:32:29.000000000 -0500
> +++ ../tilecache.mod/TileCache/Caches/Disk.py 2009-10-05 15:50:57.000000000 -0400
> @@ -4,11 +4,14 @@
>  import sys, os, time, warnings
>  
>  class Disk (Cache):
> -    def __init__ (self, base = None, umask = '002', **kwargs):
> +    def __init__ (self, base = None, sendfile = False, expire = False,  umask = '002', **kwargs):
>          Cache.__init__(self, **kwargs)
>          self.basedir = base
>          self.umask = int(umask, 0)
> -        
> +        self.sendfile = sendfile and sendfile.lower() in ["yes", "y", "t", "true"]
> +        self.expire = expire
> +        if expire != False:
> +            self.expire = long(expire) * 3600
>          if sys.platform.startswith("java"):
>              from java.io import File
>              self.file_module = File
> @@ -64,8 +67,11 @@
>      def get (self, tile):
>          filename = self.getKey(tile)
>          if self.access(filename, 'read'):
> -            tile.data = file(filename, "rb").read()
> -            return tile.data
> +            if self.sendfile:
> +                return filename
> +            else:
> +                tile.data = file(filename, "rb").read()
> +                return tile.data
>          else:
>              return None
>  
> diff -ru tilecache-2.10/TileCache/Service.py ../tilecache.mod/TileCache/Service.py
> --- tilecache-2.10/TileCache/Service.py 2009-01-18 21:11:52.000000000 -0500
> +++ ../tilecache.mod/TileCache/Service.py 2009-10-06 10:47:46.000000000 -0400
> @@ -4,7 +4,7 @@
>  
>  class TileCacheException(Exception): pass
>  
> -import sys, cgi, time, os, traceback, ConfigParser
> +import sys, cgi, time, os, traceback, email, ConfigParser
>  import Cache, Caches
>  import Layer, Layers
>  
> @@ -291,8 +291,18 @@
>          fields = parse_formvars(environ)
>  
>          format, image = service.dispatchRequest( fields, path_info, req_method, host )
> -        start_response("200 OK", [('Content-Type',format)])
> -        return [image]
> +        headers = [('Content-Type',format)]
> +        if format != "text/xml":
> +            if service.cache.sendfile:
> +                headers.append(('X-SendFile', image))
> +            if service.cache.expire:
> +                headers.append(('Expires', email.Utils.formatdate(time.time() + service.cache.expire, False, True)))
> +
> +        start_response("200 OK", headers)
> +        if service.cache.sendfile and format != "text/xml":
> +            return []
> +        else:
> +            return [image]
>  
>      except TileCacheException, E:
>          start_response("404 Tile Not Found", [('Content-Type','text/plain')])
> @@ -321,12 +331,18 @@
>          host += os.environ["SCRIPT_NAME"]
>          req_method = os.environ["REQUEST_METHOD"]
>          format, image = service.dispatchRequest( params, path_info, req_method, host )
> -        print "Content-type: %s\n" % format
> -
> -        if sys.platform == "win32":
> -            binaryPrint(image)
> -        else:    
> -            print image
> +        print "Content-type: %s" % format
> +        if format != "text/xml":
> +            if service.cache.sendfile:
> +                print "X-SendFile: %s" % image
> +            if service.cache.expire:
> +                print "Expires: %s" % email.Utils.formatdate(time.time() + service.cache.expire, False, True)
> +        print "\n"
> +        if (not service.cache.sendfile) or (format == "text/xml"):
> +            if sys.platform == "win32":
> +                binaryPrint(image)
> +            else:    
> +                print image
>      except TileCacheException, E:
>          print "Cache-Control: max-age=10, must-revalidate" # make the client reload        
>          print "Content-type: text/plain\n"

--
Christopher Schmidt
MetaCarta
_______________________________________________
Tilecache mailing list
[hidden email]
http://openlayers.org/mailman/listinfo/tilecache

[Alexandre Dube]

Just read about X-SendFile [1].  I'll try that.

Regards,

Alexandre

[1]
http://n2.nabble.com/PATCH-X-SendFile-and-Expires-headers-td3771732.html#a3771732

[hidden email] wrote:

> This patch adds X-SendFile and the ability to set an Expires header.  It is a patch against tilecache 2.10.
>
> Once applied, you can specify two new options under [cache] in the tilecache.cfg:
>
> sendfile=yes
>     This instructs tilecache to not read the tile image and send the fully qualified file name back to the web server in a X-SendFile header.  This avoids both python and the web server from holding the tile in memory at the same time.  The web server serves the tile directly to the client.
> expires=n
>     ...where nnnn is the number of hours from access time you wish to have the tiles possibly cached in the client browser.
>
> I've tested this with lighttpd and apache (with mod_xsendfile http://tn123.ath.cx/mod_xsendfile/), and it works perfectly.  I've not tried seeding the cache with this.  You might have to disable sendfile to do that.  I'll test tomorrow.
>
> Best Regards,
> Mark Deneen
> Saucon Technologies
> ------------------------------------------------------------------------
>
> _______________________________________________
> Tilecache mailing list
> [hidden email]
> http://openlayers.org/mailman/listinfo/tilecache

--
Alexandre Dubé
Mapgears
www.mapgears.com

_______________________________________________
Tilecache mailing list
[hidden email]
http://openlayers.org/mailman/listinfo/tilecache