Missing data overnight from daylight saving changes

4 messages Options
Embed this post
Permalink
Richard Curtis

Missing data overnight from daylight saving changes

Reply Threaded More More options
Print post
Permalink
Hi,
  I have a simple rrd to store filesystem utilisation on an hourly basis.
The data is gathered on other hosts, then processed into rrd via a
script run at 4am.

The rrd was created as:
rrdtool create myrouter.rrd    DS:used:GAUGE:3600:0:100
RRA:LAST:0.5:1:8760   --start 1256403600 --step 3600

And the following values entered:
rrdtool update myrouter.rrd 1256410800:39
rrdtool update myrouter.rrd 1256414400:39
rrdtool update myrouter.rrd 1256418000:39
rrdtool update myrouter.rrd 1256421600:39
rrdtool update myrouter.rrd 1256425200:39
rrdtool update myrouter.rrd 1256432400:39
rrdtool update myrouter.rrd 1256436000:39
rrdtool update myrouter.rrd 1256439600:39
rrdtool update myrouter.rrd 1256443200:39

but I have a gap in data between midnight of the 25-Oct (BST) and 2am
(GMT) on 26-Oct.
An rrdtool dump shows:
    <!-- 2009-10-24 20:00:00 BST / 1256410800 --> <row><v>
3.9000000000e+01 </v></row>
                        <!-- 2009-10-24 21:00:00 BST / 1256414400 -->
<row><v> 3.9000000000e+01 </v></row>
                        <!-- 2009-10-24 22:00:00 BST / 1256418000 -->
<row><v> 3.9000000000e+01 </v></row>
                        <!-- 2009-10-24 23:00:00 BST / 1256421600 -->
<row><v> 3.9000000000e+01 </v></row>
                        <!-- 2009-10-25 00:00:00 BST / 1256425200 -->
<row><v> 3.9000000000e+01 </v></row>
                        <!-- 2009-10-25 01:00:00 BST / 1256428800 -->
<row><v> NaN </v></row>
                        <!-- 2009-10-25 01:00:00 GMT / 1256432400 -->
<row><v> NaN </v></row>
                        <!-- 2009-10-25 02:00:00 GMT / 1256436000 -->
<row><v> 3.9000000000e+01 </v></row>
                        <!-- 2009-10-25 03:00:00 GMT / 1256439600 -->
<row><v> 3.9000000000e+01 </v></row>
                        <!-- 2009-10-25 04:00:00 GMT / 1256443200 -->
<row><v> 3.9000000000e+01 </v></row>

Am I missing something here?  I did insert a value for 256432400, but
it looks like rrdtool is expecting a value for both 01:00 BST and
01:00 GMT.
Do I need to do some magic when I insert the values and double input
when the clocks go back?

Richard

_______________________________________________
rrd-users mailing list
[hidden email]
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users
A Darren Dunham

Re: Missing data overnight from daylight saving changes

Reply Threaded More More options
Print post
Permalink
On Mon, Oct 26, 2009 at 03:53:57PM +0000, Richard Curtis wrote:
> Hi,
>   I have a simple rrd to store filesystem utilisation on an hourly basis.
> The data is gathered on other hosts, then processed into rrd via a
> script run at 4am.
>
> The rrd was created as:
> rrdtool create myrouter.rrd    DS:used:GAUGE:3600:0:100
> RRA:LAST:0.5:1:8760   --start 1256403600 --step 3600

So you've told RRD that unless a new value arrives within 3600 seconds,
it should assume the data between updates is unknown.

> And the following values entered:
> rrdtool update myrouter.rrd 1256410800:39
> rrdtool update myrouter.rrd 1256414400:39
> rrdtool update myrouter.rrd 1256418000:39
> rrdtool update myrouter.rrd 1256421600:39
> rrdtool update myrouter.rrd 1256425200:39
> rrdtool update myrouter.rrd 1256432400:39

And the gap between these two updates is 7200, which is greater than
3600.  So the data between is marked unknown.

> rrdtool update myrouter.rrd 1256436000:39
> rrdtool update myrouter.rrd 1256439600:39
> rrdtool update myrouter.rrd 1256443200:39
>
> but I have a gap in data between midnight of the 25-Oct (BST) and 2am
> (GMT) on 26-Oct.

Yup.

>                         <!-- 2009-10-24 22:00:00 BST / 1256418000 -->
> <row><v> 3.9000000000e+01 </v></row>
>                         <!-- 2009-10-24 23:00:00 BST / 1256421600 -->
> <row><v> 3.9000000000e+01 </v></row>
>                         <!-- 2009-10-25 00:00:00 BST / 1256425200 -->
> <row><v> 3.9000000000e+01 </v></row>
>                         <!-- 2009-10-25 01:00:00 BST / 1256428800 -->
> <row><v> NaN </v></row>
>                         <!-- 2009-10-25 01:00:00 GMT / 1256432400 -->
> <row><v> NaN </v></row>
>                         <!-- 2009-10-25 02:00:00 GMT / 1256436000 -->
> <row><v> 3.9000000000e+01 </v></row>
>                         <!-- 2009-10-25 03:00:00 GMT / 1256439600 -->
> <row><v> 3.9000000000e+01 </v></row>
>                         <!-- 2009-10-25 04:00:00 GMT / 1256443200 -->
> <row><v> 3.9000000000e+01 </v></row>
>
> Am I missing something here?  I did insert a value for 256432400, but
> it looks like rrdtool is expecting a value for both 01:00 BST and
> 01:00 GMT.

It needs a value for 1256428800 to keep the sequence intact.  

It doesn't care that one is 01:00 this or 01:00 that.  It just cares
that there's a gap of 7200 seconds, which exceeds the heartbeat limit of
3600.

> Do I need to do some magic when I insert the values and double input
> when the clocks go back?

Did you actaully collect a value for that point in time?  (One hour
after midnight BST).  It's difficult to tell if you're collecting one
point every hour and throwing it away in your post-processing script, or
if you're only collecting data on local clock hours and are missing the
time point.

Either way, RRD appears to be behaving correctly.  While the "fetch" is
translating the timestamps into local timezones, it really doesn't care
about the tiemzone.  It just needs the update every 3600 seconds and you
haven't provided that here.

--
Darren

_______________________________________________
rrd-users mailing list
[hidden email]
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users
Richard Curtis

Re: Missing data overnight from daylight saving changes

Reply Threaded More More options
Print post
Permalink
>> Am I missing something here?  I did insert a value for 256432400, but
>> it looks like rrdtool is expecting a value for both 01:00 BST and
>> 01:00 GMT.
>
> It needs a value for 1256428800 to keep the sequence intact.
>
> It doesn't care that one is 01:00 this or 01:00 that.  It just cares
> that there's a gap of 7200 seconds, which exceeds the heartbeat limit of
> 3600.

Ah, now it makes sense.
My script to collect the data runs via cron which ensures the script
only ones once at DST change, so I am missing the data point for 01:00
BST (rather, my loader script assumes the entry for 01:00 on the 25th
Oct is GMT so there is nothing inserted.

I've fixed in now based on this - I allow a 7200 second gap which
allows the graph to continue uninterrupted.

Thanks for your help on this!

Richard

_______________________________________________
rrd-users mailing list
[hidden email]
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users
oetiker

Re: Missing data overnight from daylight saving changes

Reply Threaded More More options
Print post
Permalink
Richard,

Today Richard Curtis wrote:

> >> Am I missing something here?  I did insert a value for 256432400, but
> >> it looks like rrdtool is expecting a value for both 01:00 BST and
> >> 01:00 GMT.
> >
> > It needs a value for 1256428800 to keep the sequence intact.
> >
> > It doesn't care that one is 01:00 this or 01:00 that.  It just cares
> > that there's a gap of 7200 seconds, which exceeds the heartbeat limit of
> > 3600.
>
> Ah, now it makes sense.
> My script to collect the data runs via cron which ensures the script
> only ones once at DST change, so I am missing the data point for 01:00
> BST (rather, my loader script assumes the entry for 01:00 on the 25th
> Oct is GMT so there is nothing inserted.
>
> I've fixed in now based on this - I allow a 7200 second gap which
> allows the graph to continue uninterrupted.
I missed the sart of this thread, but note that rrdtool internally
only deals in GMT/UTC as such it does not care about daylight
saving time ... it is merily a presentation thing.

cheers
tobi


> _______________________________________________
> rrd-users mailing list
> [hidden email]
> https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users
>
>

--
Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten, Switzerland
http://it.oetiker.ch [hidden email] ++41 62 775 9902 / sb: -9900
_______________________________________________
rrd-users mailing list
[hidden email]
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users