Can I fill a hole in an rrd?

6 messages Options
Embed this post
Permalink
Marc MERLIN

Can I fill a hole in an rrd?

Reply Threaded More More options
Print post
Permalink
If my monitoring fails, I get a hole in the file until I fix it.

The way to recover right now is:
1) stop rrd updates
2) delete, recreate rrd file
3) spend 20mn+ with a script that replays all my raw data and creates an rrd
update compatible file
4) feed the data into rrd with rrdtool update
5) restart rrd updates.

So, this both sucks for being time consuming and still creates a hole
between the time of the last update in my generated file, and feeding this
back to my newly created rrd file and when I restart rrd updates

Isn't there a way to get rrd update to fill in samples in the middle of my
file since I'm giving unix time indexes anyway?

Marc
--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems & security ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/ 

_______________________________________________
rrd-users mailing list
[hidden email]
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users
Marc MERLIN

Re: Can I fill a hole in an rrd?

Reply Threaded More More options
Print post
Permalink
On Tue, Oct 27, 2009 at 01:30:14PM -0700, Koaps wrote:
> well, are you doing a xport then restore? or replaying the values using update?
>
> If you xport to xml you can find the timestamp range where the hole is, take the last successful update value and replace all the NaN's in the timestamp range with the last successful update value.

I haven't done that yet since it's a different format than the one I used to
load up the RRD, but my understanding is that it's still kind of the same
problem that I have to stop updates, dump the file, fix it, restore it, and
restart updates, which will give me a hole.

I would really like to just feed the missing data samples in the RRD with
a bunch of
rrdtool update housepower_21.rrd 1242148828:500000853:500000071:500000070:499999218:499999999
commands to feed data samples in the middle of the file (since I'm giving
the time index) and not just at the end.

Is that not possible?

Marc
--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems & security ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/ 

_______________________________________________
rrd-users mailing list
[hidden email]
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users
Koaps

Re: Can I fill a hole in an rrd?

Reply Threaded More More options
Print post
Permalink
Well according to the man page you can but rrdtool throws an error.


http://www.mail-archive.com/rrd-developers@.../msg03067.html


You should be able to specify a negative time also but it does not work either, you will get an illegal attempt to update using time error.

>From that thread it was supposed to be patched in, but I guess it broke again.

-c


--- On Tue, 10/27/09, Marc MERLIN <[hidden email]> wrote:

> From: Marc MERLIN <[hidden email]>
> Subject: Re: [rrd-users] Can I fill a hole in an rrd?
> To: "Koaps" <[hidden email]>
> Cc: [hidden email]
> Date: Tuesday, October 27, 2009, 2:31 PM
> On Tue, Oct 27, 2009 at 01:30:14PM
> -0700, Koaps wrote:
> > well, are you doing a xport then restore? or replaying
> the values using update?
> >
> > If you xport to xml you can find the timestamp range
> where the hole is, take the last successful update value and
> replace all the NaN's in the timestamp range with the last
> successful update value.
>
> I haven't done that yet since it's a different format than
> the one I used to
> load up the RRD, but my understanding is that it's still
> kind of the same
> problem that I have to stop updates, dump the file, fix it,
> restore it, and
> restart updates, which will give me a hole.
>
> I would really like to just feed the missing data samples
> in the RRD with
> a bunch of
> rrdtool update housepower_21.rrd
> 1242148828:500000853:500000071:500000070:499999218:499999999
> commands to feed data samples in the middle of the file
> (since I'm giving
> the time index) and not just at the end.
>
> Is that not possible?
>
> Marc
> --
> "A mouse is a device used to point at the xterm you want to
> type in" - A.S.R.
> Microsoft is to operating systems & security ....
>                
>                
>       .... what McDonalds is to gourmet
> cooking
> Home page: http://marc.merlins.org/  
>


     

_______________________________________________
rrd-users mailing list
[hidden email]
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users
William Owen-2

Re: Can I fill a hole in an rrd?

Reply Threaded More More options
Print post
Permalink
I see two realistic options here. The best thing I can think of would be to dump to XML, manually insert the values you want to update into the XML and then import the XML into a new RRD. That, however would be tedious for a large dataset and you may run into issues normalizing your own data in the different CFs and RRAs. The other option would be to stream ALL values out of the one missing values, insert your data into the stream and then rrdupdate a new one sequentially with those values. You'd need to be careful here as fetch will pick the most "appropriate" RRA and feed data from that based on your fetch times.

-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Koaps
Sent: Tuesday, October 27, 2009 6:36 PM
To: Marc MERLIN
Cc: [hidden email]
Subject: Re: [rrd-users] Can I fill a hole in an rrd?

Well according to the man page you can but rrdtool throws an error.


http://www.mail-archive.com/rrd-developers@.../msg03067.html


You should be able to specify a negative time also but it does not work either, you will get an illegal attempt to update using time error.

>From that thread it was supposed to be patched in, but I guess it broke again.

-c


--- On Tue, 10/27/09, Marc MERLIN <[hidden email]> wrote:

> From: Marc MERLIN <[hidden email]>
> Subject: Re: [rrd-users] Can I fill a hole in an rrd?
> To: "Koaps" <[hidden email]>
> Cc: [hidden email]
> Date: Tuesday, October 27, 2009, 2:31 PM
> On Tue, Oct 27, 2009 at 01:30:14PM
> -0700, Koaps wrote:
> > well, are you doing a xport then restore? or replaying
> the values using update?
> >
> > If you xport to xml you can find the timestamp range
> where the hole is, take the last successful update value and
> replace all the NaN's in the timestamp range with the last
> successful update value.
>
> I haven't done that yet since it's a different format than
> the one I used to
> load up the RRD, but my understanding is that it's still
> kind of the same
> problem that I have to stop updates, dump the file, fix it,
> restore it, and
> restart updates, which will give me a hole.
>
> I would really like to just feed the missing data samples
> in the RRD with
> a bunch of
> rrdtool update housepower_21.rrd
> 1242148828:500000853:500000071:500000070:499999218:499999999
> commands to feed data samples in the middle of the file
> (since I'm giving
> the time index) and not just at the end.
>
> Is that not possible?
>
> Marc
> --
> "A mouse is a device used to point at the xterm you want to
> type in" - A.S.R.
> Microsoft is to operating systems & security ....
>
>
>       .... what McDonalds is to gourmet
> cooking
> Home page: http://marc.merlins.org/
>




_______________________________________________
rrd-users mailing list
[hidden email]
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

_______________________________________________
rrd-users mailing list
[hidden email]
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users
oetiker

Re: Can I fill a hole in an rrd?

Reply Threaded More More options
Print post
Permalink
In reply to this post by Koaps
Yesterday Koaps wrote:

> Well according to the man page you can but rrdtool throws an error.
>
>
> http://www.mail-archive.com/rrd-developers@.../msg03067.html
>
>
> You should be able to specify a negative time also but it does
> not work either, you will get an illegal attempt to update using
> time error.

Note that you can not go back in time with rrdtool. This is due
to the way it handles updates (re-sampling). The bug you are citing
(which is fixed) is about sending updates relative to the current
time. YOu still have to observe the strict ordering along the
timeline.


> >From that thread it was supposed to be patched in, but I guess it broke again.

cheers
tobi

--
Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten, Switzerland
http://it.oetiker.ch [hidden email] ++41 62 775 9902 / sb: -9900

_______________________________________________
rrd-users mailing list
[hidden email]
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users
Marc MERLIN

Re: Can I fill a hole in an rrd?

Reply Threaded More More options
Print post
Permalink
In reply to this post by William Owen-2
On Tue, Oct 27, 2009 at 09:48:08PM -0500, William Owen wrote:
> I see two realistic options here. The best thing I can think of would be
> to dump to XML, manually insert the values you want to update into the XML
> and then import the XML into a new RRD. That, however would be tedious
> for a large dataset and you may run into issues normalizing your own data
> in the different CFs and RRAs. The other option would be to stream ALL
> values out of the one missing values, insert your data into the stream and
> then rrdupdate a new one sequentially with those values. You'd need to be
> careful here as fetch will pick the most "appropriate" RRA and feed data
> from that based on your fetch times.

This is what I've been doing more or less: just recreate the rrd and feed
everything back into it, but it's just slow and tedious.

Hopefully a new version of rrdtool will allow for replacing samples at given
time offsets without having to change/touch everything else (yes, that means
that dependencies might have to be recalculated, but I'm hoping it can't be
that hard).

Marc
--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems & security ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/ 

_______________________________________________
rrd-users mailing list
[hidden email]
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users