MRTG Mailinglists › MRTG Developers Mailinglist

Calculating 95th percentile

11 messages

Westlake, Simon () Calculating 95th percentile

Reply Threaded

Hello all,

I'm attempting to write a script to calculate 95th percentile usage
using RRD files generated using MRTG. I'm using PHP and PHP4RRDTool in
order to pull the information and calculate it and I'm running into an
issue which has maybe been hidden from me in the past as I use MRTG as a
collector for my RRD files.

A quick rundown of my method:

I have a loop that goes through and pulls 5 minute intervals of data
from the RRD file (by doing --start -10m --end -5m for example, and
moving backwards in steps of 5) for the last calendar month. It adds
this data to an array, sorts it from smallest to largest, strips the top
5% and calculates the average.

The script itself seems to work fine, but the data I get returned seems
weird.

After reading the RRD documentation, it appears that I should be able to
pull the information I want by using the following command:

$opts = array ( "AVERAGE", "--end", "-$timeframe", "--start",
"-$start");

Where the $opts array is an array of the options to send. This works and
gives me a value, however, the values seem to repeat identically. Here
is a (somewhat brief) example of the data returned over a series of 5
minute queries:

33748465.7228
33748465.7228
33748465.7228
33748465.7228
33748465.7228
33748465.7228
33748465.7228
33748465.7228
33748465.7228
34784619.7841
<34784619.7841 repeats many times>
34784619.7841
34993760.8949
34993760.8949
34993760.8949
34993760.8949
34993760.8949
34993760.8949
34993760.8949
34993760.8949

This seems wrong to me - even as an average, I would not be seeing
identical values over separate 5 minute intervals.

I have tried various changes to see if I receive different information,
and I do not.

Am I missing something blindly obvious here? Shouldn't different 5
minute intervals return different information?

--
Simon Westlake
Time Warner Cable Business Class
Network Engineer
Ph: 414.908.4791 | Cell: 414.688.7956
This E-mail and any of its attachments may contain Time Warner
Cable proprietary information, which is privileged, confidential,
or subject to copyright belonging to Time Warner Cable. This E-mail
is intended solely for the use of the individual or entity to which
it is addressed. If you are not the intended recipient of this
E-mail, you are hereby notified that any dissemination,
distribution, copying, or action taken in relation to the contents
of and attachments to this E-mail is strictly prohibited and may be
unlawful. If you have received this E-mail in error, please notify
the sender immediately and permanently delete the original and any
copy of this E-mail and any printout.

Jo Rhett () Re: Calculating 95th percentile

Reply Threaded

Much simpler to just do this:

CDEF:pctbits=ds1bits,ds0bits,MAX
VDEF:pct=pctbits,95,PERCENT

On Oct 10, 2007, at 9:06 AM, Westlake, Simon wrote:

> Hello all,
>
> I'm attempting to write a script to calculate 95th percentile usage
> using RRD files generated using MRTG. I'm using PHP and PHP4RRDTool in
> order to pull the information and calculate it and I'm running into an
> issue which has maybe been hidden from me in the past as I use MRTG
> as a
> collector for my RRD files.
>
> A quick rundown of my method:
>
> I have a loop that goes through and pulls 5 minute intervals of data
> from the RRD file (by doing --start -10m --end -5m for example, and
> moving backwards in steps of 5) for the last calendar month. It adds
> this data to an array, sorts it from smallest to largest, strips
> the top
> 5% and calculates the average.
>
> The script itself seems to work fine, but the data I get returned
> seems
> weird.
>
> After reading the RRD documentation, it appears that I should be
> able to
> pull the information I want by using the following command:
>
> $opts = array ( "AVERAGE", "--end", "-$timeframe", "--start",
> "-$start");
>
> Where the $opts array is an array of the options to send. This
> works and
> gives me a value, however, the values seem to repeat identically. Here
> is a (somewhat brief) example of the data returned over a series of 5
> minute queries:
>
> 33748465.7228
> 33748465.7228
> 33748465.7228
> 33748465.7228
> 33748465.7228
> 33748465.7228
> 33748465.7228
> 33748465.7228
> 33748465.7228
> 34784619.7841
> <34784619.7841 repeats many times>
> 34784619.7841
> 34993760.8949
> 34993760.8949
> 34993760.8949
> 34993760.8949
> 34993760.8949
> 34993760.8949
> 34993760.8949
> 34993760.8949
>
> This seems wrong to me - even as an average, I would not be seeing
> identical values over separate 5 minute intervals.
>
> I have tried various changes to see if I receive different
> information,
> and I do not.
>
> Am I missing something blindly obvious here? Shouldn't different 5
> minute intervals return different information?
>
> --
> Simon Westlake
> Time Warner Cable Business Class
> Network Engineer
> Ph: 414.908.4791 | Cell: 414.688.7956
> This E-mail and any of its attachments may contain Time Warner
> Cable proprietary information, which is privileged, confidential,
> or subject to copyright belonging to Time Warner Cable. This E-mail
> is intended solely for the use of the individual or entity to which
> it is addressed. If you are not the intended recipient of this
> E-mail, you are hereby notified that any dissemination,
> distribution, copying, or action taken in relation to the contents
> of and attachments to this E-mail is strictly prohibited and may be
> unlawful. If you have received this E-mail in error, please notify
> the sender immediately and permanently delete the original and any
> copy of this E-mail and any printout.
>
> _______________________________________________
> mrtg-developers mailing list
> [hidden email]
> https://lists.oetiker.ch/cgi-bin/listinfo/mrtg-developers

--
Jo Rhett
senior geek

Silicon Valley Colocation
Support Phone: 408-400-0550

Westlake, Simon () Re: Calculating 95th percentile

Reply Threaded

In reply to this post by Westlake, Simon

Would that actually perform the correct function though? Wouldn't it also return the same problematic data?

--
Simon Westlake
Network Engineer
Desk: (414) 908-4791
Cell: (414) 688-7956

-----Original Message-----
From: Jo Rhett [mailto:[hidden email]]
Sent: Wednesday, October 10, 2007 12:35 PM Eastern Standard Time
To: Westlake, Simon
Cc: [hidden email]
Subject: Re: [mrtg-developers] Calculating 95th percentile

Much simpler to just do this:

CDEF:pctbits=ds1bits,ds0bits,MAX
VDEF:pct=pctbits,95,PERCENT

On Oct 10, 2007, at 9:06 AM, Westlake, Simon wrote:

--
Jo Rhett
senior geek

Silicon Valley Colocation
Support Phone: 408-400-0550

This E-mail and any of its attachments may contain Time Warner
Cable proprietary information, which is privileged, confidential,
or subject to copyright belonging to Time Warner Cable. This E-mail
is intended solely for the use of the individual or entity to which
it is addressed. If you are not the intended recipient of this
E-mail, you are hereby notified that any dissemination,
distribution, copying, or action taken in relation to the contents
of and attachments to this E-mail is strictly prohibited and may be
unlawful. If you have received this E-mail in error, please notify
the sender immediately and permanently delete the original and any
copy of this E-mail and any printout.

Jo Rhett () Re: Calculating 95th percentile

Reply Threaded

You have a problem in your script. The 95th percentile function
works perfectly fine. I don't have time to debug your script but I
did have time to give you a better method ;-)

On Oct 10, 2007, at 9:54 AM, Westlake, Simon wrote:

> Would that actually perform the correct function though? Wouldn't
> it also return the same problematic data?
>
> --
> Simon Westlake
> Network Engineer
> Desk: (414) 908-4791
> Cell: (414) 688-7956
>
> -----Original Message-----
> From: Jo Rhett [mailto:[hidden email]]
> Sent: Wednesday, October 10, 2007 12:35 PM Eastern Standard Time
> To: Westlake, Simon
> Cc: [hidden email]
> Subject: Re: [mrtg-developers] Calculating 95th percentile
>
> Much simpler to just do this:
>
> CDEF:pctbits=ds1bits,ds0bits,MAX
> VDEF:pct=pctbits,95,PERCENT
>
> On Oct 10, 2007, at 9:06 AM, Westlake, Simon wrote:
>> Hello all,
>>
>> I'm attempting to write a script to calculate 95th percentile usage
>> using RRD files generated using MRTG. I'm using PHP and
>> PHP4RRDTool in
>> order to pull the information and calculate it and I'm running
>> into an
>> issue which has maybe been hidden from me in the past as I use MRTG
>> as a
>> collector for my RRD files.
>>
>> A quick rundown of my method:
>>
>> I have a loop that goes through and pulls 5 minute intervals of data
>> from the RRD file (by doing --start -10m --end -5m for example, and
>> moving backwards in steps of 5) for the last calendar month. It adds
>> this data to an array, sorts it from smallest to largest, strips
>> the top
>> 5% and calculates the average.
>>
>> The script itself seems to work fine, but the data I get returned
>> seems
>> weird.
>>
>> After reading the RRD documentation, it appears that I should be
>> able to
>> pull the information I want by using the following command:
>>
>> $opts = array ( "AVERAGE", "--end", "-$timeframe", "--start",
>> "-$start");
>>
>> Where the $opts array is an array of the options to send. This
>> works and
>> gives me a value, however, the values seem to repeat identically.
>> Here
>> is a (somewhat brief) example of the data returned over a series of 5
>> minute queries:
>>
>> 33748465.7228
>> 33748465.7228
>> 33748465.7228
>> 33748465.7228
>> 33748465.7228
>> 33748465.7228
>> 33748465.7228
>> 33748465.7228
>> 33748465.7228
>> 34784619.7841
>> <34784619.7841 repeats many times>
>> 34784619.7841
>> 34993760.8949
>> 34993760.8949
>> 34993760.8949
>> 34993760.8949
>> 34993760.8949
>> 34993760.8949
>> 34993760.8949
>> 34993760.8949
>>
>> This seems wrong to me - even as an average, I would not be seeing
>> identical values over separate 5 minute intervals.
>>
>> I have tried various changes to see if I receive different
>> information,
>> and I do not.
>>
>> Am I missing something blindly obvious here? Shouldn't different 5
>> minute intervals return different information?
>>
>> --
>> Simon Westlake
>> Time Warner Cable Business Class
>> Network Engineer
>> Ph: 414.908.4791 | Cell: 414.688.7956
>> This E-mail and any of its attachments may contain Time Warner
>> Cable proprietary information, which is privileged, confidential,
>> or subject to copyright belonging to Time Warner Cable. This E-mail
>> is intended solely for the use of the individual or entity to which
>> it is addressed. If you are not the intended recipient of this
>> E-mail, you are hereby notified that any dissemination,
>> distribution, copying, or action taken in relation to the contents
>> of and attachments to this E-mail is strictly prohibited and may be
>> unlawful. If you have received this E-mail in error, please notify
>> the sender immediately and permanently delete the original and any
>> copy of this E-mail and any printout.
>>
>> _______________________________________________
>> mrtg-developers mailing list
>> [hidden email]
>> https://lists.oetiker.ch/cgi-bin/listinfo/mrtg-developers
>
> --
> Jo Rhett
> senior geek
>
> Silicon Valley Colocation
> Support Phone: 408-400-0550
>
>
> This E-mail and any of its attachments may contain Time Warner
> Cable proprietary information, which is privileged, confidential,
> or subject to copyright belonging to Time Warner Cable. This E-mail
> is intended solely for the use of the individual or entity to which
> it is addressed. If you are not the intended recipient of this
> E-mail, you are hereby notified that any dissemination,
> distribution, copying, or action taken in relation to the contents
> of and attachments to this E-mail is strictly prohibited and may be
> unlawful. If you have received this E-mail in error, please notify
> the sender immediately and permanently delete the original and any
> copy of this E-mail and any printout.

--
Jo Rhett
senior geek

Silicon Valley Colocation
Support Phone: 408-400-0550

Westlake, Simon () Re: Calculating 95th percentile

Reply Threaded

In reply to this post by Westlake, Simon

I do appreciate the input, but the error is not the 95th percentile function. The issue I am having, before the calculation, is the duplicate data return. I am just trying to understand this issue - my belief was that if I pulled info from a.rrd with --start -20m and --end -15m it would be different than -10m and -5m and it is not.

--
Simon Westlake
Network Engineer
Desk: (414) 908-4791
Cell: (414) 688-7956

-----Original Message-----
From: Jo Rhett [mailto:[hidden email]]
Sent: Wednesday, October 10, 2007 01:25 PM Eastern Standard Time
To: Westlake, Simon
Cc: [hidden email]
Subject: Re: [mrtg-developers] Calculating 95th percentile

You have a problem in your script. The 95th percentile function
works perfectly fine. I don't have time to debug your script but I
did have time to give you a better method ;-)

On Oct 10, 2007, at 9:54 AM, Westlake, Simon wrote:

Jo Rhett () Re: Calculating 95th percentile

Reply Threaded

Which is why I recommend you stop playing with your script and just
let it do the calculation for you.

If you insist on fixing your script for no perceptible gain, then
start by doing command-line fetch commands with the same input and
you'll begin to understand where you've gone wrong.

On Oct 10, 2007, at 10:30 AM, Westlake, Simon wrote:

> I do appreciate the input, but the error is not the 95th percentile
> function. The issue I am having, before the calculation, is the
> duplicate data return. I am just trying to understand this issue -
> my belief was that if I pulled info from a.rrd with --start -20m
> and --end -15m it would be different than -10m and -5m and it is not.
>
> --
> Simon Westlake
> Network Engineer
> Desk: (414) 908-4791
> Cell: (414) 688-7956
>
> -----Original Message-----
> From: Jo Rhett [mailto:[hidden email]]
> Sent: Wednesday, October 10, 2007 01:25 PM Eastern Standard Time
> To: Westlake, Simon
> Cc: [hidden email]
> Subject: Re: [mrtg-developers] Calculating 95th percentile
>
> You have a problem in your script. The 95th percentile function
> works perfectly fine. I don't have time to debug your script but I
> did have time to give you a better method ;-)
>
> On Oct 10, 2007, at 9:54 AM, Westlake, Simon wrote:
>
>> Would that actually perform the correct function though? Wouldn't
>> it also return the same problematic data?
>>
>> --
>> Simon Westlake
>> Network Engineer
>> Desk: (414) 908-4791
>> Cell: (414) 688-7956
>>
>> -----Original Message-----
>> From: Jo Rhett [mailto:[hidden email]]
>> Sent: Wednesday, October 10, 2007 12:35 PM Eastern Standard Time
>> To: Westlake, Simon
>> Cc: [hidden email]
>> Subject: Re: [mrtg-developers] Calculating 95th percentile
>>
>> Much simpler to just do this:
>>
>> CDEF:pctbits=ds1bits,ds0bits,MAX
>> VDEF:pct=pctbits,95,PERCENT
>>
>> On Oct 10, 2007, at 9:06 AM, Westlake, Simon wrote:
>>> Hello all,
>>>
>>> I'm attempting to write a script to calculate 95th percentile usage
>>> using RRD files generated using MRTG. I'm using PHP and
>>> PHP4RRDTool in
>>> order to pull the information and calculate it and I'm running
>>> into an
>>> issue which has maybe been hidden from me in the past as I use MRTG
>>> as a
>>> collector for my RRD files.
>>>
>>> A quick rundown of my method:
>>>
>>> I have a loop that goes through and pulls 5 minute intervals of data
>>> from the RRD file (by doing --start -10m --end -5m for example, and
>>> moving backwards in steps of 5) for the last calendar month. It adds
>>> this data to an array, sorts it from smallest to largest, strips
>>> the top
>>> 5% and calculates the average.
>>>
>>> The script itself seems to work fine, but the data I get returned
>>> seems
>>> weird.
>>>
>>> After reading the RRD documentation, it appears that I should be
>>> able to
>>> pull the information I want by using the following command:
>>>
>>> $opts = array ( "AVERAGE", "--end", "-$timeframe", "--start",
>>> "-$start");
>>>
>>> Where the $opts array is an array of the options to send. This
>>> works and
>>> gives me a value, however, the values seem to repeat identically.
>>> Here
>>> is a (somewhat brief) example of the data returned over a series
>>> of 5
>>> minute queries:
>>>
>>> 33748465.7228
>>> 33748465.7228
>>> 33748465.7228
>>> 33748465.7228
>>> 33748465.7228
>>> 33748465.7228
>>> 33748465.7228
>>> 33748465.7228
>>> 33748465.7228
>>> 34784619.7841
>>> <34784619.7841 repeats many times>
>>> 34784619.7841
>>> 34993760.8949
>>> 34993760.8949
>>> 34993760.8949
>>> 34993760.8949
>>> 34993760.8949
>>> 34993760.8949
>>> 34993760.8949
>>> 34993760.8949
>>>
>>> This seems wrong to me - even as an average, I would not be seeing
>>> identical values over separate 5 minute intervals.
>>>
>>> I have tried various changes to see if I receive different
>>> information,
>>> and I do not.
>>>
>>> Am I missing something blindly obvious here? Shouldn't different 5
>>> minute intervals return different information?
>>>
>>> --
>>> Simon Westlake
>>> Time Warner Cable Business Class
>>> Network Engineer
>>> Ph: 414.908.4791 | Cell: 414.688.7956
>>> This E-mail and any of its attachments may contain Time Warner
>>> Cable proprietary information, which is privileged, confidential,
>>> or subject to copyright belonging to Time Warner Cable. This E-mail
>>> is intended solely for the use of the individual or entity to which
>>> it is addressed. If you are not the intended recipient of this
>>> E-mail, you are hereby notified that any dissemination,
>>> distribution, copying, or action taken in relation to the contents
>>> of and attachments to this E-mail is strictly prohibited and may be
>>> unlawful. If you have received this E-mail in error, please notify
>>> the sender immediately and permanently delete the original and any
>>> copy of this E-mail and any printout.
>>>
>>> _______________________________________________
>>> mrtg-developers mailing list
>>> [hidden email]
>>> https://lists.oetiker.ch/cgi-bin/listinfo/mrtg-developers
>>
>> --
>> Jo Rhett
>> senior geek
>>
>> Silicon Valley Colocation
>> Support Phone: 408-400-0550
>>
>>
>> This E-mail and any of its attachments may contain Time Warner
>> Cable proprietary information, which is privileged, confidential,
>> or subject to copyright belonging to Time Warner Cable. This E-mail
>> is intended solely for the use of the individual or entity to which
>> it is addressed. If you are not the intended recipient of this
>> E-mail, you are hereby notified that any dissemination,
>> distribution, copying, or action taken in relation to the contents
>> of and attachments to this E-mail is strictly prohibited and may be
>> unlawful. If you have received this E-mail in error, please notify
>> the sender immediately and permanently delete the original and any
>> copy of this E-mail and any printout.
>
> --
> Jo Rhett
> senior geek
>
> Silicon Valley Colocation
> Support Phone: 408-400-0550
>
>
> This E-mail and any of its attachments may contain Time Warner
> Cable proprietary information, which is privileged, confidential,
> or subject to copyright belonging to Time Warner Cable. This E-mail
> is intended solely for the use of the individual or entity to which
> it is addressed. If you are not the intended recipient of this
> E-mail, you are hereby notified that any dissemination,
> distribution, copying, or action taken in relation to the contents
> of and attachments to this E-mail is strictly prohibited and may be
> unlawful. If you have received this E-mail in error, please notify
> the sender immediately and permanently delete the original and any
> copy of this E-mail and any printout.

--
Jo Rhett
senior geek

Silicon Valley Colocation
Support Phone: 408-400-0550

Westlake, Simon () Re: Calculating 95th percentile

Reply Threaded

In reply to this post by Westlake, Simon

I have implemented the function, thanks. In my lack of RRD knowledge, I
looked for a way to do it in a manner I was accustomed to, not really
realizing RRDTool had the power to do it for me.

My only desire to fix my script was to gain a better understanding of
the cause of the problem so that I wouldn't come back and post another
(obviously) stupid question due to lack of understanding :)

--
Simon Westlake
Time Warner Cable Business Class
Network Engineer
Ph: 414.908.4791 | Cell: 414.688.7956
-----Original Message-----
From: Jo Rhett [mailto:[hidden email]]
Sent: Wednesday, October 10, 2007 2:17 PM
To: Westlake, Simon
Cc: [hidden email]
Subject: Re: [mrtg-developers] Calculating 95th percentile

Which is why I recommend you stop playing with your script and just
let it do the calculation for you.

If you insist on fixing your script for no perceptible gain, then
start by doing command-line fetch commands with the same input and
you'll begin to understand where you've gone wrong.

On Oct 10, 2007, at 10:30 AM, Westlake, Simon wrote:

Time

>> To: Westlake, Simon
>> Cc: [hidden email]
>> Subject: Re: [mrtg-developers] Calculating 95th percentile
>>
>> Much simpler to just do this:
>>
>> CDEF:pctbits=ds1bits,ds0bits,MAX
>> VDEF:pct=pctbits,95,PERCENT
>>
>> On Oct 10, 2007, at 9:06 AM, Westlake, Simon wrote:
>>> Hello all,
>>>
>>> I'm attempting to write a script to calculate 95th percentile usage
>>> using RRD files generated using MRTG. I'm using PHP and
>>> PHP4RRDTool in
>>> order to pull the information and calculate it and I'm running
>>> into an
>>> issue which has maybe been hidden from me in the past as I use MRTG
>>> as a
>>> collector for my RRD files.
>>>
>>> A quick rundown of my method:
>>>
>>> I have a loop that goes through and pulls 5 minute intervals of data
>>> from the RRD file (by doing --start -10m --end -5m for example, and
>>> moving backwards in steps of 5) for the last calendar month. It adds
>>> this data to an array, sorts it from smallest to largest, strips
>>> the top
>>> 5% and calculates the average.
>>>
>>> The script itself seems to work fine, but the data I get returned
>>> seems
>>> weird.
>>>
>>> After reading the RRD documentation, it appears that I should be
>>> able to
>>> pull the information I want by using the following command:
>>>
>>> $opts = array ( "AVERAGE", "--end", "-$timeframe", "--start",
>>> "-$start");
>>>
>>> Where the $opts array is an array of the options to send. This
>>> works and
>>> gives me a value, however, the values seem to repeat identically.
>>> Here
>>> is a (somewhat brief) example of the data returned over a series
>>> of 5
>>> minute queries:
>>>
>>> 33748465.7228
>>> 33748465.7228
>>> 33748465.7228
>>> 33748465.7228
>>> 33748465.7228
>>> 33748465.7228
>>> 33748465.7228
>>> 33748465.7228
>>> 33748465.7228
>>> 34784619.7841
>>> <34784619.7841 repeats many times>
>>> 34784619.7841
>>> 34993760.8949
>>> 34993760.8949
>>> 34993760.8949
>>> 34993760.8949
>>> 34993760.8949
>>> 34993760.8949
>>> 34993760.8949
>>> 34993760.8949
>>>
>>> This seems wrong to me - even as an average, I would not be seeing
>>> identical values over separate 5 minute intervals.
>>>
>>> I have tried various changes to see if I receive different
>>> information,
>>> and I do not.
>>>
>>> Am I missing something blindly obvious here? Shouldn't different 5
>>> minute intervals return different information?
>>>
>>> --
>>> Simon Westlake
>>> Time Warner Cable Business Class
>>> Network Engineer
>>> Ph: 414.908.4791 | Cell: 414.688.7956
>>> This E-mail and any of its attachments may contain Time Warner
>>> Cable proprietary information, which is privileged, confidential,
>>> or subject to copyright belonging to Time Warner Cable. This E-mail
>>> is intended solely for the use of the individual or entity to which
>>> it is addressed. If you are not the intended recipient of this
>>> E-mail, you are hereby notified that any dissemination,
>>> distribution, copying, or action taken in relation to the contents
>>> of and attachments to this E-mail is strictly prohibited and may be
>>> unlawful. If you have received this E-mail in error, please notify
>>> the sender immediately and permanently delete the original and any
>>> copy of this E-mail and any printout.
>>>
>>> _______________________________________________
>>> mrtg-developers mailing list
>>> [hidden email]
>>> https://lists.oetiker.ch/cgi-bin/listinfo/mrtg-developers
>>
>> --
>> Jo Rhett
>> senior geek
>>
>> Silicon Valley Colocation
>> Support Phone: 408-400-0550
>>
>>
>> This E-mail and any of its attachments may contain Time Warner
>> Cable proprietary information, which is privileged, confidential,
>> or subject to copyright belonging to Time Warner Cable. This E-mail
>> is intended solely for the use of the individual or entity to which
>> it is addressed. If you are not the intended recipient of this
>> E-mail, you are hereby notified that any dissemination,
>> distribution, copying, or action taken in relation to the contents
>> of and attachments to this E-mail is strictly prohibited and may be
>> unlawful. If you have received this E-mail in error, please notify
>> the sender immediately and permanently delete the original and any
>> copy of this E-mail and any printout.
>
> --
> Jo Rhett
> senior geek
>
> Silicon Valley Colocation
> Support Phone: 408-400-0550
>
>
> This E-mail and any of its attachments may contain Time Warner
> Cable proprietary information, which is privileged, confidential,
> or subject to copyright belonging to Time Warner Cable. This E-mail
> is intended solely for the use of the individual or entity to which
> it is addressed. If you are not the intended recipient of this
> E-mail, you are hereby notified that any dissemination,
> distribution, copying, or action taken in relation to the contents
> of and attachments to this E-mail is strictly prohibited and may be
> unlawful. If you have received this E-mail in error, please notify
> the sender immediately and permanently delete the original and any
> copy of this E-mail and any printout.

Jo Rhett () Re: Calculating 95th percentile

Reply Threaded

On Oct 10, 2007, at 12:59 PM, Westlake, Simon wrote:
> My only desire to fix my script was to gain a better understanding of
> the cause of the problem so that I wouldn't come back and post another
> (obviously) stupid question due to lack of understanding :)

The way you'll learn is by doing what I said for debug:

> start by doing command-line fetch commands with the same input and

Read the comments on the rrdfetch manpage carefully, they are concise
but very true. fetch rarely returns what you expect until you learn
how to expect what fetch will provide ;-)

I always end up doing command-line queries to confirm that I will get
what I want.

--
Jo Rhett
senior geek

Silicon Valley Colocation
Support Phone: 408-400-0550

Westlake, Simon () Re: Calculating 95th percentile

Reply Threaded

In reply to this post by Westlake, Simon

So, I've been playing with CDEFS and VDEFS after reading this email and
trying to find a way to do exactly what I need.

As a short introduction, the code you provided works great for drawing a
line at 95% of the max value. If I can find a way to do what I need this
simply, it will make my life a lot easier.

However, what I need to calculate is 95th percentile in billing terms
(see http://en.wikipedia.org/wiki/Burstable_billing). The problem with
the vdef 95 percent method is that if there is any time a user bursts to
100 megs, it will calculate the 95th percentile as 95 megs. The
universally accepted method (from a billing perspective) of calculating
95th percentile is to take all your samples (typically 1 every 5
minutes), sort them from smallest to largest, discard the top 5% and
then bill at the largest value. By this method, if a user had burst to
100 megs 4.9% of the month and had maxed at 12 megs the rest of the
month, the 95th percentile would be calculated as 12 megs. Quite a big
difference..

This was why I had begun by trying to pull a 5 minute sample for every 5
minutes in the billing period and sorting them in an array and
discarding the top 5%.

Before I go too far in figuring out the problem with my fetch statement,
is there some reasonable way of doing this simply with RRDTool that I
should investigate, or for a calculation of this complexity, will I need
to do the calculation via some other means? Looking at the RRDTool
documentation, I don't really see an easy way to do this.

--
Simon Westlake
Time Warner Cable Business Class
Network Engineer
Ph: 414.908.4791 | Cell: 414.688.7956
-----Original Message-----
From: Jo Rhett [mailto:[hidden email]]
Sent: Wednesday, October 10, 2007 2:17 PM
To: Westlake, Simon
Cc: [hidden email]
Subject: Re: [mrtg-developers] Calculating 95th percentile

Which is why I recommend you stop playing with your script and just
let it do the calculation for you.

If you insist on fixing your script for no perceptible gain, then
start by doing command-line fetch commands with the same input and
you'll begin to understand where you've gone wrong.

On Oct 10, 2007, at 10:30 AM, Westlake, Simon wrote:

Time

Alex van den Bogaerdt-2 () Re: Calculating 95th percentile

Reply Threaded

On Thu, Oct 11, 2007 at 11:59:48AM -0400, Westlake, Simon wrote:

> As a short introduction, the code you provided works great for drawing a
> line at 95% of the max value. If I can find a way to do what I need this
> simply, it will make my life a lot easier.

It does not, at least not when I wrote that code.

> However, what I need to calculate is 95th percentile in billing terms
> (see http://en.wikipedia.org/wiki/Burstable_billing). The problem with
> the vdef 95 percent method is that if there is any time a user bursts to
> 100 megs, it will calculate the 95th percentile as 95 megs. The
> universally accepted method (from a billing perspective) of calculating
> 95th percentile is to take all your samples (typically 1 every 5
> minutes), sort them from smallest to largest, discard the top 5% and
> then bill at the largest value. By this method, if a user had burst to
> 100 megs 4.9% of the month and had maxed at 12 megs the rest of the
> month, the 95th percentile would be calculated as 12 megs. Quite a big
> difference..

Using somewhat old code, which may have changed since then:

- sort: NaN < -INF < finite_values < INF

This gives you sorted rates:
qsort(array,step,sizeof(double),vdef_percent_compar);

This computes the Xth percentile position:
field = (steps-1)*dst->vf.param/100;
This takes the rate from that position:
dst->vf.val = array[field];

I don't know why you think the percentile function is just shorthand
for "*95/100". It really is not.

> Before I go too far in figuring out the problem with my fetch statement,
> is there some reasonable way of doing this simply with RRDTool that I
> should investigate, or for a calculation of this complexity, will I need
> to do the calculation via some other means? Looking at the RRDTool
> documentation, I don't really see an easy way to do this.

Investigate what happens in the scenario you talk about. Update with
all zeros, except one update which should be 100M. Then look what value
is reported back.

Alex
P.S. What is this doing on mrtg-developers, while it should clearly
be on rrdtool-users ?

Westlake, Simon () Re: Calculating 95th percentile

Reply Threaded

In reply to this post by Westlake, Simon

Thanks for the info Alex, I will give it a shot. Sorry for posting to
the wrong list!

--
Simon Westlake
Time Warner Cable Business Class
Network Engineer
Ph: 414.908.4791 | Cell: 414.688.7956
-----Original Message-----
From: [hidden email]
[mailto:[hidden email]] On Behalf Of Alex van
den Bogaerdt
Sent: Thursday, October 11, 2007 11:15 AM
To: [hidden email]
Subject: Re: [mrtg-developers] Calculating 95th percentile

On Thu, Oct 11, 2007 at 11:59:48AM -0400, Westlake, Simon wrote:

> As a short introduction, the code you provided works great for drawing
a
> line at 95% of the max value. If I can find a way to do what I need
this
> simply, it will make my life a lot easier.

It does not, at least not when I wrote that code.

> However, what I need to calculate is 95th percentile in billing terms
> (see http://en.wikipedia.org/wiki/Burstable_billing). The problem with
> the vdef 95 percent method is that if there is any time a user bursts
to
> 100 megs, it will calculate the 95th percentile as 95 megs. The
> universally accepted method (from a billing perspective) of
calculating
> 95th percentile is to take all your samples (typically 1 every 5
> minutes), sort them from smallest to largest, discard the top 5% and
> then bill at the largest value. By this method, if a user had burst to
> 100 megs 4.9% of the month and had maxed at 12 megs the rest of the
> month, the 95th percentile would be calculated as 12 megs. Quite a big
> difference..

Using somewhat old code, which may have changed since then:

- sort: NaN < -INF < finite_values < INF

This gives you sorted rates:
qsort(array,step,sizeof(double),vdef_percent_compar);

This computes the Xth percentile position:
field = (steps-1)*dst->vf.param/100;
This takes the rate from that position:
dst->vf.val = array[field];

I don't know why you think the percentile function is just shorthand
for "*95/100". It really is not.

> Before I go too far in figuring out the problem with my fetch
statement,
> is there some reasonable way of doing this simply with RRDTool that I
> should investigate, or for a calculation of this complexity, will I
need
> to do the calculation via some other means? Looking at the RRDTool
> documentation, I don't really see an easy way to do this.

Investigate what happens in the scenario you talk about. Update with
all zeros, except one update which should be 100M. Then look what value
is reported back.

Alex
P.S. What is this doing on mrtg-developers, while it should clearly
be on rrdtool-users ?

_______________________________________________
mrtg-developers mailing list
[hidden email]
https://lists.oetiker.ch/cgi-bin/listinfo/mrtg-developers
This E-mail and any of its attachments may contain Time Warner
Cable proprietary information, which is privileged, confidential,
or subject to copyright belonging to Time Warner Cable. This E-mail
is intended solely for the use of the individual or entity to which
it is addressed. If you are not the intended recipient of this
E-mail, you are hereby notified that any dissemination,
distribution, copying, or action taken in relation to the contents
of and attachments to this E-mail is strictly prohibited and may be
unlawful. If you have received this E-mail in error, please notify
the sender immediately and permanently delete the original and any
copy of this E-mail and any printout.