Opennms with RAMSan = iowait?

23 messages Options
Embed this post
Permalink
1 2
Rupert Ogilvie

Opennms with RAMSan = iowait?

Reply Threaded More More options
Print post
Permalink
Some javascript/style in this post has been disabled (why?)

Hi all,

I’m profiling a ramsan-20 drive at the moment to see how it improves on ONMS performance. I’ve installed opennms and the postgresql database onto the ramsan and tried to make sure that there aren’t any symlinks pointing back to the main drive. I’ve got a reasonable deployment (10k nodes). I’ve increased the JAVA_HEAP_SIZE to 3GB. The server its running on has 32GB of memory installed, and a centos 5 basic install. What I’m finding strange is that when ONMS is running it steadily consumes all the memory and additionally I’m getting iowait which seems very unlikely with a RAMSan drive – I’ve check the performance with iostat and it shows that the iops and MB/s are significantly below what the drive should be capable of.

Has anyone else any experience with a deployment like this and any suggestions about how to control the memory usage or a bottleneck which could be causing the iowait?

Many thanks for your time,

Rupert

-------------------------------------------------------- Intergence Systems is a limited company registered in England and Wales. Registered number: 04667187. Registered office: Quern House, Mill Court, Great Shelford, CB22 5LD. The content of this message and any attached file are confidential and/or privileged and are intended for the recipient only. If you are not the intended recipient, any unauthorised review, use, re-transmission, dissemination, copying, disclosure or other use of, or taking of any action in reliance of this information is strictly prohibited. If you receive this message in error please contact the sender immediately and then delete this email from your system. Copyright in this email and attachments created by us belongs to Intergence Systems Ltd. Any attachment with this message should be checked for viruses before it is opened. Intergence Systems Ltd cannot be held responsible for any failure by the recipient to test for viruses before opening any attachments. Should you communicate with anyone at Intergence Systems Ltd by email you consent to us monitoring and reading any such correspondence. --------------------------------------------------------
------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss
Duncan Hill

Re: Opennms with RAMSan = iowait?

Reply Threaded More More options
Print post
Permalink
Quoting Rupert Ogilvie <[hidden email]>:

> server its running on has 32GB of memory installed, and a centos 5  
> basic install. What I'm finding strange is that when ONMS is running  
> it steadily consumes all the memory and additionally I'm getting  
> iowait which seems very unlikely with a RAMSan drive

Could you clarify what you mean by 'OpenNMS consumes all the memory'?

- All the allocate heap?
- All of the OS memory?


------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss
jcat

Re: Opennms with RAMSan = iowait?

Reply Threaded More More options
Print post
Permalink
Some javascript/style in this post has been disabled (why?)
Yes, i fit's heap then you've got an unusual issue.
If it's just system memory, then it's normal Linux behaviour (it's just fs cache).


Cheers,
Just


On Thu, 2009-11-05 at 14:25 +0000, Duncan Hill wrote:
Quoting Rupert Ogilvie <[hidden email]>:

> server its running on has 32GB of memory installed, and a centos 5  
> basic install. What I'm finding strange is that when ONMS is running  
> it steadily consumes all the memory and additionally I'm getting  
> iowait which seems very unlikely with a RAMSan drive

Could you clarify what you mean by 'OpenNMS consumes all the memory'?

- All the allocate heap?
- All of the OS memory?


------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day 
trial. Simplify your report design, integration and deployment - and focus on 
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss


--
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.
------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss
Rupert Ogilvie

Re: Opennms with RAMSan = iowait?

Reply Threaded More More options
Print post
Permalink
Some javascript/style in this post has been disabled (why?)

Hi

Thanks for your replies. Apologies for the vagueness - it’s whichever memory is shown by the top command, I think that’s OS memory?

The IOWait seems to be due to thread limitations, we’ve got 8 cores but only 2 are ever being used. Is there a way of enabling multithreading? I’ve turned on the multithreading garbage collection and followed the other advice in the performance tuning section of the wiki but it’s not had much effect overall.

Cheers

Rupert

 

From: jcat [mailto:[hidden email]]
Sent: 05 November 2009 22:51
To: General OpenNMS Discussion
Subject: Re: [opennms-discuss] Opennms with RAMSan = iowait?

 

Yes, i fit's heap then you've got an unusual issue.
If it's just system memory, then it's normal Linux behaviour (it's just fs cache).


Cheers,
Just


On Thu, 2009-11-05 at 14:25 +0000, Duncan Hill wrote:

 
Quoting Rupert Ogilvie <[hidden email]>:
 
> server its running on has 32GB of memory installed, and a centos 5  
> basic install. What I'm finding strange is that when ONMS is running  
> it steadily consumes all the memory and additionally I'm getting  
> iowait which seems very unlikely with a RAMSan drive
 
Could you clarify what you mean by 'OpenNMS consumes all the memory'?
 
- All the allocate heap?
- All of the OS memory?
 
 
------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day 
trial. Simplify your report design, integration and deployment - and focus on 
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ
 
opennms-discuss mailing list
 
To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss
 


--
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.

-------------------------------------------------------- Intergence Systems is a limited company registered in England and Wales. Registered number: 04667187. Registered office: Quern House, Mill Court, Great Shelford, CB22 5LD. The content of this message and any attached file are confidential and/or privileged and are intended for the recipient only. If you are not the intended recipient, any unauthorised review, use, re-transmission, dissemination, copying, disclosure or other use of, or taking of any action in reliance of this information is strictly prohibited. If you receive this message in error please contact the sender immediately and then delete this email from your system. Copyright in this email and attachments created by us belongs to Intergence Systems Ltd. Any attachment with this message should be checked for viruses before it is opened. Intergence Systems Ltd cannot be held responsible for any failure by the recipient to test for viruses before opening any attachments. Should you communicate with anyone at Intergence Systems Ltd by email you consent to us monitoring and reading any such correspondence. --------------------------------------------------------
------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss
Duncan Hill

Re: Opennms with RAMSan = iowait?

Reply Threaded More More options
Print post
Permalink
Quoting Rupert Ogilvie <[hidden email]>:

> Hi
> Thanks for your replies. Apologies for the vagueness - it?s  
> whichever memory is shown by the top command, I think that?s OS  
> memory?

Used is a combination of actively used application memory, buffers and cache.

Linux will throw all of your RAM at caching disk activity until that  
RAM is needed for applications.  Perfectly normal beast.

> The IOWait seems to be due to thread limitations, we?ve got 8 cores  
> but only 2 are ever being used. Is there a way of enabling  
> multithreading? I?ve turned on the multithreading garbage collection

Odd.  Unfortunately, I don't have an 8-core machine to run our ONMS  
install (it runs on a DL380 G4) to see if I can replicate that.

I will say that we used to see hardly any I/O after we put in a BBWC  
on the RAID card.  I just looked at we're back to 90% wait on one CPU  
when we get a mass RRD/SQL update going.  Need more disks in that  
server.


------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss
Rupert Ogilvie

Re: Opennms with RAMSan = iowait?

Reply Threaded More More options
Print post
Permalink
Hi Duncan,
I've tried running the databases on SAS RAID with BBWC and on the PCI-E RAMSan drive, but the IOWait when on the RAID is even worse than with the RAMSan! I don't think the bottleneck is with the write to drive as the RAMSan can handle ~700MBs throughput and the iostat results show its not getting close to this (I think it intermittently peaks at around 200MBs). Any suggestions you've got would be great,
Cheers
Rupert

-----Original Message-----
From: Duncan Hill [mailto:[hidden email]]
Sent: 06 November 2009 09:34
To: [hidden email]
Subject: Re: [opennms-discuss] Opennms with RAMSan = iowait?

Quoting Rupert Ogilvie <[hidden email]>:

> Hi
> Thanks for your replies. Apologies for the vagueness - it?s  
> whichever memory is shown by the top command, I think that?s OS  
> memory?

Used is a combination of actively used application memory, buffers and cache.

Linux will throw all of your RAM at caching disk activity until that  
RAM is needed for applications.  Perfectly normal beast.

> The IOWait seems to be due to thread limitations, we?ve got 8 cores  
> but only 2 are ever being used. Is there a way of enabling  
> multithreading? I?ve turned on the multithreading garbage collection

Odd.  Unfortunately, I don't have an 8-core machine to run our ONMS  
install (it runs on a DL380 G4) to see if I can replicate that.

I will say that we used to see hardly any I/O after we put in a BBWC  
on the RAID card.  I just looked at we're back to 90% wait on one CPU  
when we get a mass RRD/SQL update going.  Need more disks in that  
server.


------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss
--------------------------------------------------------
Intergence Systems is a limited company registered in England and Wales. Registered number: 04667187. Registered office: Quern House, Mill Court, Great Shelford, CB22 5LD.

The content of this message and any attached file are confidential and/or privileged and are intended for the recipient only. If you are not the intended recipient, any unauthorised review, use, re-transmission, dissemination, copying, disclosure or other use of, or taking of any action in reliance of this information is strictly prohibited. If you receive this message in error please contact the sender immediately and then delete this email from your system. Copyright in this email and attachments created by us belongs to Intergence Systems Ltd. Any attachment with this message should be checked for viruses before it is opened. Intergence Systems Ltd cannot be held responsible for any failure by the recipient to test for viruses before opening any attachments. Should you communicate with anyone at Intergence Systems Ltd by email you consent to us monitoring and reading any such correspondence.
--------------------------------------------------------


------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss
Michael Danko

Re: Opennms with RAMSan = iowait?

Reply Threaded More More options
Print post
Permalink
Throughput is one thing, seeks are another. The sps and mps stats in iostat can give you an idea if that's what you're waiting on. If this is abnormally high, even caching writes won't help much since the BBWC will just delay a bunch of seeks. What type of filesystem are you using?

I've always had a heck of a time troubleshooting these issues on linux, tools in the dtrace toolkit on solaris make troubleshooting issues like this dead simple. Perhaps an admin with more linux experience can chime in with how to figure out which process is tying up the most disk IO. After I've got a better idea on what's tying up the IO, knowing where to make tweaks is easier.  
With most of the complaints on the list regarding performance always coming back to IO, I think I'll keep the better way to skin the RRD cat in the back of my mind today.





Sent from my Verizon Wireless BlackBerry

-----Original Message-----
From: Rupert Ogilvie <[hidden email]>
Date: Fri, 6 Nov 2009 04:11:09
To: General OpenNMS Discussion<[hidden email]>
Subject: Re: [opennms-discuss] Opennms with RAMSan = iowait?

Hi Duncan,
I've tried running the databases on SAS RAID with BBWC and on the PCI-E RAMSan drive, but the IOWait when on the RAID is even worse than with the RAMSan! I don't think the bottleneck is with the write to drive as the RAMSan can handle ~700MBs throughput and the iostat results show its not getting close to this (I think it intermittently peaks at around 200MBs). Any suggestions you've got would be great,
Cheers
Rupert

-----Original Message-----
From: Duncan Hill [mailto:[hidden email]]
Sent: 06 November 2009 09:34
To: [hidden email]
Subject: Re: [opennms-discuss] Opennms with RAMSan = iowait?

Quoting Rupert Ogilvie <[hidden email]>:

> Hi
> Thanks for your replies. Apologies for the vagueness - it?s  
> whichever memory is shown by the top command, I think that?s OS  
> memory?

Used is a combination of actively used application memory, buffers and cache.

Linux will throw all of your RAM at caching disk activity until that  
RAM is needed for applications.  Perfectly normal beast.

> The IOWait seems to be due to thread limitations, we?ve got 8 cores  
> but only 2 are ever being used. Is there a way of enabling  
> multithreading? I?ve turned on the multithreading garbage collection

Odd.  Unfortunately, I don't have an 8-core machine to run our ONMS  
install (it runs on a DL380 G4) to see if I can replicate that.

I will say that we used to see hardly any I/O after we put in a BBWC  
on the RAID card.  I just looked at we're back to 90% wait on one CPU  
when we get a mass RRD/SQL update going.  Need more disks in that  
server.


------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss
--------------------------------------------------------
Intergence Systems is a limited company registered in England and Wales. Registered number: 04667187. Registered office: Quern House, Mill Court, Great Shelford, CB22 5LD.

The content of this message and any attached file are confidential and/or privileged and are intended for the recipient only. If you are not the intended recipient, any unauthorised review, use, re-transmission, dissemination, copying, disclosure or other use of, or taking of any action in reliance of this information is strictly prohibited. If you receive this message in error please contact the sender immediately and then delete this email from your system. Copyright in this email and attachments created by us belongs to Intergence Systems Ltd. Any attachment with this message should be checked for viruses before it is opened. Intergence Systems Ltd cannot be held responsible for any failure by the recipient to test for viruses before opening any attachments. Should you communicate with anyone at Intergence Systems Ltd by email you consent to us monitoring and reading any such correspondence.
--------------------------------------------------------


------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss
------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss
Duncan Hill

Re: Opennms with RAMSan = iowait?

Reply Threaded More More options
Print post
Permalink
Quoting [hidden email]:

> With most of the complaints on the list regarding performance always  
> coming back to IO, I think I'll keep the better way to skin the RRD  
> cat in the back of my mind today.

If I could fit our IOdrive in the G4, I'd give that a spin to see what  
effect it has on the RRDs.  Unfortunately, £2500 for 80 GB of storage  
is not what most people want to spend.


------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss
Michael Danko

Re: Opennms with RAMSan = iowait?

Reply Threaded More More options
Print post
Permalink
Yes, but my point is that even if you have 50K RRD files, and jrobin is
capable of 2500 RRD updates per second on relatively low end equipment,
that something doesn't add up. Even multiplying the benchmark figure by
5 means it should only take a minute to update that much data.

There is a lot of file system mojo working against linux deployments.
See: http://www.opennms.org/wiki/RRD_performance_fundamentals 

To the 4th point under maximizing RRD storage performance, extN
traditionally has had a 4kb block size as limited by page size. The
rabbit hole goes pretty deep here, and I think I'll avoid going to far
with that -- it could be argued all day. Also try reading
http://oss.oetiker.ch/rrdtool-trac/wiki/TuningRRD which has some great
explinations as to what's going on with file systems and RRD storage,
and some tricks on how to get things done more efficiently with linux
storage. Some of the notes on this page don't apply to Jrobin though.

In particular, the block device level optimization may help you out.
It's why I asked what your seek times look like.

- Mike






On Fri, 2009-11-06 at 14:04 +0000, Duncan Hill wrote:

> Quoting [hidden email]:
>
> > With most of the complaints on the list regarding performance always  
> > coming back to IO, I think I'll keep the better way to skin the RRD  
> > cat in the back of my mind today.
>
> If I could fit our IOdrive in the G4, I'd give that a spin to see what  
> effect it has on the RRDs.  Unfortunately, £2500 for 80 GB of storage  
> is not what most people want to spend.
>
>
> ------------------------------------------------------------------------------
> Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
> trial. Simplify your report design, integration and deployment - and focus on
> what you do best, core application coding. Discover what's new with
> Crystal Reports now.  http://p.sf.net/sfu/bobj-july
> _______________________________________________
> Please read the OpenNMS Mailing List FAQ:
> http://www.opennms.org/index.php/Mailing_List_FAQ
>
> opennms-discuss mailing list
>
> To *unsubscribe* or change your subscription options, see the bottom of this page:
> https://lists.sourceforge.net/lists/listinfo/opennms-discuss



------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss
Rupert Ogilvie

Re: Opennms with RAMSan = iowait?

Reply Threaded More More options
Print post
Permalink
Thanks for the responses. If you are using SSD surely seek times stop having any effect as that’s one of main benefits of the technology? File system-wise I've been using XFS.


-----Original Message-----
From: Mike Danko [mailto:[hidden email]]
Sent: 06 November 2009 15:15
To: General OpenNMS Discussion
Subject: Re: [opennms-discuss] Opennms with RAMSan = iowait?

Yes, but my point is that even if you have 50K RRD files, and jrobin is
capable of 2500 RRD updates per second on relatively low end equipment,
that something doesn't add up. Even multiplying the benchmark figure by
5 means it should only take a minute to update that much data.

There is a lot of file system mojo working against linux deployments.
See: http://www.opennms.org/wiki/RRD_performance_fundamentals 

To the 4th point under maximizing RRD storage performance, extN
traditionally has had a 4kb block size as limited by page size. The
rabbit hole goes pretty deep here, and I think I'll avoid going to far
with that -- it could be argued all day. Also try reading
http://oss.oetiker.ch/rrdtool-trac/wiki/TuningRRD which has some great
explinations as to what's going on with file systems and RRD storage,
and some tricks on how to get things done more efficiently with linux
storage. Some of the notes on this page don't apply to Jrobin though.

In particular, the block device level optimization may help you out.
It's why I asked what your seek times look like.

- Mike






On Fri, 2009-11-06 at 14:04 +0000, Duncan Hill wrote:

> Quoting [hidden email]:
>
> > With most of the complaints on the list regarding performance always  
> > coming back to IO, I think I'll keep the better way to skin the RRD  
> > cat in the back of my mind today.
>
> If I could fit our IOdrive in the G4, I'd give that a spin to see what  
> effect it has on the RRDs.  Unfortunately, £2500 for 80 GB of storage  
> is not what most people want to spend.
>
>
> ------------------------------------------------------------------------------
> Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
> trial. Simplify your report design, integration and deployment - and focus on
> what you do best, core application coding. Discover what's new with
> Crystal Reports now.  http://p.sf.net/sfu/bobj-july
> _______________________________________________
> Please read the OpenNMS Mailing List FAQ:
> http://www.opennms.org/index.php/Mailing_List_FAQ
>
> opennms-discuss mailing list
>
> To *unsubscribe* or change your subscription options, see the bottom of this page:
> https://lists.sourceforge.net/lists/listinfo/opennms-discuss



------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss

--------------------------------------------------------
Intergence Systems is a limited company registered in England and Wales. Registered number: 04667187. Registered office: Quern House, Mill Court, Great Shelford, CB22 5LD.

The content of this message and any attached file are confidential and/or privileged and are intended for the recipient only. If you are not the intended recipient, any unauthorised review, use, re-transmission, dissemination, copying, disclosure or other use of, or taking of any action in reliance of this information is strictly prohibited. If you receive this message in error please contact the sender immediately and then delete this email from your system. Copyright in this email and attachments created by us belongs to Intergence Systems Ltd. Any attachment with this message should be checked for viruses before it is opened. Intergence Systems Ltd cannot be held responsible for any failure by the recipient to test for viruses before opening any attachments. Should you communicate with anyone at Intergence Systems Ltd by email you consent to us monitoring and reading any such correspondence.
--------------------------------------------------------
------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss
Jeff Gehlbach

Re: Opennms with RAMSan = iowait?

Reply Threaded More More options
Print post
Permalink
In reply to this post by Michael Danko
On Nov 6, 2009, at 8:56 AM, [hidden email] wrote:

> Throughput is one thing, seeks are another. The sps and mps stats in  
> iostat can give you an idea if that's what you're waiting on. If  
> this is abnormally high, even caching writes won't help much since  
> the BBWC will just delay a bunch of seeks. What type of filesystem  
> are you using?

I think you might be onto something here.  Rupert, you said in your  
initial post that OpenNMS and PostgreSQL are on the RamSan.  If the  
PostgreSQL tablespaces (usually in /var/lib/pgsql/data on CentOS) are  
on the same I/O path as the RRD files, you're going to have contention  
for seeks.  Moving the tablespaces to mechanical disks should help  
considerably; you might play around with the transaction log's  
location too.

-jeff

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss
Les Mikesell

Re: Opennms with RAMSan = iowait?

Reply Threaded More More options
Print post
Permalink
Jeff Gehlbach wrote:

> On Nov 6, 2009, at 8:56 AM, [hidden email] wrote:
>
>> Throughput is one thing, seeks are another. The sps and mps stats in  
>> iostat can give you an idea if that's what you're waiting on. If  
>> this is abnormally high, even caching writes won't help much since  
>> the BBWC will just delay a bunch of seeks. What type of filesystem  
>> are you using?
>
> I think you might be onto something here.  Rupert, you said in your  
> initial post that OpenNMS and PostgreSQL are on the RamSan.  If the  
> PostgreSQL tablespaces (usually in /var/lib/pgsql/data on CentOS) are  
> on the same I/O path as the RRD files, you're going to have contention  
> for seeks.  Moving the tablespaces to mechanical disks should help  
> considerably; you might play around with the transaction log's  
> location too.

I don't know if anyone mentioned this yet, but postgresql likes to use
fsync() around transactions (as it should...) and Linux has a fairly
horrible implementation of fsync(), at least on ext? filesystems - it
flushes the entire outstanding filesystem buffer to disk, not just the
specified file, and waits for it to complete.  So you really want the
database on a filesystem of its own.

--
  Les Mikesell
    [hidden email]

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss
Benjamin Reed

Re: Opennms with RAMSan = iowait?

Reply Threaded More More options
Print post
Permalink
In reply to this post by Michael Danko
On Nov 6, 2009, at 8:56 AM, [hidden email] wrote:

> With most of the complaints on the list regarding performance always  
> coming back to IO, I think I'll keep the better way to skin the RRD  
> cat in the back of my mind today.

Yeah, it's been brought up here for quite a while that we need to  
finish the (incomplete) JDBC backend in JRobin.  Traditional RRD is  
optimized for reads (and graph rendering) at the expense of writes.

Since most OpenNMS users are writing vastly more data than they read,  
we need to move it to something more efficient at writes.

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss
Graeme Fowler

Re: Opennms with RAMSan = iowait?

Reply Threaded More More options
Print post
Permalink
On Sat, 2009-11-07 at 18:19 -0500, Benjamin Reed wrote:
> Yeah, it's been brought up here for quite a while that we need to  
> finish the (incomplete) JDBC backend in JRobin.  Traditional RRD is  
> optimized for reads (and graph rendering) at the expense of writes.

...indeed, however...

> Since most OpenNMS users are writing vastly more data than they read,  
> we need to move it to something more efficient at writes.

...or rejig the way the RRD data is stored. At the moment, the data is
stored in a tree which looks like this:

rrd/snmp/node/element.rrd
rrd/response/node/element.rrd

That is, there's a separate RRD for each and every node/element pairing.
That results in a lot of files, and therefore a lot of data being
written in individual operations every polling cycle.

I appreciate that this is a well-formed, predictable and simple storage
method however it comes up time and time again as a significant limiting
factor for OpenNMS, especially in deployments which grow past the
performance threshold of the box the install lives on.

Might I suggest (and hey, shoot me for doing so if you like) that
looking at that node/element relationship might be worth doing?

I'd say that, given the fixed nature of an RRD database, you could
create a number of RRDs for specific purposes and then increase the
number of RRDs as the number of node/element items increases.

You would then end up with something like:

rrd/response/rrd1
rrd/response/rrd2
rrd/response/rrdN
...
rrd/snmp/rrd1
rrd/snmp/rrd2
rrd/snmp/rrdN

Each of these RRD files could contain up to 100 (1000? no idea what the
limit would be) data sources (DS) each with the relevant number of RRAs
with appropriate consolidation functions. Obviously each DS would then
reflect a node/element item and should be tracked in the backend DB.

This way, the data being written into each RRD file can be "batched" -
100 (1000?) updates will then take place in a single operation which
will greatly reduce the number of writes & syncs required to complete
each one.

One downside here is that OpenNMS could end up with large amounts of
"dead" data in the RRD files as nodes are removed - it's difficult (if
not impossible) to accurately edit an RRD to cleanse it of old data. I
can't speak for JRobin though.

Another downside is that this is, quite evidently, not backwards
compatible...

I know this doesn't solve the immediate problem, but it might work for
the future to greatly improve IO performance.

Graeme


------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss
Benjamin Reed

Re: Opennms with RAMSan = iowait?

Reply Threaded More More options
Print post
Permalink
On 11/10/09 6:13 AM, Graeme Fowler wrote:

> Each of these RRD files could contain up to 100 (1000? no idea what the
> limit would be) data sources (DS) each with the relevant number of RRAs
> with appropriate consolidation functions. Obviously each DS would then
> reflect a node/element item and should be tracked in the backend DB.

The problem is that it's a pain to recreate rrds when the structure
needs to change, so if a device loses an interface what do you do with
rrd775 that now has dead data in the middle of it?

In the end, I'm just not sure it's worth trying to keep "working around"
RRD's behavior...

--
Benjamin Reed
The OpenNMS Group
http://www.opennms.org/




------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss

signature.asc (267 bytes) Download Attachment
Les Mikesell

Re: Opennms with RAMSan = iowait?

Reply Threaded More More options
Print post
Permalink
Benjamin Reed wrote:

>
>> Each of these RRD files could contain up to 100 (1000? no idea what the
>> limit would be) data sources (DS) each with the relevant number of RRAs
>> with appropriate consolidation functions. Obviously each DS would then
>> reflect a node/element item and should be tracked in the backend DB.
>
> The problem is that it's a pain to recreate rrds when the structure
> needs to change, so if a device loses an interface what do you do with
> rrd775 that now has dead data in the middle of it?
>
> In the end, I'm just not sure it's worth trying to keep "working around"
> RRD's behavior...

At the extreme high end it's probably easier to scale up database writes
than filesystems, but then you need a DBA that knows how to maintain the
beast.  How about a quick hack to cache the whole mess in RAM and do
very lazy disk updates?  An extra 10 gigs or so is pretty cheap compared
to high end disk systems or databases with DBA to tune them.  As long as
the thresholds are getting checked I don't mind losing 15 minutes of
history in the unlikely event of a crash (which is probably going to
take longer than that to fix anyway).

--
   Les Mikesell
    [hidden email]



------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss
Benjamin Reed

Re: Opennms with RAMSan = iowait?

Reply Threaded More More options
Print post
Permalink
On 11/10/09 12:16 PM, Les Mikesell wrote:

> At the extreme high end it's probably easier to scale up database writes
> than filesystems, but then you need a DBA that knows how to maintain the
> beast.  How about a quick hack to cache the whole mess in RAM and do
> very lazy disk updates?  An extra 10 gigs or so is pretty cheap compared
> to high end disk systems or databases with DBA to tune them.  As long as
> the thresholds are getting checked I don't mind losing 15 minutes of
> history in the unlikely event of a crash (which is probably going to
> take longer than that to fix anyway).

We already queue up RRD writes if I/O gets too backed up.  Brozow
implemented the "new" rrdcache feature rrdtool added in 1.4 in OpenNMS a
number of years ago in our RRD backend.  We don't *pre* cache though, so
it's more of a "high I/O eventually fixes itself" rather than "keeping
it down" in the first place though.

--
Benjamin Reed
The OpenNMS Group
http://www.opennms.org/




------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss

signature.asc (267 bytes) Download Attachment
Robert Drake

Re: Opennms with RAMSan = iowait?

Reply Threaded More More options
Print post
Permalink
In reply to this post by Les Mikesell
Les Mikesell wrote:

>
> At the extreme high end it's probably easier to scale up database writes
> than filesystems, but then you need a DBA that knows how to maintain the
> beast.  How about a quick hack to cache the whole mess in RAM and do
> very lazy disk updates?  An extra 10 gigs or so is pretty cheap compared
> to high end disk systems or databases with DBA to tune them.  As long as
> the thresholds are getting checked I don't mind losing 15 minutes of
> history in the unlikely event of a crash (which is probably going to
> take longer than that to fix anyway).
>

I've thought that trading disk space for I/O might be a nice way to fix
it as well.  If you're RRD currently takes 6 gigs but you have 70gb
available, it could write out logs to the entire disk over the course of
the day, then roll it up at 3am when nobody is looking, or once a week
when really nobody is looking.

Kinda like Mysql binary logs.

But, I also thought (at the time) that the Jrobin system was highly
optimized, and I was the only one having this problem because I was
running on sub-par hardware.

hmm.. I think converting to a plug-in method is best, so people can
design all sorts of hacks for actual storage (memcached across several
servers might be nice.  Or anything that spreads it across several servers)

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss
Les Mikesell

Re: Opennms with RAMSan = iowait?

Reply Threaded More More options
Print post
Permalink
In reply to this post by Benjamin Reed
Benjamin Reed wrote:

> On 11/10/09 12:16 PM, Les Mikesell wrote:
>
>> At the extreme high end it's probably easier to scale up database writes
>> than filesystems, but then you need a DBA that knows how to maintain the
>> beast.  How about a quick hack to cache the whole mess in RAM and do
>> very lazy disk updates?  An extra 10 gigs or so is pretty cheap compared
>> to high end disk systems or databases with DBA to tune them.  As long as
>> the thresholds are getting checked I don't mind losing 15 minutes of
>> history in the unlikely event of a crash (which is probably going to
>> take longer than that to fix anyway).
>
> We already queue up RRD writes if I/O gets too backed up.  Brozow
> implemented the "new" rrdcache feature rrdtool added in 1.4 in OpenNMS a
> number of years ago in our RRD backend.  We don't *pre* cache though, so
> it's more of a "high I/O eventually fixes itself" rather than "keeping
> it down" in the first place though.

Can you cache and throttle the writes even before i/o is backed up?
Also, if this is already working would it be a good idea to give java a
huge amount of heap instead of leaving the RAM for the OS disk buffering?

--
   Les Mikesell
    [hidden email]



------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss
Benjamin Reed

Re: Opennms with RAMSan = iowait?

Reply Threaded More More options
Print post
Permalink
In reply to this post by Robert Drake
On 11/10/09 12:51 PM, Robert Drake wrote:

> hmm.. I think converting to a plug-in method is best, so people can
> design all sorts of hacks for actual storage (memcached across several
> servers might be nice.  Or anything that spreads it across several servers)

We do have a plugin system, you could write a RRDStrategy that emulates
our existing interface, and does... whatever.  As long as it can
synthesize the data back when it's needed.

--
Benjamin Reed
The OpenNMS Group
http://www.opennms.org/




------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ

opennms-discuss mailing list

To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-discuss

signature.asc (267 bytes) Download Attachment
1 2