rrdcached + collectd issues
|
| 1 2 |
|
|
||||||||
|
|
||||||||
|
|
|
||||||||
|
|
|
||||||||
|
|
|
||||||||
|
In reply to this post
by Thorsten von Eicken-2
Hi Thorsten,
On Fri, Oct 09, 2009 at 11:46:26AM -0700, Thorsten von Eicken wrote: > I've compiled rrdcched with -pg to get gprof > output, but haven't been successful. Just a short note on that: Most of the actual work in RRDCacheD is done in threads. gprof is not able to handle that very well (basically not at all). > It looks to like that's mostly the journal replay stuff and very little > more. Journal replay is done during startup in the main thread, so gprof is able to grab information about that but probably misses about everything else. You might want to have a look at callgrind, which is one of the valgrind tools that may be used for call graph tracing and which supports multi- threaded programs as well. Cheers, Sebastian -- Sebastian "tokkee" Harl +++ GnuPG-ID: 0x8501C7FC +++ http://tokkee.org/ Those who would give up Essential Liberty to purchase a little Temporary Safety, deserve neither Liberty nor Safety. -- Benjamin Franklin _______________________________________________ rrd-developers mailing list [hidden email] https://lists.oetiker.ch/cgi-bin/listinfo/rrd-developers |
||||||||
signature.asc (204 bytes)
|
In reply to this post
by Thorsten von Eicken-2
Hi Thorsten,
On Fri, Oct 09, 2009 at 11:46:26AM -0700, Thorsten von Eicken wrote: > - rrdtool trunk rev 1889 (had trouble compiling collectd with newer > versions) this is a bug introduced in RRDtool in revision 1906: src/rrd_client.h will do the following check / include: #ifdef HAVE_CONFIG_H #include "../rrd_config.h" #endif Of course “HAVE_CONFIG_H” is defined when building collectd - it's using the autotools, too. The (now globally available) config file will try to include "../rrd_config.h" and fail because it is not there. I don't think this inclusion is necessary when building RRDtool: Basically the same check / inclusion is done within src/rrd_tool.h. All we have to do is to make sure all .c-files within RRDtool include this header before including src/rrd_client.h. The only file doing this in the wrong order is src/rrd_client.c (I just checked). With the attached patch against revision 1934 I was able to build the rrdcached plugin of collectd without problems. Regards, —octo -- Florian octo Forster Hacker in training GnuPG: 0x91523C3D http://verplant.org/ commit 0299e04d2ed41324a38f6d3d90e1614a0b0e3f0d Author: Florian Forster <[hidden email]> Date: Sat Oct 10 12:16:21 2009 +0200 src/rrd_client.[ch]: Fix build issues introduced in revision 1906. diff --git a/src/rrd_client.c b/src/rrd_client.c index a4f1ba9..b018ee0 100644 --- a/src/rrd_client.c +++ b/src/rrd_client.c @@ -21,8 +21,8 @@ **/ #include "rrd.h" -#include "rrd_client.h" #include "rrd_tool.h" +#include "rrd_client.h" #include <stdlib.h> #include <string.h> diff --git a/src/rrd_client.h b/src/rrd_client.h index 787c2b6..6c48dec 100644 --- a/src/rrd_client.h +++ b/src/rrd_client.h @@ -22,14 +22,6 @@ #ifndef __RRD_CLIENT_H #define __RRD_CLIENT_H 1 -#if defined(_WIN32) && !defined(__CYGWIN__) && !defined(__CYGWIN32__) && !defined(HAVE_CONFIG_H) -#include "../win32/config.h" -#else -#ifdef HAVE_CONFIG_H -#include "../rrd_config.h" -#endif -#endif - #ifndef WIN32 # ifdef HAVE_STDINT_H # include <stdint.h> _______________________________________________ rrd-developers mailing list [hidden email] https://lists.oetiker.ch/cgi-bin/listinfo/rrd-developers |
||||||||
|
|
In reply to this post
by Thorsten von Eicken-2
Hi Thorsten,
On Fri, Oct 09, 2009 at 04:41:55PM -0700, Thorsten von Eicken wrote: > > This sounds like collectd not sending updates to rrdcached. If they > > are not in the journal, then rrdcached has not received them. > > Yes, the question is whether it's collectd's fault or rrdcached's > fault.. assuming you're receiving those values via collectd's Network plugin, this is what's going on: * There are two threads handling incoming network traffic. The first reads the packets from a network socket and appends them to a linked list. * The second thread parses the data and dispatches the included data to the daemon, resulting in roughly 20 “value lists” per packet. * The dispatch thread will call “rrdc_update” within the rrdcached plugin, resulting in a single update instruction being sent to RRDCacheD. The call returns after a status has been returned by the daemon. (This is what Kevin meant with the BATCH operation, where this call would return immediately without waiting for a status to be returned.) If RRDCacheD takes too long to answer, the dispatch thread will wait there and not dequeue any more values from that queue of received and unparsed packets. If this is the cache, you should see some (linear?) memory growth of the collectd process. You can also try to forcibly quit collectd (kill -9) and immediately restart collectd. If the data RRD files were lagging behind is simply lost, this is a indication of the data being within collectd and waiting to be sent to RRDCacheD. (It's not yet possible to “watch” the length of this queue directly. I'll add some measurements to the Network plugin so we can see what's going on eventually …) > Yeah, as I mentioned above, I'm very steady at 3k updates received per > sec. Are you talking about network packets or separate updates here? Depending on your data every packet can contain about 20–30 separate values, to the difference is significant ;) > > > I/O is not a problem as I mentioned, it's pure CPU. I've compiled > > > rrdcched with -pg to get gprof output, but haven't been > > > successful. *The* data structure within RRDCacheD that is *supposed* to grow as more data is to be cached is “cache_tree”. So *the* call that is supposed to be the limiting factor is this line within “handle_request_update”: ci = g_tree_lookup (cache_tree, file); (I'm talking about normal operation, of course. Replaying the journal is special.) > >> % cumulative self self total > >> time seconds seconds calls s/call s/call name > >> 55.12 62.39 62.39 280843249 0.00 0.00 buffer_get_field If the CPU really was busy for 250 minutes (and not stuck doing I/O), then about 62.39/15000 (~4.2 %) of the time was spent in “buffer_get_field”. It might be possible to optimize that function further, but I don't think it's worth it. The real bottleneck is probably somewhere else. One possible way to make this faster is to use something like: *ptr = strcspn (buffer, "\\ "); if (*ptr == ' ') /* Normal case: Field without backslash */ else /* More complex escape sequence handling */ I would surprised if this was the cause of those performance issues, though. Looking at the code it looks like the schoolbook case for branch prediction, something modern CPUs are *very* good in … Overall I have the feeling that the update command is slower than expected – at least this would explain your issues. It'd be best if you could try to get some reliable profiling data. Without it, optimization makes not much sense :/ Regards, —octo -- Florian octo Forster Hacker in training GnuPG: 0x91523C3D http://verplant.org/ _______________________________________________ rrd-developers mailing list [hidden email] https://lists.oetiker.ch/cgi-bin/listinfo/rrd-developers |
||||||||
|
|
|
||||||||
|
In reply to this post
by Florian Forster-2
|
||||||||
|
|
In reply to this post
by Florian Forster-2
|
||||||||
|
|
|
||||||||
|
Am 12.10.2009 20:33, schrieb Thorsten von Eicken:
> Thorsten von Eicken wrote: >> - I'm wondering how we could overcome the RRD working set issue. Even >> with rrdcached and long cache periods (e.g. I use 1 hour) it seems >> that the system comes to a crawl if the RRD working set exceeds >> memory. One idea that came to mind is to use the caching in rrdcached >> to convert the random small writes that are typical for RRDs to more >> of a sequential access pattern. If we could tweak the RRD creation >> and the cache write-back algorithm such that RRDs are always accessed >> in the same order, and we manage to get the RRDs allocated on disk in >> that order, then we could use the cache to essentially do one sweep >> through the disk per cache flush period (e.g. per hour in my case). >> Of course on-demand flushes and other things would interrupt this >> sweep, but the bulk of accesses could end up being more or less >> sequential. I believe that doing the cache write-back in a specific >> order is not too difficult, what I'm not sure of is how to make it >> such that the RRD files get allocated on disk in the that order too. >> Any thoughts? >> > One further thought, instead of trying to allocate RRDs sequentially, > if there is a way to query/detect where each RRD file is allocated on > disk, then rrdcached could sort the dirty tree nodes by disk location > and write them in that order. I don't know whether Linux (or FreeBSD) > have a way to query disk location or to at least infer it. > > TvE unrelated to this as modern harddrives mayreallocate certain sectors if they feel that one particular sector cannot be read\written properly. Thus trying to infer the physical location of data will not be accurate. To take this even further not even the volume you write onto hast to exist physically, e.g. you might have a RAID or LVM in which case one logical sector corresponds to more than one physical location or as in cases of a RAM disc none at all (at least no permanent one). To make the picture complete there's even another factor that makes this nothing you want to do in software: Modern harddrives usually "plan" their read and write requests automatically already. So when write accesses occure right behind each other the harddrive will already figure out the best way to write them - unless you enforce synchronous request completion with e.g. O_DIRECT or thelike. Regards, BenBE. _______________________________________________ rrd-developers mailing list [hidden email] https://lists.oetiker.ch/cgi-bin/listinfo/rrd-developers |
||||||||
|
|
|
||||||||
|
|
In reply to this post
by Thorsten von Eicken-2
|
||||||||
|
|
In reply to this post
by Thorsten von Eicken-2
Am 13.10.2009 08:50, schrieb Thorsten von Eicken:
> Benny Baumann wrote: >> Am 12.10.2009 20:33, schrieb Thorsten von Eicken: >> >>> Thorsten von Eicken wrote: >>> >>> One further thought, instead of trying to allocate RRDs sequentially, >>> if there is a way to query/detect where each RRD file is allocated on >>> disk, then rrdcached could sort the dirty tree nodes by disk location >>> and write them in that order. I don't know whether Linux (or FreeBSD) >>> have a way to query disk location or to at least infer it. >>> >>> TvE >>> >> Even though Linux and Windows (and I guess most other OSes) allow to >> query the "logical" disc position the physical location maybe completely >> unrelated to this as modern harddrives mayreallocate certain sectors if >> they feel that one particular sector cannot be read\written properly. >> Thus trying to infer the physical location of data will not be accurate. >> To take this even further not even the volume you write onto hast to >> exist physically, e.g. you might have a RAID or LVM in which case one >> logical sector corresponds to more than one physical location or as in >> cases of a RAM disc none at all (at least no permanent one). >> >> To make the picture complete there's even another factor that makes this >> nothing you want to do in software: Modern harddrives usually "plan" >> their read and write requests automatically already. So when write >> accesses occure right behind each other the harddrive will already >> figure out the best way to write them - unless you enforce synchronous >> request completion with e.g. O_DIRECT or thelike. >> > Ben, I understand what you are saying. The question I have is not > whether it's always possible and meaningful to do this, the question > is whether it's possible to arrange this with the right set-up. Also, > I wouldn't be looking for 100% accuracy, even if it was only roughly > it could be a significant improvement. > TvE difficult process. The following parameters might have influence (I tried to order them from low-level to high-level, YMMV): - The type of storage used, like magnetic, flash, optical, ... (Influence: The way data is accessed or modified) - The encoding used to store the data (Influence: determines if two pieces of data nearby might get reached) - Physical organization of the media (Influence: Where is physical sector 0) - Sector reallocation (Influence: Remapping of data locations) - Logical structure of the media (Influence: Where is logical sector 0 and how to map from physical to logical sectors?) - Virtual Resizing of the drive (e.g. hiding parts of the drive from view) (Influence: Parts of the drive become invisible to the casual observer) - Partitioning of the media for the OS (Influence: Offset for sector mapping) - RAID / LVM (Influence: Virtualizing of the storage) - Volume File System (Influence: Remapping of partition sectors to logical file system sectors/clusters) - File Allocation (Mapping a logical file offset to a volume cluster) Taking all or at least some of these into account, humbly said, it doesn't make sense to try guessing the disc location as you not only will have to care about tousands of combinations of those factors, have a lot of more, complex work, but certainly will guess wrong most of the time. And guessing wrong will in buest case change nothing, but in worst case will harm your performance twice: Once for guessing at all and second for guessing wrong. The better alternative IMHO should be analyzing where things waste valueable time. I'm not sure about the internals of the write threads, but usually you have one "task" thread preparing a queue of stuff to write (or multiple if needed) and many (I'll come to this in a second) worker threads to flush things to disc. Now for the number: 2 threads can be better than ten, but needn't be as this depends on some things: 1. the number of CPUs in the system, 2. the drive speed, 3. data buffering. To get a feeling for this: Consider compiling a large program or software on a multi core CPU: You'll not get anything from your hardware when working with one make instance. That's for sure. Two will be better, but needn't be optimum. Although Compiling isn't as I/O-bound as flushing data to disk, there's a point where the number of processes that read data from disk (for compiling) need just as much time as things need to compile. That's why for compiling you won't do one job per CPU but usually 2 or 3: One process per CPU does I/O, the other does CPU work. So to take this for the I/O-Heavy work: Do two threads per CPU (one prepares the data, the other writes) and encourage the OS to use much buffer space in RAM, i.e. avoid using direct disc I/O with O:_DIRECT as O_DIRECT forces the OS to wait for the operation to complete. Instead use as many asynchronous calls and do e.g. select calls to find which succeeded and which failed. So one I/O thread can track say 16* I/O operations at a time, discard those that completed successfully, retry those that failed and leave the kernel working for you. One way to optimize things here is, to collect requests for one file per thread, i.e. if there are 2 requests on descriptor 7 push them to the same thread if possible instead of distributing them among threads. But since async threads cannot be paralleld on the same file (at least on Windows IIRC) you'd need to do something like this anyway. This saves some trouble with synching your threads with using that file descriptor and thus improves performance a bit. I hope this little description helps a bit with finding a suitable solution. *arbitrarily choosen, no empiric data _______________________________________________ rrd-developers mailing list [hidden email] https://lists.oetiker.ch/cgi-bin/listinfo/rrd-developers |
||||||||
|
In reply to this post
by Thorsten von Eicken-2
|
||||||||
|
|
|
||||||||
|
|
In reply to this post
by Thorsten von Eicken-2
|
||||||||
|
|
||||||||
| Free Embeddable Forum Powered by Nabble | Help |
