What triggers garbage collection?

[BruceMount]

What triggers garbage collection?  I have an Eiffel 6.4 program that read over 14 gigabytes of text, spread across many files.  The simplified pseudo code is:

For All Files
--Read a whole file
--Process data from the file
end

A moderate amount of data is kept and the rest is not used, which results in a lot of memory garbage.  The program eventually crashes due to a "Special allocation: No more memory." exception.

However, when I inherit from MEMORY and add a full_collect inside the loop, the program runs to the end.

How come a full_collect is not automatically triggered when a "No more memory" exception occurs?  Wouldn't that be exactly one of the times to trigger the garbage collector?

Thanks,

--Bruce

[Emmanuel Stapf]

> What triggers garbage collection?  I have an Eiffel 6.4 program that read
> over 14 gigabytes of text, spread across many files.  The simplified
> pseudo code is:
>
> For All Files
> --Read a whole file
> --Process data from the file
> end
>
> A moderate amount of data is kept and the rest is not used, which results
> in a lot of memory garbage.  The program eventually crashes due to a
> "Special allocation: No more memory." exception.

On 32-bit machines, the maximum amount of data that the runtime can allocate for a
SPECIAL is 128MB and that's most likely what you are hitting.
 
> However, when I inherit from MEMORY and add a full_collect inside the
> loop, the program runs to the end.
>
> How come a full_collect is not automatically triggered when a "No more
> memory" exception occurs?  Wouldn't that be exactly one of the times to
> trigger the garbage collector?

The question is to find out why by triggering the GC on a regular basis your
program does not allocate a SPECIAL larger than 128MB. We can only answer that
question if you can provide a reproducible example.

Regards,
Manu

[BruceMount]

Manu:

Thanks for your response.

> The question is to find out why by triggering the GC on
> a regular basis your program does not allocate a SPECIAL
> larger than 128MB. We can only answer that question if
> you can provide a reproducible example.

I doubt I'll be able to do this, given that it does not seem to hit the limit when reading only a few gigabytes of text.  The only time I hit this problem is when I read a lot of text (over 10 GB) and there is probably no easy way to give you 10-13 GB of text to reproduce it.

I just would have though that running out of memory would trigger a full_collect.

Thanks Manu,

--Bruce

[Emmanuel Stapf]

> I doubt I'll be able to do this, given that it does not seem to hit the
> limit when reading only a few gigabytes of text.  The only time I hit
> this problem is when I read a lot of text (over 10 GB) and there is
> probably no easy way to give you 10-13 GB of text to reproduce it.
>
> I just would have though that running out of memory would trigger a
> full_collect.

You are not running out of physical memory in this particular case, you are simply
hitting a runtime memory limit preventing the allocation of an object larger than
128MB in a 32-bit system. In such cases, the run-time has not much choice but to
raise an exception. It would be interesting if you could run the same program on a
64-bit system and compare the outcome.

Regards,
Manu

[Mark Bolstad]

Could you please explain how there is a 128MB limit for allocation of an
object? I'm sure it's some internal thing, but shouldn't it have a
theoretical limit of 4 GB on 32-bit?

This implies that the following should fail:

a: ARRAY [INTEGER_32]
...
create a. make (1, 100000000) -- An ~400MB array

Mark


On Tue, Sep 29, 2009 at 7:40 PM, Emmanuel Stapf [ES] <[hidden email]>wrote:

>
>
> > I doubt I'll be able to do this, given that it does not seem to hit the
> > limit when reading only a few gigabytes of text. The only time I hit
> > this problem is when I read a lot of text (over 10 GB) and there is
> > probably no easy way to give you 10-13 GB of text to reproduce it.
> >
> > I just would have though that running out of memory would trigger a
> > full_collect.
>
> You are not running out of physical memory in this particular case, you are
> simply
> hitting a runtime memory limit preventing the allocation of an object
> larger than
> 128MB in a 32-bit system. In such cases, the run-time has not much choice
> but to
> raise an exception. It would be interesting if you could run the same
> program on a
> 64-bit system and compare the outcome.
>
> Regards,
> Manu
>
>  
>

[Non-text portions of this message have been removed]

[Emmanuel Stapf]

> Could you please explain how there is a 128MB limit for allocation of an
> object? I'm sure it's some internal thing, but shouldn't it have a
> theoretical limit of 4 GB on 32-bit?

Historically we use 5 bits of the 32-bit to add some information about the
allocated memory, the remaining is for the size. When this scheme was devised 15
years ago, allocating 128MB was beyond the available memory. We never tried to
improve the situation because of the advent of 64-bit computing for which the
maximum size is 2^59.
 
> This implies that the following should fail:
>
> a: ARRAY [INTEGER_32]
> ...
> create a. make (1, 100000000) -- An ~400MB array

On 32-bit system, it should, but not on 64-bit system.

Regards,
Manu

[Colin LeMahieu]

On most OSs you'll be limited to 2gb of user memory.  Accounting for thread
stacks and executable areas you'll typically be down to about 1gb of heap
memory assuming no fragmentation.  Empirically I've found allocating over
~384mb of memory in a contiguous region is risking an out of memory
exception due to LOB heap fragmentation in C#.  C# doesn't compact the large
object heap.

On Tue, Sep 29, 2009 at 9:24 PM, Mark Bolstad <[hidden email]>wrote:

>
>
> Could you please explain how there is a 128MB limit for allocation of an
> object? I'm sure it's some internal thing, but shouldn't it have a
> theoretical limit of 4 GB on 32-bit?
>
> This implies that the following should fail:
>
> a: ARRAY [INTEGER_32]
> ...
> create a. make (1, 100000000) -- An ~400MB array
>
> Mark
>
> On Tue, Sep 29, 2009 at 7:40 PM, Emmanuel Stapf [ES] <[hidden email]<manus%40eiffel.com>
> >wrote:
>
>
> >
> >
> > > I doubt I'll be able to do this, given that it does not seem to hit the
> > > limit when reading only a few gigabytes of text. The only time I hit
> > > this problem is when I read a lot of text (over 10 GB) and there is
> > > probably no easy way to give you 10-13 GB of text to reproduce it.
> > >
> > > I just would have though that running out of memory would trigger a
> > > full_collect.
> >
> > You are not running out of physical memory in this particular case, you
> are
> > simply
> > hitting a runtime memory limit preventing the allocation of an object
> > larger than
> > 128MB in a 32-bit system. In such cases, the run-time has not much choice
> > but to
> > raise an exception. It would be interesting if you could run the same
> > program on a
> > 64-bit system and compare the outcome.
> >
> > Regards,
> > Manu
> >
> >
> >
>
> [Non-text portions of this message have been removed]
>
>  
>

[Non-text portions of this message have been removed]

[BruceMount]

Indeed, I have something like this, as I read and ENTIRE text file into a stream in one big chunk (up to 18 meg).  At the time I thought this would not be an issue, given that plenty of RAM was available, but Manu's insight about 5-bit allocations on 32-bit systems changes that.

I'll read the file in smaller (slower) chunks and see if that helps.

THANKS GUYS!

--Bruce



Colin LeMahieu <clemahieu@...> wrote:
> ....Empirically I've found allocating over ~384mb of
> memory in a contiguous region is risking an out of memory
> exception due to LOB heap fragmentation in C#.

Mark Bolstad <[hidden email]:
> > This implies that the following should fail:
> >
> > a: ARRAY [INTEGER_32]
> > ...
> > create a. make (1, 100000000) -- An ~400MB array
> >
> > Mark