(PMShell) memory problems with 4os2 and heavy batch processing (?)

[Andreas Buchinger]

I think there are some memory (or other resources?) problems when using 4os2.

I discovered this when compiling gcc projects from within 4os2 several times.
Programs from xcenter can not be started then, F1 for opening help or option
does not work... simply no new program can be started. I think this is a problem
with pmshell resources. But didn't see any obvious problem with theseus4. Though
I'm not very familiar with theseus.

Reseting the wps most times help. But sometimes 2 xcenter objects can not be
closed and so the wps do not restart. Reboot required.

At first I thought this is a problem only when compiling with gcc, but now I
found a simple way to reproduce this. f.i. "for /r %i in (*beatles*mp3) do dir
%i" started on a 300GB partition with 98.021 files in 26.585 dirs triggers this
problem. dir /s on this partition no problem.

So problem is with -
1) heavy batch processing (compiling projects with gcc)
2) for /r (not sure if /r does make a difference or the amount of files)

Ideas?

[Steven Levine]

In <ii3k6s$tt6$***@dough.gmane.org>, on 01/30/11
at 01:05 PM, Andreas Buchinger <andi.b-***@public.gmane.org> said:

Hi Andreas,

Post by Andreas Buchinger
I think there are some memory (or other resources?) problems when using
4os2. I discovered this when compiling gcc projects from within 4os2 several
times.

When working with today's large apps, one must always remain aware that
linear address space in lower memory is a scare resource.

Post by Andreas Buchinger
Programs from xcenter can not be started then, F1 for opening
help or option does not work... simply no new program can be started. I
think this is a problem with pmshell resources.

It's not pmshell specifically, but pmshell can contribute. The mostly
likely issue is that some app has used up enough lower memory to prevent
the shared memory boundary from moving down. This will cause all sorts of
failures. Things have actually gotten better now that many apps are built
with gcc and libc063. It puts the heap in the upper private arena by
default and this makes it less likely to run out of lower memory.
OpenWatcom apps don't do this by default so they are more likely to run
out of memory. The reason that it does not happen more often is that the
bulk of the apps built by OpenWatcom do not use lots of memory. One
exception is fm/2.

Post by Andreas Buchinger
But didn't see any
obvious problem with theseus4. Though I'm not very familiar with
theseus.

Make sure you have Theseus started before you run out of memory. Use
System->Linear usage by process to find the process that is using the most
lower memory. Often this will be pmshell because some WPS classes are not
careful about how much memory they retain.

Post by Andreas Buchinger
Reseting the wps most times help.

If you catch the problem before the out of memory condition damages the
WPS internally, resetting the WPS with CADH will allow you to resover.

Post by Andreas Buchinger
But sometimes 2 xcenter objects can not
be closed and so the wps do not restart.

There are some defects in the WPS's out of memory error recovery logic.
This is hard code to make perfect. The result is that one or more thread
get stuck in the exit list and never terminate.

Post by Andreas Buchinger
At first I thought this is a problem only when compiling with gcc, but
now I found a simple way to reproduce this. f.i. "for /r %i in
(*beatles*mp3) do dir %i" started on a 300GB partition with 98.021 files
in 26.585 dirs triggers this problem. dir /s on this partition no
problem.

I've not looked at this code yet, but most likely 4OS2 is doing a
recursive walk of the directory tree and building a large filename list
which it then sorts and prunes. This can use a significant amount of
memory given the number of files in your partition.

Post by Andreas Buchinger
1) heavy batch processing (compiling projects with gcc)

This is most likely gcc using a lot of memory for in-memory data
structures. You can use Theseus's Process->Memory leak detection feature
to watch the usage grow.

Post by Andreas Buchinger
2) for /r (not sure if /r does make a difference or the amount of files)

/r will make a big difference.

Post by Andreas Buchinger
Ideas?

I recommend opening a TRAC ticket so that we remember how to trigger the
error. I owe TRAC a couple of tickets myself. I have the best of
intentions to fix the related defects, but I can't seem to get 4OS2 to the
top of the list.

Regards,

Steven