Memory usuage reduction

I've a few OS/2 Rexx apps, which sequently adds data, adds missing data
that wasn't added, and finally fixes detectable data errors. After each
step, which actually modifies the data, a sort is required. Optimization
is possible, w.r.t. both execution speed and memory usuage. That may
even include respecting the sorting order to avoid a sort after that
data processing step. Optimization has no high priority, it's far more
important that the data is right. Buggy ORexx interpreter, Rexx code.

Despite optimization possibilities: what are the known tricks to avoid
heavy use of ((unfreed) shared) memory, requiring reboots after a while?
Does e.g. "DROP" actually help? Or "stem.=''"? Or ...?



---

> Despite optimization possibilities: what are the known tricks to avoid
> heavy use of ((unfreed) shared) memory, requiring reboots after a while?
> Does e.g. "DROP" actually help? Or "stem.=''"? Or ...?

REXX goes to great lengths to ensure that nothing is left after your
program finishes except any data changes that it may have made.

So you must be thinking about the amount of memory that it uses while
running.

As far as I know, DROP does not recover the memory, as that would be a
colossal performance impact, but it may be implementation dependant.

Variables in memory only get longer, they don't shrink, even when the
data they hold gets shorter (same performance hit as above). So avoid
generating long lists, even if you shrink them later.
Incidentally, for your own performance, try not to build a list by
adding stuff to the end of the previous list. Every such additions
required the allocation of a larger block of memory, and the overhead
can get huge:
List = list new_name /* do this enough; program might take days */

At the moment, the only Rexx process on my system is RxApi.exe which is
using ~13MB of virtual memory after days of very heavy usage (including
batch programs which run for hours and deal with 100's of thousands of
files). When an actual REXX program is running, there is a
corresponding rexx.exe process, but that goes away when the program exits.

--
Steve Swift
http://www.swiftys.org.uk/swifty.html
http://www.ringers.org.uk

On Aug 23, 3:23*am, Steve Swift <Steve.J.Sw...@gmail.com> wrote:
> > Despite optimization possibilities: what are the known tricks to avoid
> > heavy use of ((unfreed) shared) memory, requiring reboots after a while?
> > Does e.g. "DROP" actually help? Or "stem.=''"? Or ...?
>
> REXX goes to great lengths to ensure that nothing is left after your
> program finishes except any data changes that it may have made.
>
> So you must be thinking about the amount of memory that it uses while

On OS/2, the Object Rexx object heap is maintained in shared memory,
which allows
objects to be shared across processes. There are multiple downsides
to this,
the biggest being that this shared memory will not go away as long as
there
are any processes running ORexx. The heap will be garbage collected
frequently,
but inadvertently pinning objects into memory can force the heap to
grow to the
point where it can't be expanded any more. A frequent cause of this
is storing
object references in .environment.

>
> As far as I know, DROP does not recover the memory, as that would be a
> colossal performance impact, but it may be implementation dependant.
Drop will clear out the references to the objects stored in the
variables, but
it does not release any memory. Generally, just doing a return from a
method
will accomplish the same thing.


>
> Variables in memory only get longer, they don't shrink, even when the
> data they hold gets shorter (same performance hit as above). So avoid
> generating long lists, even if you shrink them later.
> Incidentally, for your own performance, try not to build a list by
> adding stuff to the end of the previous list. Every such additions
> required the allocation of a larger block of memory, and the overhead
> can get huge:
> List = list new_name /* do this enough; program might take days */

Not strictly true. Variables have no size....only the objects
referenced by
thos variables have size. And not every addition requires an
expansion of the
list or array objects. The expansion rules are more intelligent than
that. The
best strategy when working with arrays is to allocate the array object
with an
initial best guess size to avoid the frequent expansions.


>
> At the moment, the only Rexx process on my system is RxApi.exe which is
> using ~13MB of virtual memory after days of very heavy usage (including
> batch programs which run for hours and deal with 100's of thousands of
> files). *When an actual REXX program is running, there is a
> corresponding rexx.exe process, but that goes away when the program exits..

Again, the situation is different on OS/2 because of the used of
shared memory
for the object heap.

>
> --
> Steve Swifthttp://www.swiftys.org.uk/swifty.htmlhttp://www.ringers.org.uk

>>> Despite optimization possibilities: what are the known tricks to
>>> avoid heavy use of ((unfreed) shared) memory, requiring reboots
>>> after a while? Does e.g. "DROP" actually help? Or "stem.=''"?
>>> Or ...?

> A frequent cause of this is storing object references in .environment.

> Drop will clear out the references to the objects stored in the
> variables, but it does not release any memory.

Thanks.

Using "CALL CharOut target,CharIn(source,,16*1024*1024)" is (close to)
fatal, and because DROP data doesn't seems to release the memory used
with "data=CharIn(source,,16*1024*1024)", that would mean there's no
solution other than using another Rexx interpreter (CRexx, Regina).

I'm not using 16 MB of shared memory per step. But 4*4 MB also counts,
turning Rexx apps in new CMD.EXE-sessions into "/**/;'@EXIT'" (reboot
required to be able to "SAY 'Hello!'" again.

>> Variables in memory only get longer, they don't shrink

> The expansion rules are more intelligent than that.

I actually use a "items=items newitem" somewhere, but I already limited
the number of new items to end up with a reasonable performance. Anyway
IMO the main problem is basicly using 4*4 MB. That's 4 days of 2 MB of
source and 2 MB of processed data. I could supply some code to avoid
having to guess possible causes, but I don't think that adds a lot of
information.

>> At the moment, the only Rexx process on my system is RxApi.exe
>> which is using ~13MB of virtual memory after days of very heavy
>> usage (including batch programs which run for hours and deal with
>> 100's of thousands of files). When an actual REXX program is
>> running, there is a corresponding rexx.exe process, but that goes
>> away when the program exits.

> Again, the situation is different on OS/2 because of the used of
> shared memory for the object heap.

Indeed, ObjectRexx cannot come close to the numbers mentioned above.
One day requires "luck", my source files now is about 1 MB (but near
the end of each year about 2 MB), and completing my 6 Rexx apps takes
a few minutes (exluding download time). That's enough to nearly always
make Rexx unusable from then on, requiring a reboot of a pc with a boot
problem. Too bad there's no serious escape.



---

spamgate@hotmai1.com (ML) wrote:


> Using "CALL CharOut target,CharIn(source,,16*1024*1024)" is (close to)
> fatal...

Maybe that's just an example of you using a 16MB buffer, rather than what
you think you actually need to do?

> , and because DROP data doesn't seems to release the memory used
> with "data=CharIn(source,,16*1024*1024)", that would mean there's no
> solution other than using another Rexx interpreter (CRexx, Regina).

Why wouldn't something along the lines of:

do forever
data=CharIn(source,,1024*1024)
CALL CharOut target,data
end

work? Surely the size of 'data' will not grow?

(With suitable extra logic to stop the loop once all the data's been seen?)


--
Jeremy C B Nicoll - my opinions are my own.

>> Using "CALL CharOut target,CharIn(source,,16*1024*1024)" is (close
>> to) fatal...

> Maybe that's just an example of you using a 16MB buffer, rather than
> what you think you actually need to do?

That's right, in addition to 'mb16=""' and 'DROP mb16'. The problem also
occurs when not using the 'RxVariablePool': no 'free(buffer);'

Please note I'm not sure an accumulated 16 MB is exceeding 'the' limit,
but I recall having the same problem while some splitting large files
in smaller chuncks of about 16 MB.

> Why wouldn't something along the lines of:

> do forever
> data=CharIn(source,,1024*1024)
> CALL CharOut target,data
> end

> work? Surely the size of 'data' will not grow?

This strategy actually demonstrates my problem rather good. Let's say
I'm using 1 MB per app (IRL about 0.1-4.2 MB). But there are 16 apps
(IRL about 6-7 apps). So App1.CMD uses 2.8 MB, App2.CMD uses 2.8 MB,
and so on. That exceeds the limit, albeit no single app ever comes
close to the limit: 16 times 'malloc(1_MB_stem);', but not 16 times
'free(1_MB_stem);', seems to come down to 'malloc(16_MB_stem);', as
long as there's no way to force a true 'free(stem);' with e.g. DROP.



---

ML wrote:
> >> Using "CALL CharOut target,CharIn(source,,16*1024*1024)" is (close
> >> to) fatal...
>
> > Maybe that's just an example of you using a 16MB buffer, rather than
> > what you think you actually need to do?
>
> That's right, in addition to 'mb16=""' and 'DROP mb16'. The problem also
> occurs when not using the 'RxVariablePool': no 'free(buffer);'
>
> Please note I'm not sure an accumulated 16 MB is exceeding 'the' limit,
> but I recall having the same problem while some splitting large files
> in smaller chuncks of about 16 MB.
>
> > Why wouldn't something along the lines of:
>
> > do forever
> > data=CharIn(source,,1024*1024)
> > CALL CharOut target,data
> > end
>
> > work? Surely the size of 'data' will not grow?
>
> This strategy actually demonstrates my problem rather good. Let's say
> I'm using 1 MB per app (IRL about 0.1-4.2 MB). But there are 16 apps
> (IRL about 6-7 apps). So App1.CMD uses 2.8 MB, App2.CMD uses 2.8 MB,
> and so on. That exceeds the limit, albeit no single app ever comes
> close to the limit: 16 times 'malloc(1_MB_stem);', but not 16 times
> 'free(1_MB_stem);', seems to come down to 'malloc(16_MB_stem);', as
> long as there's no way to force a true 'free(stem);' with e.g. DROP.
>
>
>
> ---

That's not the way the ooRexx memory system works. Each allocation is
not a malloc()/free(). The interpreter manages its own garbage
collected memory heap, and allocations are made from that heap. If a
request is unable to be fulfilled from the existing memory heap, a
garbage collection cycle is run, and if there still is no block of
memory of sufficient size to satisfy the request, the heap is expanded
to sufficient size (if it can). Once that large allocation has been
made and it is no longer referenced, the storage will be reclaimed at
the next GC cycle and become part of the general memory pool. The heap
generally will not decrease in size, as there is almost always at least
one live object allocated in every segment of the memory heap. The
memory is not returned to the system, but it is available for other uses
within the interpreter.

Generally, however, continued very large memory allocations will
continue to expand the heap because the memory for the prior allocations
is likely to have been carved into smaller pieces already once it's been
returned to the general storage pool. On OS/2, this can be particularly
bad because the memory heap resides in shared memory, which means it
competes with the rest of the system for area reserved for shared memory
with things like loaded DLLs.

Rick

>> Using "CALL CharOut target,CharIn(source,,16*1024*1024)" is (close
>> to) fatal...

> Maybe that's just an example of you using a 16MB buffer, rather than
> what you think you actually need to do?

That's right, in addition to 'mb16=""' and 'DROP mb16'. The problem also
occurs when not using the 'RxVariablePool': no 'free(buffer);'
Please note I'm not sure an accumulated 16 MB is exceeding 'the' limit,
but I recall having the same problem while some splitting large files
in smaller chuncks of about 16 MB.

> Why wouldn't something along the lines of:

> do forever
> data=CharIn(source,,1024*1024)
> CALL CharOut target,data
> end

> work? Surely the size of 'data' will not grow?

This strategy actually demonstrates my problem rather good. Let's say
I'm using 1 MB per app (IRL about 0.1-4.2 MB). But there are 16 apps
(IRL about 6-7 apps). So App1.CMD uses 2.8 MB, App2.CMD uses 2.8 MB,
and so on. That exceeds the limit, albeit no single app ever comes
close to the limit: 16 times 'malloc(1_MB_stem);', but not 16 times
'free(1_MB_stem);', seems to come down to 'malloc(16_MB_stem);', as
long as there's no way to force a true 'free(stem);' with e.g. DROP.



---

Jeremy Nicoll - news posts wrote:
> Why wouldn't something along the lines of:
>
> do forever
> data=CharIn(source,,1024*1024)
> CALL CharOut target,data
> end
>
> work?

I do something very similar in my "compare.rex" program, except that I
do two charin's. It has never caused any observable problems.

I set the buffer proportional to the size of the files being compared,
but with an upper limit (16MB as I recall) to prevent the progress bar's
progress becoming choppy.

--
Steve Swift
http://www.swiftys.org.uk/swifty.html
http://www.ringers.org.uk

>> Why wouldn't something along the lines of:

>> do forever
>> data=CharIn(source,,1024*1024)
>> CALL CharOut target,data
>> end

>> work?

> I do something very similar in my "compare.rex" program, except
> that I do two charin's. It has never caused any observable
> problems.

O well, why not. Let's try it with such a COMPARE.REX: ;-)


[K:\]copy con COMPARE.REX
/**/;SAY "Hello world!";EXIT^Z
1 file(s) copied.

[K:\]compare
SYS1041: The name compare is not recognized as an
internal or external command, operable program or batch file.

[K:\]compare.rex
SYS0193: K:\COMPARE.REX cannot be run in a OS/2 session.



---