Re: V-Series/VOS 16 porting tips for PL/I, collected

robin wrote:
>
> But if STRINGSIZE is not enabled, storage can be
> corrupted. There is another section of the manual
> that is explicit about the conditions under which this occurs
> (but I don't have the manual at hand to quote).
>
PL/I string assignment semantics specify that if the source of a string
assignment is longer than the (maximum) length of the target, STRINGSIZE
is raised if enabled. Upon normal return or if STRINGSIZE is disabled,
the source, truncated to the (maximum) length of the target, is
assigned. Thus storage corruption can never occur simply because
STRINGSIZE would have been raised if enabled. Also direct assignment to
a string element variable can never cause storage corruption.

However if the assignment is to a SUBSTR pseudo variable, storage
corruption could occur if STRINGRANGE is not enabled. Upon normal return
from STRINGRANGE, the SUBSTR reference is forced to lie within the
string specified in the first argument. Thus enabling STRINGRANGE can
prevent storage corruption caused by invalid SUBSTR pseudo variable indices.

Storage corruption can also occur in a string assignment to a based
string variable (if an invalid pointer value is specified) or a string
parameter of an external procedure if the entry declaration in the
calling procedure does not match the parameter declaration. However
neither STRINGRANGE nor STRINGSIZE is of any help in detecting these
situations.

robin calvert wrote:
>
> "Jay Levitt" <jay+news(at)jay.fm> wrote in message
> > robin wrote:
> >
> > > Re these tips, it's important that STRINGSIZE and STRINGRANGE
> > > be enabled, as well as FOFL and SIZE.
> >
> > Unfortunately, on Stratus, we only have FOFL (in the form of
> > -fixedoverflow) and STRINGRANGE (in the form of -check).
> > We explicitly don't have SIZE, and I don't know what
> > STRINGSIZE is, but I don't think we have it either...
>
> STRINGSIZE causes a check to be included that the destination
> (target) of a character or bit operation does not exceed the
> length of that target. And if the target is the shorter, the
> moved characters are truncated. In this way, storage following
> the destination area is not corrupted.
>
> If you do not have this facility, you need to include code that
> not only checks for this condition, but prevents it from
> happening.


Let me clarify the situation for Stratus VOS customers. I would not
want to confuse them with information that may be accurate for other
platforms but not for our platform. VOS PL/I truncates strings at
assignment time and this eliminates most cases of data corruption.

Stratus provides a PL/I compiler that is very close to the Subset G
version of ANSI PL/I. The reference is ANSI X3.74-1987 (R1993):
Information Systems - Programming Language - PL/I General-Purpose Subset.

You can view our manual by pointing the Internet Explorer web browser at

http://stratadoc.stratus.com/

click on the VOS Operating System, pick a release, and pick a platform.

The PL/I manual is filed under "Languages and Subroutines",
and the PL/I Language manual is number R009.

Or you can click on the following link, if the mail system doesn't mung
it up too badly (remove any added carriage-returns):

http://stratadoc.stratus.com/vos/16....h1r009-04.html

VOS PL/I does not provide the stringsize condition and there is no
substitute for it. You don't need it to avoid data corruption,
because the compiler always knows the length of the target string,
and will not write beyond that point. Even in the case where the
target is a parameter declared "char (*)", the calling program has
passed a hidden argument that supplies the extent of the string. You
might want stringsize so you can catch cases of truncation, but this
feature was omitted from Subset G, presumably in the belief that
string truncation is the norm, not the exception.

VOS PL/I does not provide the stringrange condition. Instead, it
provides the -check control argument on the pl1 command. When
-check is used, the compiler adds code to ensure that references to
the substr() built-in function stay within the declared length of
the string; if the reference is invalid the code signals the error
condition. Without -check, the compiled code presumes that the
programmer knows what she or he is doing and executes the substr
without checking.

Most of the cases of string corruption that I've seen, or have
bitten me, have involved strings with expression extents (i.e.,
strings whose length is an expression). If some variable in the
expression extent is uninitialized or incorrect, then storage can
be corrupted or unitialized storage can be referenced. But it is
unclear to me that stringsize or stringrange would be of much use
here, because to the compiled code, the length of the string *is*
the value of the expression. There is no way to know that 1 is ok
and 32766 is bad. Just catching cases of truncation isn't
sufficient; the bogus length could still not result in any truncation.

You really need some sort of meta-data that can be used to validate
the putative extent. That's not practical in a compiled environment.
Instead, I encourage people to use a different control argument,
-check_uninitialized, in which the compiler performs additional
analysis to find variables that are unset in each flow unit. (A
flow unit is a collection of statements that have a single entry
and single exit).

The bottom line, as now said by many people, is that programmers
should use the error-catching features of the language and the
compiler. The costs associated with encountering latent defects
far outweigh the slight performance gain that comes from avoiding
the checking code.

Here is a trivial example of -check in action.

display t_check.pl1 -line_numbers

%es#m105_user>Languages>Green>t_check.pl1 08-01-17 12:47:20 est

1 t_check: proc;
2
3 declare n bin (31);
4 declare s char (32);
5 declare i bin (15);
6
7 on error begin; n = oncode (); goto caught; end;
8
9 put list ('enter i: ');
10 get list (i);
11 s = '';
12 substr (s, i, 1) = 'x';
13 put list (i, s);
14 put skip;
15 return;
16
17 caught:
18 put list ('caught the error; oncode = ', n);
19 put skip;
20
21 end;


ready 12:47:20 0.006 14
pl1 t_check -check
PL/I: t_check.pl1
ready 12:47:29 0.056 493
bind t_check
ready 12:47:34 0.061 148
t_check
enter i: 0
caught the error; oncode = 1161
ready 12:47:41 0.008 21
display_error 1161
e$stringrange. A sub-string reference extends beyond the range of the string.
ready 12:47:45 0.002 9

Thanks
PG
--
Paul Green, Senior Technical Consultant, Stratus Technologies.
Voice: +1 978-461-7557; FAX: +1 978-461-3610; Mobile: +1 (978) 235-2451;
AIM: PaulGreen

"Jay Levitt" <jay+news@jay.fm> wrote in message news:1uurgq36lk9fa.dlg@jay.fm...
> On Mon, 14 Jan 2008 21:54:20 GMT, robin wrote:
>
> >> Unfortunately, on Stratus, we only have FOFL (in the form of
> >> -fixedoverflow) and STRINGRANGE (in the form of -check). We explicitly
> >> don't have SIZE, and I don't know what STRINGSIZE is, but I don't think we
> >> have it either...
> >
> > If you do not have this facility, you need to include code that
> > not only checks for this condition, but prevents it from
> > happening.
>
> Totally untrue; I have never included such code. I just make sure that all
> the really important data is stored at the beginning of the string.

No check needed? - This is what you wrote:

"Jay Levitt" <jay+news@jay.fm> wrote in message news:1uurgq36lk9fa.dlg@jay.fm...

"declare big1 char(32000)var;
"declare big2 char(32000)var;
"declare small char(4096)var;

" big1 = copy ('1234567890', 3100); /* 31000 bytes long */
" small = big; /* should truncate at 4096 */
" big2 = small || big1; /* should concatenate and truncate at 32000, */
" /* but overflows the string instead even with */
" /* -check enabled */ "

robin wrote:
> "Jay Levitt" <jay+news@jay.fm> wrote in message news:1uurgq36lk9fa.dlg@jay.fm...
>
>>On Mon, 14 Jan 2008 21:54:20 GMT, robin wrote:
>>
>>
>>>>Unfortunately, on Stratus, we only have FOFL (in the form of
>>>>-fixedoverflow) and STRINGRANGE (in the form of -check). We explicitly
>>>>don't have SIZE, and I don't know what STRINGSIZE is, but I don't think we
>>>>have it either...
>>>
>>>If you do not have this facility, you need to include code that
>>>not only checks for this condition, but prevents it from
>>>happening.
>>
>>Totally untrue; I have never included such code. I just make sure that all
>>the really important data is stored at the beginning of the string.
>
>
> No check needed? - This is what you wrote:
>
> "Jay Levitt" <jay+news@jay.fm> wrote in message news:1uurgq36lk9fa.dlg@jay.fm...
>
> "declare big1 char(32000)var;
> "declare big2 char(32000)var;
> "declare small char(4096)var;
>
> " big1 = copy ('1234567890', 3100); /* 31000 bytes long */
> " small = big; /* should truncate at 4096 */
> " big2 = small || big1; /* should concatenate and truncate at 32000, */
> " /* but overflows the string instead even with */
> " /* -check enabled */ "
>
>

So? Did you run this?

In article <478fe2a2$0$5009$4c368faf@roadrunner.com>,
Peter Flass <Peter_Flass@Yahoo.com> wrote:

>
> > "declare big1 char(32000)var;
> > "declare big2 char(32000)var;
> > "declare small char(4096)var;
> >
> > " big1 = copy ('1234567890', 3100); /* 31000 bytes long */
> > " small = big; /* should truncate at 4096 */
> > " big2 = small || big1; /* should concatenate and truncate at 32000, */
> > " /* but overflows the string instead even with */
> > " /* -check enabled */ "
> >

I did run it.

The program fails, but the problem is NOT a buffer overflow. (I ran on
a V-something, running 15.3.0ae. I don't have access to a VOS 16 system
with PL/I.)

First, here's the code I actually used:

---
test_overflow: proc options(main);

declare big1 char(32000)var;
declare big2 char(32000)var;
declare small char(4096)var;

big1 = copy ('1234567890', 3100); /* 31000 bytes long */
put skip list ('big1 - maxlength ', maxlength(big1),
'length ', length(big1));
small = big1; /* should truncate at 4096 */
put skip list ('small - maxlength ', maxlength(small),
'length ', length(small));

big2 = small || big1; /* should concatenate and truncate at 32000, */
/* but overflows the string instead even with */
/* -check enabled */
put skip list ('big2 - maxlength ', maxlength(big2),
'length ', length(big2));

end;
---

Compile and bind with no options.

The program fails.

---
test_overflow

big1 - maxlength 32000 length 31000
small - maxlength 4096 length 4096 The default error
handler has
+been invoked.
An attempt has been made to extend the stack a negative amount.
Referencing address 7FE50000x.
Error occurred in procedure test_overflow, line 14.
Command was test_overflow.pm.
---

BUT -- This isn't an overflow of the buffer. The char varying length is
stored in a fixed bin(15), so 31000 + 4096 is out of range. It's
clearer when the program is compiled with -fixedoverflow. The error
changes to:

---
test_overflow

big1 - maxlength 32000 length 31000
small - maxlength 4096 length 4096 The default error
handler has
+been invoked.
The magnitude of a fixed-point value is too large.
Referencing address 7FE50000x.
Error occurred in procedure test_overflow, line 14.
Command was test_overflow.pm.
The default action is to raise the error condition.
Default handler for error condition invoked.
---

If you reduce the first line of code to
big1 = copy ('1234567890', 2800); /* 28000 bytes long */

The program runs correctly, truncating to the specified length.

test_overflow

big1 - maxlength 32000 length 28000
small - maxlength 4096 length 4096
big2 - maxlength 32000 length 32000

Note that a buffer overrun to 32096 is possible, but didn't happen.

Steve Ketcham
DRA

In article <steve-A63998.22344517012008@comcast.dca.giganews.com>,
steve@somewhere.invalid says...
> In article <478fe2a2$0$5009$4c368faf@roadrunner.com>,
> Peter Flass <Peter_Flass@Yahoo.com> wrote:
>
> >
> > > "declare big1 char(32000)var;
> > > "declare big2 char(32000)var;
> > > "declare small char(4096)var;
> > >
> > > " big1 = copy ('1234567890', 3100); /* 31000 bytes long */
> > > " small = big; /* should truncate at 4096 */
> > > " big2 = small || big1; /* should concatenate and truncate at 32000, */
> > > " /* but overflows the string instead even with */
> > > " /* -check enabled */ "
> > >
>
> I did run it.
>
> The program fails, but the problem is NOT a buffer overflow. (I ran on
> a V-something, running 15.3.0ae. I don't have access to a VOS 16 system
> with PL/I.)
>
> First, here's the code I actually used:
>
> ---
> test_overflow: proc options(main);
>
> declare big1 char(32000)var;
> declare big2 char(32000)var;
> declare small char(4096)var;
>
> big1 = copy ('1234567890', 3100); /* 31000 bytes long */
> put skip list ('big1 - maxlength ', maxlength(big1),
> 'length ', length(big1));
> small = big1; /* should truncate at 4096 */
> put skip list ('small - maxlength ', maxlength(small),
> 'length ', length(small));
>
> big2 = small || big1; /* should concatenate and truncate at 32000, */
> /* but overflows the string instead even with */
> /* -check enabled */
> put skip list ('big2 - maxlength ', maxlength(big2),
> 'length ', length(big2));
>
> end;
> ---
>
> Compile and bind with no options.
>
> The program fails.
>

I tried this under VOS release 16.2. It fails but not due to buffer
oveflow. Rather, due to attempting to extend the stack a negative
amount. Looking at the disassembled code, I see why. The compiler is
adding the length of big1 and small and subtracting the result from the
stack pointer. Since 32000+4096 is larger than 32767, the result is
negative, and the hardware apparently objects. This may be a compiler
bug, let me check...

Steve Ketcham wrote:
> In article <478fe2a2$0$5009$4c368faf@roadrunner.com>,
> Peter Flass <Peter_Flass@Yahoo.com> wrote:
>
>>
>>> "declare big1 char(32000)var;
>>> "declare big2 char(32000)var;
>>> "declare small char(4096)var;
>>>
>>> " big1 = copy ('1234567890', 3100); /* 31000 bytes long */
>>> " small = big; /* should truncate at 4096 */
>>> " big2 = small || big1; /* should concatenate and truncate at 32000, */
>>> " /* but overflows the string instead even with */
>>> " /* -check enabled */ "
>>>
>
> I did run it.
>
> The program fails, but the problem is NOT a buffer overflow. (I ran on
> a V-something, running 15.3.0ae. I don't have access to a VOS 16 system
> with PL/I.)
>
> First, here's the code I actually used:
>
> ---
> test_overflow: proc options(main);
>
> declare big1 char(32000)var;
> declare big2 char(32000)var;
> declare small char(4096)var;
>
> big1 = copy ('1234567890', 3100); /* 31000 bytes long */
> put skip list ('big1 - maxlength ', maxlength(big1),
> 'length ', length(big1));
> small = big1; /* should truncate at 4096 */
> put skip list ('small - maxlength ', maxlength(small),
> 'length ', length(small));
>
> big2 = small || big1; /* should concatenate and truncate at 32000, */
> /* but overflows the string instead even with */
> /* -check enabled */
> put skip list ('big2 - maxlength ', maxlength(big2),
> 'length ', length(big2));
>
> end;
> ---
>
> Compile and bind with no options.
>
> The program fails.
>
> ---
> test_overflow
>
> big1 - maxlength 32000 length 31000
> small - maxlength 4096 length 4096 The default error
> handler has
> +been invoked.
> An attempt has been made to extend the stack a negative amount.
> Referencing address 7FE50000x.
> Error occurred in procedure test_overflow, line 14.
> Command was test_overflow.pm.
> ---
>
> BUT -- This isn't an overflow of the buffer. The char varying length is
> stored in a fixed bin(15), so 31000 + 4096 is out of range. It's
> clearer when the program is compiled with -fixedoverflow. The error
> changes to:
>
> ---
> test_overflow
>
> big1 - maxlength 32000 length 31000
> small - maxlength 4096 length 4096 The default error
> handler has
> +been invoked.
> The magnitude of a fixed-point value is too large.
> Referencing address 7FE50000x.
> Error occurred in procedure test_overflow, line 14.
> Command was test_overflow.pm.
> The default action is to raise the error condition.
> Default handler for error condition invoked.
> ---
>
> If you reduce the first line of code to
> big1 = copy ('1234567890', 2800); /* 28000 bytes long */
>
> The program runs correctly, truncating to the specified length.
>
> test_overflow
>
> big1 - maxlength 32000 length 28000
> small - maxlength 4096 length 4096
> big2 - maxlength 32000 length 32000
>
> Note that a buffer overrun to 32096 is possible, but didn't happen.
>
> Steve Ketcham
> DRA

Here is the output from running the test with Personal PL/I for OS/2:

big1 - maxlength 32000 length 31000
small - maxlength 4096 length 4096
big2 - maxlength 32000 length 32000

(some blanks removed to prevent line wrap).

Your problem is that some implementations specify that it is the
programmer's responsibility to ensure that intermediate results in
string expressions do not exceed 32767 bytes in length; if this limit is
violated the result is undefined. This will not raise STRINGSIZE since
no assignment to a string variable is involved. This seems to be the
case for your complier. You can get the original version to work by
changing the assignment to big2 to

big2=big1||substr(small,1,min(length(small),32767-length(big1)));

Other implementations, such as Personal PL/I can cope with intermediate
results which are (or at least try to be) longer that the 32767
implementation limit on string length.

I added the following three lines to the program:

big2=substr(big1||small,4097);
put skip list('big2 - maxlength ',maxlength(big2),
'length ',length(big2));
put skip list(substr(big2,26900,10));

Here is the resulting output:

big2 - maxlength 32000 length 28671
6789012345

The last line is the last 5 bytes of big1 followed by the first 5 bytes
of small.

STRINGSIZE was enabled during the run and it was raised twice: once on
the assignment to small and a second time on the first assignment to
big2. The result of the second assignment to big2 shows that the
intermediate result was truncated to 32767 bytes since 32767-4096=28671;
however STRINGSIZE was not raised on this truncation.

"James J. Weinkam" <jjw@cs.sfu.ca> wrote in message news:Ziwjj.4825$vp3.2341@edtnps90...
> robin wrote:
> >
> > But if STRINGSIZE is not enabled, storage can be
> > corrupted. There is another section of the manual
> > that is explicit about the conditions under which this occurs
> > (but I don't have the manual at hand to quote).
> >
> PL/I string assignment semantics specify that if the source of a string
> assignment is longer than the (maximum) length of the target, STRINGSIZE
> is raised if enabled. Upon normal return or if STRINGSIZE is disabled,
> the source, truncated to the (maximum) length of the target, is
> assigned. Thus storage corruption can never occur simply because
> STRINGSIZE would have been raised if enabled. Also direct assignment to
> a string element variable can never cause storage corruption.

You need to read the manual.
Storage corruption can occur under those conditions
if STRINGSIZE is not enabled.

"If STRINGSIZE is disabled, and the length of the source and/or target
"is determined at run time, and the target is too short to contain the
"source, UNPREDICTABLE RESULTS CAN OCCUR." [emphasis added]
p. 20, SC-27-1460.

Why didn't you read the manual, like I invited?

> However if the assignment is to a SUBSTR pseudo variable, storage
> corruption could occur if STRINGRANGE is not enabled. Upon normal return
> from STRINGRANGE, the SUBSTR reference is forced to lie within the
> string specified in the first argument. Thus enabling STRINGRANGE can
> prevent storage corruption caused by invalid SUBSTR pseudo variable indices.

As expected, but I was not talking about STRINGRANGE.

robin wrote:
> "James J. Weinkam" <jjw@cs.sfu.ca> wrote in message news:Ziwjj.4825$vp3.2341@edtnps90...
>> robin wrote:
>>> But if STRINGSIZE is not enabled, storage can be
>>> corrupted. There is another section of the manual
>>> that is explicit about the conditions under which this occurs
>>> (but I don't have the manual at hand to quote).
>>>
>> PL/I string assignment semantics specify that if the source of a string
>> assignment is longer than the (maximum) length of the target, STRINGSIZE
>> is raised if enabled. Upon normal return or if STRINGSIZE is disabled,
>> the source, truncated to the (maximum) length of the target, is
>> assigned. Thus storage corruption can never occur simply because
>> STRINGSIZE would have been raised if enabled. Also direct assignment to
>> a string element variable can never cause storage corruption.
>
> You need to read the manual.
> Storage corruption can occur under those conditions
> if STRINGSIZE is not enabled.
>
> "If STRINGSIZE is disabled, and the length of the source and/or target
> "is determined at run time, and the target is too short to contain the
> "source, UNPREDICTABLE RESULTS CAN OCCUR." [emphasis added]
> p. 20, SC-27-1460.

A Google search on "SC-27-1460" or "SC27-1460" turns up nothing. In
contrast, a Google search on the manual reference that I cite below,
"GC28-8204," turns up numerous references to the IBM PL/I LRM. How do
you explain this?
>
> Why didn't you read the manual, like I invited?


Why do you assume that I haven't read "the" manual? In fact, I have
read several of them. I will quote from just one to show that these
rules have been around for a long time:

From GC28-8201-4, IBM System/360 Operating System PL/I (F) Language
Reference Manual, December 1972. p. 366 (Section J: Statements,
Assignment statement) ...

4. The following rules apply to string element assignment:

a. The assignment is performed from left to right starting from the
leftmost position.

b.If the target variable is a fixed length string, the expression value
is truncated on the right if it is too long or padded on the right (with
blanks for character string, zeros for bit strings) if the value is too
short. (Note that a string pseudo variable is considered to be a fixed
length string). The resulting value is assigned to the target.

c. If the target is a VARYING string and the value of the expression is
longer than the maximum length declared for the variable, the value is
truncated on the right. The target string obtains a current length equal
to its maximum length. If the value is not longer than the maximum
length, the value is assigned; the target string obtains a current
length equal to the length of the value.

The Optimizing Compiler LRM (GC33-0009-4) says word for word the same
thing with the addition of the fact that if the expression value is too
long, the STRINGSIZE condition (which was introduced with the Optimizimg
Compiler) will be raised if enabled.

For the Personal PL/I for OS/2 LRM and other more recent compilers the
style of presentation is different. For these products, the LRMs say
that the the value of the expression is converted to the attributes of
the assignment target. For string length conversion the result is the
same as I quoted above, except for the situation you cite, which applies
only if the string lengths are determined at run time rather than being
compile time constants. This situation does not apply to the case under
discussion in this thread. Moreover in the F and Optimizing Compilers,
the conversion rules for string length conversion make no mention of
problems in the case of dynamically determined lengths so the situation
you cite in the later compilers is an unfortunate regression, probably
arising from the fact that the more recent compilers were written by C
programmers who don't really understand PL/I.

I rest my case.

>
>> However if the assignment is to a SUBSTR pseudo variable, storage
>> corruption could occur if STRINGRANGE is not enabled. Upon normal return
>> from STRINGRANGE, the SUBSTR reference is forced to lie within the
>> string specified in the first argument. Thus enabling STRINGRANGE can
>> prevent storage corruption caused by invalid SUBSTR pseudo variable indices.
>
> As expected, but I was not talking about STRINGRANGE.
>
>

On Thu, 17 Jan 2008 20:44:12 GMT, robin wrote:

>>> If you do not have this facility, you need to include code that
>>> not only checks for this condition, but prevents it from
>>> happening.
>>
>> Totally untrue; I have never included such code. I just make sure that all
>> the really important data is stored at the beginning of the string.
>
> No check needed? - This is what you wrote:

Yeah, it was a... oh, never mind.

--
Jay Levitt |
Boston, MA | My character doesn't like it when they
Faster: jay at jay dot fm | cry or shout or hit.
http://www.jay.fm | - Kristoffer