Re: V-Series/VOS 16 porting tips for PL/I, collected

In <F5F42E77A43DD944B6D664B00A5401CB037148A5@EXNA.cor p.stratus.com>, on
01/17/2008
at 01:22 PM, "Paul Green" <Paul.Green@stratus.B0GVS.com> said:

>VOS PL/I does not provide the stringsize condition and there is no
>substitute for it. You don't need it to avoid data corruption,

Yes you do, because unintended truncation is one form of data corruption.

>The bottom line, as now said by many people, is that programmers should
>use the error-catching features of the language and the compiler.

The STRINGSIZE conditioion is one of those.

--
Shmuel (Seymour J.) Metz, SysProg and JOAT <http://patriot.net/~shmuel>

Unsolicited bulk E-mail subject to legal action. I reserve the
right to publicly post or ridicule any abusive E-mail. Reply to
domain Patriot dot net user shmuel+news to contact me. Do not
reply to spamtrap@library.lspace.org

"James J. Weinkam" <jjw@cs.sfu.ca> wrote in message newsmBkj.35640$fj2.30055@edtnps82...
> robin wrote:
> > "James J. Weinkam" <jjw@cs.sfu.ca> wrote in message news:Ziwjj.4825$vp3.2341@edtnps90...
> >> robin wrote:
> >>> But if STRINGSIZE is not enabled, storage can be
> >>> corrupted. There is another section of the manual
> >>> that is explicit about the conditions under which this occurs
> >>> (but I don't have the manual at hand to quote).
> >>>
> >> PL/I string assignment semantics specify that if the source of a string
> >> assignment is longer than the (maximum) length of the target, STRINGSIZE
> >> is raised if enabled. Upon normal return or if STRINGSIZE is disabled,
> >> the source, truncated to the (maximum) length of the target, is
> >> assigned. Thus storage corruption can never occur simply because
> >> STRINGSIZE would have been raised if enabled. Also direct assignment to
> >> a string element variable can never cause storage corruption.
> >
> > You need to read the manual.
> > Storage corruption can occur under those conditions
> > if STRINGSIZE is not enabled.
> >
> > "If STRINGSIZE is disabled, and the length of the source and/or target
> > "is determined at run time, and the target is too short to contain the
> > "source, UNPREDICTABLE RESULTS CAN OCCUR." [emphasis added]
> > p. 20, SC-27-1460.
>
> A Google search on "SC-27-1460" or "SC27-1460" turns up nothing.

Why not read the manual?

> In
> contrast, a Google search on the manual reference that I cite below,
> "GC28-8204," turns up numerous references to the IBM PL/I LRM. How do
> you explain this?
> >
> > Why didn't you read the manual, like I invited?
>
> Why do you assume that I haven't read "the" manual? In fact, I have
> read several of them.

But you missed the section that I quoted.

> I will quote from just one to show that these
> rules have been around for a long time:
>
> From GC28-8201-4, IBM System/360 Operating System PL/I (F) Language
> Reference Manual, December 1972. p. 366 (Section J: Statements,
> Assignment statement) ...
>
> 4. The following rules apply to string element assignment:
>
> a. The assignment is performed from left to right starting from the
> leftmost position.
>
> b.If the target variable is a fixed length string, the expression value
> is truncated on the right if it is too long or padded on the right (with
> blanks for character string, zeros for bit strings) if the value is too
> short. (Note that a string pseudo variable is considered to be a fixed
> length string). The resulting value is assigned to the target.
>
> c. If the target is a VARYING string and the value of the expression is
> longer than the maximum length declared for the variable, the value is
> truncated on the right. The target string obtains a current length equal
> to its maximum length. If the value is not longer than the maximum
> length, the value is assigned; the target string obtains a current
> length equal to the length of the value.
>
> The Optimizing Compiler LRM (GC33-0009-4) says word for word the same
> thing with the addition of the fact that if the expression value is too
> long, the STRINGSIZE condition (which was introduced with the Optimizimg
> Compiler) will be raised if enabled.
>
> For the Personal PL/I for OS/2 LRM and other more recent compilers the
> style of presentation is different. For these products, the LRMs say
> that the the value of the expression is converted to the attributes of
> the assignment target. For string length conversion the result is the
> same as I quoted above, except for the situation you cite, which applies
> only if the string lengths are determined at run time rather than being
> compile time constants. This situation does not apply to the case under
> discussion in this thread. Moreover in the F and Optimizing Compilers,
> the conversion rules for string length conversion make no mention of
> problems in the case of dynamically determined lengths so the situation
> you cite in the later compilers is an unfortunate regression, probably
> arising from the fact that the more recent compilers were written by C
> programmers who don't really understand PL/I.

The first of the new IBM compilers was written principally in PL/I,
so presumably the implementors were fully conversant with
the language.

> >> However if the assignment is to a SUBSTR pseudo variable, storage
> >> corruption could occur if STRINGRANGE is not enabled. Upon normal return
> >> from STRINGRANGE, the SUBSTR reference is forced to lie within the
> >> string specified in the first argument. Thus enabling STRINGRANGE can
> >> prevent storage corruption caused by invalid SUBSTR pseudo variable indices.
> >
> > As expected, but I was not talking about STRINGRANGE.

robin wrote:
>
> But you missed the section that I quoted.
>
I did not miss the section you quoted. It does not apply to the
situation that was being discussed in the thread, although it is
important that programmers using recent compilers be aware of it when it
does apply, namely when some of the lengths are determined at run time
rather than at compile time.

"Paul Green" <Paul.Green@stratus.B0GVS.com> wrote in message
news:F5F42E77A43DD944B6D664B00A5401CB037148A5@EXNA .corp.stratus.com...

> Let me clarify the situation for Stratus VOS customers. I would not
> want to confuse them with information that may be accurate for other
> platforms but not for our platform. VOS PL/I truncates strings at
> assignment time and this eliminates most cases of data corruption.
>
> Stratus provides a PL/I compiler that is very close to the Subset G
> version of ANSI PL/I. The reference is ANSI X3.74-1987 (R1993):
> Information Systems - Programming Language - PL/I General-Purpose Subset.
>
> You can view our manual by pointing the Internet Explorer web browser at
>
> http://stratadoc.stratus.com/
>
> click on the VOS Operating System, pick a release, and pick a platform.
>
> The PL/I manual is filed under "Languages and Subroutines",
> and the PL/I Language manual is number R009.
>
> Or you can click on the following link, if the mail system doesn't mung
> it up too badly (remove any added carriage-returns):
>
http://stratadoc.stratus.com/vos/16....ext=r009-04&fi
le=ch1r009-04.html
>
> VOS PL/I does not provide the stringsize condition and there is no
> substitute for it. You don't need it to avoid data corruption,
> because the compiler always knows the length of the target string,
> and will not write beyond that point. Even in the case where the
> target is a parameter declared "char (*)", the calling program has
> passed a hidden argument that supplies the extent of the string. You
> might want stringsize so you can catch cases of truncation, but this
> feature was omitted from Subset G, presumably in the belief that
> string truncation is the norm, not the exception.

String truncation would not be the norm.
Inadvertant truncation of trailing characters of a string
could be a programming error, and having the ability to
detect when it occurs is an important safeguard.

Such truncation can occur on string assignment,
on output (when the string is longer than the format item).
and on assignment to a parameter whose length is (*), and
on input (when the data string is longer than the receiving variable),
for example.

> VOS PL/I does not provide the stringrange condition. Instead, it
> provides the -check control argument on the pl1 command. When
> -check is used, the compiler adds code to ensure that references to
> the substr() built-in function stay within the declared length of
> the string; if the reference is invalid the code signals the error
> condition. Without -check, the compiled code presumes that the
> programmer knows what she or he is doing and executes the substr
> without checking.
>
> Most of the cases of string corruption that I've seen, or have
> bitten me, have involved strings with expression extents (i.e.,
> strings whose length is an expression). If some variable in the
> expression extent is uninitialized or incorrect, then storage can
> be corrupted or unitialized storage can be referenced. But it is
> unclear to me that stringsize or stringrange would be of much use
> here, because to the compiled code, the length of the string *is*
> the value of the expression. There is no way to know that 1 is ok
> and 32766 is bad. Just catching cases of truncation isn't
> sufficient; the bogus length could still not result in any truncation.
>
> You really need some sort of meta-data that can be used to validate
> the putative extent. That's not practical in a compiled environment.
> Instead, I encourage people to use a different control argument,
> -check_uninitialized, in which the compiler performs additional
> analysis to find variables that are unset in each flow unit. (A
> flow unit is a collection of statements that have a single entry
> and single exit).

That's good. A compiler check for uninitialized data is an important facility.

> The bottom line, as now said by many people, is that programmers
> should use the error-catching features of the language and the
> compiler. The costs associated with encountering latent defects
> far outweigh the slight performance gain that comes from avoiding
> the checking code.

Indeed.

robin wrote:
>
>>VOS PL/I does not provide the stringrange condition.

This is surprising. Since you check lengths to do the truncation on
assignment, how much more difficult would it be to raise STRINGSIZE?
OTOH, it's been this way since Methuzela, and you probably wouldn't wnt
to change it now.

Instead, it
>>provides the -check control argument on the pl1 command. When
>>-check is used, the compiler adds code to ensure that references to
>>the substr() built-in function stay within the declared length of
>>the string; if the reference is invalid the code signals the error
>>condition. Without -check, the compiled code presumes that the
>>programmer knows what she or he is doing and executes the substr
>>without checking.

I don't believe IBM PL/I does this much for SUBSTR, in fact I know it.
It's a nice feature.

>>
>>Most of the cases of string corruption that I've seen, or have
>>bitten me, have involved strings with expression extents (i.e.,
>>strings whose length is an expression). If some variable in the
>>expression extent is uninitialized or incorrect, then storage can
>>be corrupted or unitialized storage can be referenced. But it is
>>unclear to me that stringsize or stringrange would be of much use
>>here, because to the compiled code, the length of the string *is*
>>the value of the expression.

Unless there is some other storage overlay, how does this occur? If I
declare, say, 'A CHAR(i*6)', the expression is evaluated and A is
allocated, then i is never used again for this purpose. Assuming VOS
PL/I has descriptors, all references to A should check against the
length it was allocated with.

Peter Flass wrote:
(snip)

> Unless there is some other storage overlay, how does this occur? If I
> declare, say, 'A CHAR(i*6)', the expression is evaluated and A is
> allocated, then i is never used again for this purpose. Assuming VOS
> PL/I has descriptors, all references to A should check against the
> length it was allocated with.

It I is negative, and it isn't checked, then bad things will
happen. Probably it should always be checked, though.

-- glen

"robin" <robin_v@bigpond.com> wrote in message
news:wKC6k.12673$IK1.9440@news-server.bigpond.net.au...
> "Paul Green" <Paul.Green@stratus.B0GVS.com> wrote in message
> news:F5F42E77A43DD944B6D664B00A5401CB037148A5@EXNA .corp.stratus.com...
>>
>> Stratus provides a PL/I compiler that is very close to the Subset G
>> version of ANSI PL/I. The reference is ANSI X3.74-1987 (R1993):
>> Information Systems - Programming Language - PL/I General-Purpose Subset.
>> [...]
>> VOS PL/I does not provide the stringsize condition and there is no
>> substitute for it. You don't need it to avoid data corruption,
>> because the compiler always knows the length of the target string,
>> and will not write beyond that point. Even in the case where the
>> target is a parameter declared "char (*)", the calling program has
>> passed a hidden argument that supplies the extent of the string. You
>> might want stringsize so you can catch cases of truncation, but this
>> feature was omitted from Subset G, presumably in the belief that
>> string truncation is the norm, not the exception.
>
> String truncation would not be the norm.

I was stating why I believe PL/I Subset G omits the stringrange and
stringsize conditions. But perhaps they had more mundane reasons. Appendix
F of X3.74 states, "The conditions that can be explicitly designated in the
subset were restricted. This was done to reduce implementation complexity
while retaining most of the power of PL/I exception handling."

Subset G also omits the name and subscriptrange conditions. Implementations
are free to add support back in, of course, or provide other methods to
catch these errors, but they aren't required to do so to claim conformance
to this standard.

> Inadvertant truncation of trailing characters of a string
> could be a programming error, and having the ability to
> detect when it occurs is an important safeguard.
>
> Such truncation can occur on string assignment,
> on output (when the string is longer than the format item).
> and on assignment to a parameter whose length is (*), and
> on input (when the data string is longer than the receiving variable),
> for example.

Of course.

"Peter Flass" <Peter_Flass@Yahoo.com> wrote in message
news:485b83e5$0$5738$4c368faf@roadrunner.com...
> wrote:
>>
>>>VOS PL/I does not provide the stringrange condition.
>
> This is surprising. Since you check lengths to do the truncation on
> assignment, how much more difficult would it be to raise STRINGSIZE?
> OTOH, it's been this way since Methuzela, and you probably wouldn't wnt
> to change it now.

I don't know. It has been this way for 28 years and I think this is the
first serious discussion I've ever had about the absence of STRINGSIZE. It
is certainly true that we'd have to be careful about breaking existing code.
Customers tend to assume that if their code runs without generating
exceptions then it must be correct...they sometimes get rather upset when we
add new checks for latent defects. They usually settle down after a while,
when they realize we've actually done them a favor. ;-)

>>>Most of the cases of string corruption that I've seen, or have
>>>bitten me, have involved strings with expression extents (i.e.,
>>>strings whose length is an expression). If some variable in the
>>>expression extent is uninitialized or incorrect, then storage can
>>>be corrupted or unitialized storage can be referenced. But it is
>>>unclear to me that stringsize or stringrange would be of much use
>>>here, because to the compiled code, the length of the string *is*
>>>the value of the expression.
>
> Unless there is some other storage overlay, how does this occur? If I
> declare, say, 'A CHAR(i*6)', the expression is evaluated and A is
> allocated, then i is never used again for this purpose. Assuming VOS
> PL/I has descriptors, all references to A should check against the
> length it was allocated with.

The language requires an expression use to calculate an extent to be valid
any time the extent is referenced. So if you pass "A" by reference, or
assign to A, or take the length of A, etc., then the value of "i*6" must be
valid at each of these points. There is no need for the compiler to
remember the allocation length; the burden is on the source program and the
programmer.

However, the compiler does need to do some work behind the scenes when an
array or string with expression extents is passed as an argument, because it
is not feasible to have separately-compiled code know how to evaluate an
expression in the context of its caller. Therefore, when an argument with
expression-extents is passed to a subroutine or function, our implementation
creates a descriptor (aka "dope-vector") for it. That step effectively
makes a temporary copy of the value "i*6" and passes it as a hidden
argument. The receiving code knows how to find it. If the world was
simple, we'd just pass a descriptor for every argument, which would double
the number of arguments (this is what Multics PL/I did, for example). The
VOS PL/I world is not that simple, but the concept is the same.

I wrote a short test program that demonstrates various types of string
truncation. It passes its self-tests on VOS whether I use the -check option
or not. (Recall that the -check option on VOS PL/I adds code to check for
subscriptrange and stringrange, but not stringsize). See the attached file.

PG

"Peter Flass" <Peter_Flass@Yahoo.com> wrote in message
news:485b83e5$0$5738$4c368faf@roadrunner.com...
> robin wrote:
> >
> >>VOS PL/I does not provide the stringrange condition.
>
> This is surprising. Since you check lengths to do the truncation on
> assignment, how much more difficult would it be to raise STRINGSIZE?
> OTOH, it's been this way since Methuzela, and you probably wouldn't wnt
> to change it now.
>
> Instead, it
> >>provides the -check control argument on the pl1 command. When
> >>-check is used, the compiler adds code to ensure that references to
> >>the substr() built-in function stay within the declared length of
> >>the string; if the reference is invalid the code signals the error
> >>condition. Without -check, the compiled code presumes that the
> >>programmer knows what she or he is doing and executes the substr
> >>without checking.
>
> I don't believe IBM PL/I does this much for SUBSTR, in fact I know it.

Not sure I get your drift.
IBM compilers do check for stringrange with SUBSTR (provided, of course
that STRINGRANGE is enabled).

"glen herrmannsfeldt" <gah@ugcs.caltech.edu> wrote in message
newsY2dndiTm9GSK8bVnZ2dnUVZ_sDinZ2d@comcast.com. ..
> Peter Flass wrote:

> > Unless there is some other storage overlay, how does this occur? If I
> > declare, say, 'A CHAR(i*6)', the expression is evaluated and A is
> > allocated, then i is never used again for this purpose. Assuming VOS
> > PL/I has descriptors, all references to A should check against the
> > length it was allocated with.
>
> It I is negative, and it isn't checked, then bad things will
> happen. Probably it should always be checked, though.

It's something that should be checked automatically.
In the IBM compilers, it should be simple for the compiler
to check by converting to UNSIGNED (now that those compilers
support unsigned), which would then raise the fixedoverflow/size condition.

Of course, the programmer can easily handle it in any system by
using MAX or somesuch [e.g., max(i*6, 1)]
and by subsequently checking the value of i.
Even more prudent is to check the value of i beforehand.

Even MAX by itself is not enough. Need to ensure also that
the final value is less than 32768.