Why doesn't foreach return a value

On Sat, 2 Feb 2008 19:47:13 -0800 (PST),
"tom.rmadilo" <tom.rmadilo@gmail.com> wrote:

> On Feb 1, 5:56 pm, Darren New <d...@san.rr.com> wrote:
>> tom.rmadilo wrote:
>>> Maybe it is more accurate to
>>> say that Tcl evaluation does not return to the script if an
>>> exceptional condition is returned by any command found in that
>>> script.
>> No, it isn't.
>>> The script is destroyed, evaluation stops and control is returned
>>> to the enclosing context.
>> No, it isn't.
> I guess what you guys should do is submit a massive request to update
> the Tcl documentation, because the distinctions I have made appear to
> be present in every case:
> Some commands return a meaningful value:
> gets, if, regexp, etc.

They return a NON-EMPTY value.


> Some commands return a meaningless value:
> foreach, while, for

They return an EMPTY value.

If they returned an uninitialised string pointer, which might be
pointing into some random spot in program memory, or worse, maybe into
some random stop in memory that'll fault when you try to read it, then
THAT would be a meaningless value. But empty vs. non-empty is still
perfectly valid, and hence potentially useful.


As a case in point, [proc] returns an empty value, which by your
definition, is meaningless. Yet this type of construct works, and I use
it myself on occasion as a way of wrapping an initialisation script
being executed at global scope so that it doesn't leave useless global
or namespaced variables hanging around (another use was demonstrated in
my posts about my [keyed] convenience construct):

[proc "" {args} {
foreach arg $args {
... do something with $arg ...
}
}] arg1 arg2 arg3 ,,,

Notice that it's creating a proc with an empty (but still meaningful)
name, and then executing it as a quick "lets process this stuff within
a [proc] context" one-shot command thing. The [proc] returns an empty
string, which is re-interpreted (being substituted into the first word
of the command) as the name of a command, that we happen to have just
finished creating. Note that the empty result value is very different
from, say, a C void return value that is a special case entity.

Further, [foreach] and co. often occur right at the end of a proc, for
which reason I was a little hesitant to suggest letting the inner loops
return value carry through as the default return value. Some people
rely on [foreach] (and friends) returning an empty value, to also make
their proc return an empty value. Although, moving into 8.5 where we
have a decent [return] command, I think that's a practise we need to
try and get away from.

But, the short of it is that these values are only "meaningless" if you
choose to think of them that way. And that's entirely your problem.


> Some commands don't return a value (they don't return to (but abort)
> the script where they were invoked, and maybe cause higher up scripts
> to abort as well, all without returning):
> break, continue, return, error

But that's a fallasy which we've been pounding on for ages. Two of them
return real values, and the other two return empty ones, along with a
flow-control signal. If you invoke them through [catch], you'll get
that return value and flow-control signal just like normal. In
standard code the flow-control signal is handled automatically, to
guide the how and why of that return value (which may itself be
considered irrelevant and generally ignored, and so is set to be
empty mostly symbolically), unless caught and handled differently, but
it's still being returned. And it's still as valid a value as any
other. You could initialise variables with the return value of
[break], by way of a statement like this:
catch {break} initialise-me
So how can you possibly say it doesn't return a value, or does return a
meaningless value. It DOES return an EMPTY value. And how can you say
that's not returning to the script where it was invoked? That's
returning exactly to where it was invoked.

What you need to be thinking, instead, is that the script from where it
was invoked it automatically re-throwing the return value, in the case
where it's not one that's expected (which for regular code is exactly
TCL_OK).

foreach arg $args {
if { $arg eq "foo" } {
break
}
}

Here you have three levels of code.

[foreach] is invoking {if {$arg eq "foo"} break}. It does this by
handing off the script to an evaluator function.

The evaluation breaks the script into commands, and executes each one
in turn. After each command, it checks the return code, and continues
on to the next one only if it's TCL_OK. If it's anything else (in this
case, potentially TCL_BREAK), then it handles it differently.

[if] does the same. It invokes the expression evaluator to handle the
expression, and the script evaluator to handle the body (in this case
just the command [break]). In both cases it checks the return code
first, to see if it has to do anything special, and only then bothers
to look at the actual return value. And in both cases the evaluators
themselves check return codes of anything they hand off to be performed
by something else.

So between the [foreach] command and the [break] command, something
like this happens... (I'm ignoring TCL_ERROR, for example)

- a script evaluator running the [foreach] command
- the [foreach] command is running through a list of values (or two)
- another script evaluator is spawned by [forwach] to handle the body
- the [if] command is being executed by that last script evaluator
- it has spawned an expression evaluator to test the conditional
- the expression evaluator returned true, and TCL_OK for flow control
- [if], happy with that, then spawned another script evaluator
- that script evaluator executes the [break] command
- [break] returns TCL_BREAK, leaving the default blank result value
- the evaluator (from if) wants only TCL_OK, so re-throws the TCL_BREAK
- [if] simply passes back what it was given by its script evaluator
- the next evaluator (from foreach) does exactly the same as the last
- [foreach] now receives the TCL_BREAK with an empty result value.
- it knows how to handle TCL_BREAK; stop and return TCL_OK
- the outermost script evaluator receives TCL_OK, and continues on

As a point of interest, [catch {break} var] looks basically like this:

- a script evaluator running the [catch] command
- [catch] spawns another script evaluator to run the command
- the new script evaluator executes the [break] command
- [break] returns TCL_BREAK, leaving the default blank result value
- the evaluator (from if) wants only TCL_OK, so re-throws the TCL_BREAK
- [catch] handles everything;
- stashes the result value (empty) in "var" (was given here)
- would have set the options dictionary if it were asked to
- converts the TCL_BREAK code into an integer value
- returns the integer value with the return code TCL_OK
- the starting script evaluator continues on

It's a little more complicated than that, since this [catch] is
probably a value being tested or assigned to a variable. But the
routine is basically the same;

- split the command into distinct words
- if a word includes a nested command
- invoke a script evaluator to evaluate that command
- check for TCL_OK, and re-throw anything we don't like
- if it's TCL_OK, substitute the result value back in our place

So at every point, both inside and outside of script evaluators, the
return code is being checked, and generally if it's anything other then
TCL_OK (0), the script evaluator stops what it's doing and throws it
back, ditching the rest of the script fragment. Otherwise, it just
moves onto the next command (if there is one), returning the last
result value with a TCL_OK return code if that was the last one.

And then whatever invoked the script evaluator itself will check the
return code, which will be TCL_OK unless it was re-throwing something
it didn't like, and either acts on it itself (be it TCL_OK, or
TCL_BREAK in a [foreach]), or just re-throws it, often straight back to
another script evaluator.


Now, having written all that, if I hear once more you saying that
[break] and [continue] don't return anything, there's a VERY good
chance that I'll be killfile'ing you.


> Also, the documentation is as confused as I am about return codes,
> saying that they are used to control flow, and have nothing to do with
> the return value.

No. That is absolutely correct. Return codes, being the ones literally
returned from functions at the C level, do indeed have nothing to do
with the return value, which is stored independently within the
interpreter.


> But I'm sorry to report that the commands which don't return a value,
> according to the documentation, all use a similar phrase 'returns a
> code ... which causes' something to happen.

Conceptually, the return code and the response value are considered as
a single return value unit, for the simple reason that you never have
one without the other.

It's impossible not to return a code. There is exactly one type of
function in C which returns, literally, nothing. And it returns void.
In the TCL code, ALL script-level commands, and the evaluators used to
invoke them, return an integer return code, which in all but a couple
very exceptional cases, MUST be checked and if not handled, re-thrown.
The very top-most level will then in turn translate anything other than
a simple TCL_OK return code into an error and quit. This translation
may or may not make use of the interpreters result value, which like
the return code, ALWAYS exists, though as DKF pointed out, it gets set
to an empty value before any tcl-level command is evaluated (presumably
from within the script evaluator prior to dispatching each command).


> The documentation is also annoyingly consistent about what script is
> aborted and where control is directed to continue.

Well, I hope I've cleared some of that up for you. TCL_OK generally
continues on to the next command in a script fragment, anything else
will cause a disruption to the normal flow, generally causing whatever
it is to stop where it is, and return the code (and value) it presently
has back upwards to the previous stack frame.

And I can't imagine any exception to this rule not being clearly
stated, for those exceptions are generally the specific purpose of the
command involved.


> Also, those bytecode substitutions, like totally skip over code when
> replacing break and continue. Probably the bytecode stuff is wrong,
> you can't just skip over all that important returning stuff.

Darren answered the rest of this beautifully. Read it very, very
carefully.


To recap the key points, for anyone still confused;

1) There is ALWAYS both a return CODE, and a return VALUE.

2) The return value, unless changed, will be the empty string.

3) Script level is a virtual concept. It doesn't exist. It is, in
fact, the side-effect of a script evaluator launching commands on your
behalf. This means that things happen between and around TCL commands
which can appear magical when viewed entirely from within TCL scripting.

4) Your script will never see a return code other than TCL_OK (or its
associate value), except in specific cases like [catch], because the
glue I just mentioned in point 3 that makes script commands work, will
have handled it already. This does not mean that it doesn't exist.

5) With most non-TCL_OK return codes, the value will be thrown away and
ignored unless you specifically catch it, again, most often using the
[catch] command. That same glue does however handle TCL_ERROR, which I
have left out of this message for simplicity sake. Where exactly did
you think those great long error messages came from?!?

6) Everything in TCL is a command, everything conforms to these rules,
and there is neither magic nor exceptions. All commands return, unless
TCL has been crashed, because return is the method used to unwind the
stack frame, and the stack frame must be unwound for things to continue.

7) The documentation attempts to extract from this a simpler reality in
which there is magic to handle the parts that you shouldn't need to.
But in participating in this discussion, you attempted to reach beyond
the reality the documentation paints, to the much finer-grained reality
described in the six points above, where there is no magic, just a lot
of rules and recursive machinery that constructs the seemingly complex
magic out of pain-staking consistency and simple repetitive actions.

As the saying goes; if you can't stand the heat, get out of the
fire. Just sit by, obey the documentation, and warm your feet at a
distance.


Fredderic

On Feb 3, 12:25 am, Fredderic <my-name-h...@excite.com> wrote:
> 3) Script level is a virtual concept. It doesn't exist. It is, in
> fact, the side-effect of a script evaluator launching commands on your
> behalf. This means that things happen between and around TCL commands
> which can appear magical when viewed entirely from within TCL scripting.

No, this is your main mistake. The script is central to understanding
now Tcl works. When I say that a command doesn't return. I mean it
doesn't return anywhere, most importantly to the script in which it
was called. There is nothing virtual about a script. Commands which
don't return abort the script in which they were called and may abort
parent scripts as well. Until the abort process comes to an end, no
other command will be invoked. If you reach the toplevel, everything
stops. Exceptional return codes can be caught and allow execution to
continue.

On Feb 3, 12:25 am, Fredderic <my-name-h...@excite.com> wrote:

> Conceptually, the return code and the response value are considered as
> a single return value unit, for the simple reason that you never have
> one without the other.

This is pure insanity. As soon as reality hits you, you start talking
about conceptual reality. This only points out your total lack of
language to account for the actual reality. This actual reality is
that the C code returns an integer (0-4+) called the result. But the C
function themselves can modify, or not, the interp result. They can
also do almost anything else. They can phone home. They can wake me up
in the morning. All of these extra things are called side-effects.
Your insane mind wants to include these into some single return value
unit. But you are just insane. There is no single return value unit.
It is just pure insanity, side effects are not part of a return value,
and the completion code of a Tcl command is not a return value. How
screwed up can you get it? The completion codes 0-4 NEVER, EVER are
returned to the script level. Commands like [catch] can expose the
value of these completion codes, but they don't every show up as the
return value of any command.

On Sun, 3 Feb 2008 00:45:30 -0800 (PST),
"tom.rmadilo" <tom.rmadilo@gmail.com> wrote:

> On Feb 3, 12:25 am, Fredderic <my-name-h...@excite.com> wrote:

>> 3) Script level is a virtual concept. It doesn't exist. It is, in
>> fact, the side-effect of a script evaluator launching commands on
>> your behalf. This means that things happen between and around TCL
>> commands which can appear magical when viewed entirely from within
>> TCL scripting.
> No, this is your main mistake. The script is central to understanding
> now Tcl works. When I say that a command doesn't return. I mean it
> doesn't return anywhere, most importantly to the script in which it
> was called. There is nothing virtual about a script. Commands which
> don't return abort the script in which they were called and may abort
> parent scripts as well. Until the abort process comes to an end, no
> other command will be invoked. If you reach the toplevel, everything
> stops. Exceptional return codes can be caught and allow execution to
> continue.

How did Darren put it again...

*plonk*


>> Conceptually, the return code and the response value are considered
>> as a single return value unit, for the simple reason that you never
>> have one without the other.
> This is pure insanity. As soon as reality hits you, you start talking
> about conceptual reality. This only points out your total lack of
> language to account for the actual reality. This actual reality is
> that the C code returns an integer (0-4+) called the result. But the C
> function themselves can modify, or not, the interp result. They can
> also do almost anything else. They can phone home. They can wake me up
> in the morning. All of these extra things are called side-effects.
> Your insane mind wants to include these into some single return value
> unit. But you are just insane. There is no single return value unit.
> It is just pure insanity, side effects are not part of a return value,
> and the completion code of a Tcl command is not a return value. How
> screwed up can you get it? The completion codes 0-4 NEVER, EVER are
> returned to the script level. Commands like [catch] can expose the
> value of these completion codes, but they don't every show up as the
> return value of any command.

Nah, mate. Insanity was thinking we could expand your view of the
world. You involved yourself in an issue which reaches beyond the
fairly-tale land of TCL scripting, into the much harsher realities of
TCL mechanics, where a script is in fact, as its name quite aptly
implies when applied to this subject, a series of instructions to be
carried out by a greater machine. Why do you think it is called a
"script" at all? Ah hah... You have been saying it all along, but you
have not even noticed! It is a script! " a written version of a play
or other dramatic composition; used in preparing for a performance ",
as my dictionary puts it. The play, is the task you wish to
accomplish. It itself, does not make the play, it merely directly the
players, TCL, in this case, to perform the task on your behalf.

So sing an ode to conceptual realities, sing an ode to non-trivial
units, for these are the wonders of a greater mind, these are the ways
of the magics of TCL. And wonderous these magics may be, their inner
workings are not for you... So go back, little boy. Return to the
land of fairies where you can frolic among the daisies, away from the
harsh realities of infinite reason, while you still can. Go back... Go
back...... Go back............


And in case you missed it the first time... As Darren said,

*plonk*


Fredderic

tom.rmadilo wrote:
> On Feb 2, 8:14 pm, Darren New <d...@san.rr.com> wrote:
>> tom.rmadilo wrote:
>>> I can't wait to start using the return values of all of these
>>> commands, just as soon as you document how to get them.
>> It's documented in the manual pages. And people have told you several
>> times here. Try the [catch] command, for example.
>
> Did I ever say that [catch] doesn't return a value? No! Catch:
>

I think you completely misunderstood what was suggested. The catch
command lets you run a command (including break), catch the return code
and see the return value.

The statement 'set code [catch {break} result]' shows that break does
indeed return something. That "something" is an empty string, but it
*does* return it. You can claim it doesn't return something all you want
but I think simple observation will prove you wrong.

You're writing huge amounts of sometimes inflammatory prose and yet
after all this time I can't quite figure out what point you're trying to
make. Maybe you need to rethink your approach because your signal seems
to be getting lost in all the noise.

tom.rmadilo wrote:
> This actual reality is
> that the C code returns an integer (0-4+) called the result. But the C
> function themselves can modify, or not, the interp result. They can
> also do almost anything else. They can phone home. They can wake me up
> in the morning. All of these extra things are called side-effects.
> Your insane mind wants to include these into some single return value
> unit. But you are just insane. There is no single return value unit.
> It is just pure insanity, side effects are not part of a return value,
> and the completion code of a Tcl command is not a return value. How
> screwed up can you get it? The completion codes 0-4 NEVER, EVER are
> returned to the script level. Commands like [catch] can expose the
> value of these completion codes, but they don't every show up as the
> return value of any command.

Indeed. There is a convention - ignored at your peril - that a Tcl
command implemented in C will return an integer, and store a string
value that a Tcl script will see in the interpreter result. Both
are always present. If the C function doesn't touch the interpreter
result, the result will be the empty string upon return.

Now, when we cross over onto the Tcl side, the language changes.
We speak of a Tcl command returning a code and a value. The return
code is almost always TCL_OK; so often in fact that we frequently
don't mention the 'ok' return in the documentation. The return value
is not the C return value, but the result of whatever side heffects
in the C code cause the interpreter result to be set.

Generally speaking, the return code is used to govern flow of control.
It is most often visible at the Tcl level because of its effect
on the control flow of scripts, but it can be examined directly
using the [catch] command, and can be set directly by means
of [return -code] and [return -options]. And, in fact, a Tcl
procedure can function as a control flow primitive; it can
[return -code break] or [return -code continue] to divert the flow
of control in its caller. From this standpoint, all commands
really are created alike. There is no ineffable magic in [break]
and [continue], snd they do indeed have return values (which
happen to be the empty string). It just so happens that the
looping commands [while], [foreach] and [for] discard those values.
They are visible from [catch], and if you wanted to implement
a [break] with a return value, you could write
[return -level 0 -code break $someValue]. I can't imagine
why you'd want to, unless you also were writing your own
looping command that used the value of a break rather than
discarding it. But nothing in Tcl's infrastructure precludes
it. And of course, the values associated with the 'error'
and 'return' result codes are used routinely.

--
73 de ke9tv/2, Kevin

tom.rmadilo wrote:

>
>
> On Feb 1, 5:56 pm, Darren New <d...@san.rr.com> wrote:
>> tom.rmadilo wrote:
>> > Maybe it is more accurate to
>> > say that Tcl evaluation does not return to the script if an
>> > exceptional condition is returned by any command found in that script.
>>
>> No, it isn't.
>>
>> > The script is destroyed, evaluation stops and control is returned to
>> > the enclosing context.
>>
>> No, it isn't.
>
> I guess what you guys should do is submit a massive request to update
> the Tcl documentation, because the distinctions I have made appear to
> be present in every case:
>
> Some commands return a meaningful value:
> gets, if, regexp, etc.
>
> Some commands return a meaningless value:
> foreach, while, for
>
> Some commands don't return a value (they don't return to (but abort)
> the script where they were invoked, and maybe cause higher up scripts
> to abort as well, all without returning):
> break, continue, return, error
>
> Also, the documentation is as confused as I am about return codes,
> saying that they are used to control flow, and have nothing to do with
> the return value.
>
> But I'm sorry to report that the commands which don't return a value,
> according to the documentation, all use a similar phrase 'returns a
> code ... which causes' something to happen.
>
> The documentation is also annoyingly consistent about what script is
> aborted and where control is directed to continue.
>
> Very sad, I hope you guys can fix all those documentation errors.
> That, and all the code comments, must have been what confused me.
> Also, those bytecode substitutions, like totally skip over code when
> replacing break and continue. Probably the bytecode stuff is wrong,
> you can't just skip over all that important returning stuff.
>
> I can't wait to start using the return values of all of these
> commands, just as soon as you document how to get them. Then I can
> join the confederacy of dunces.

Just a short comment:

PLONK

Arnulf Wiedemann

tom.rmadilo wrote:
>... Maybe it is more accurate to
> say that Tcl evaluation does not return to the script if an
> exceptional condition is returned by any command found in that script.

The accurate description of how things work is found here:

http://www.tcl.tk/man/tcl8.5/TclCmd/return.htm

--
| Don Porter Mathematical and Computational Sciences Division |
| donald.porter@nist.gov Information Technology Laboratory |
| http://math.nist.gov/~DPorter/ NIST |
|_________________________________________________ _____________________|