Specs versus Tests

After reading the latest column by Bertrand Meyer
on Seven Principles of Software Testing,
I'm left wondering whether there isn't another useful way
of employing tests. Or whether I'm doing funny things here.

I use (unit) tests to make specifications fail (= be wrong),
not just objects. Specs might be wrong, after all!
For a silly example, ensure 1 > 2 is certainly of limited
use. This use requires human interaction, though.

It means, I do not just try to make the executable
object fail which implements the specification. I also
do it to test the assumptions leading to specifications.
The point of failure is then me, having specified the
wrong things. The oracle, I guess, is at some slightly
different level, but still I can use a computers
for testing. A failed test might be the conclusion
I draw from one or more test routines that actually
succeed. They should actually succeed given the current
specs, but some other test (or any "decision context")
makes *me* think otherwise.

In some cases, both the specs and the test logic might
turn out not to really suite anything. They are both wrong.
At that time I reconsider the *assumptions* that have
led to wrong specs and/or wrong tests.

Tests help me (or my Delphi self) to get the specs right.

Does this make sense? Is there anything I could read?


- Georg

Georg Bauhaus schrieb:
> After reading the latest column by Bertrand Meyer
> on Seven Principles of Software Testing,

Is there any URL. It might help to understand your posting
which i at the moment do not.

scholz.lothar@gmail.com schrieb:
> Georg Bauhaus schrieb:
>> After reading the latest column by Bertrand Meyer
>> on Seven Principles of Software Testing,
>
> Is there any URL. It might help to understand your posting
> which i at the moment do not.
>

ISE sends the column as part of an electronic news letter to
subscribers to EiffelWorld (free). No URL I am aware of.

I'll try to extract a few bits and then rephrase my
question. (As always, there have been a few ambigous relative
pronouns, grammatical mistakes, and other flaws in my
sentences. Sorry.)

<extr see=column>
Meyer asks what software testing really is and how it relates to
specs. After pushing some of the glossy marketing phrases aside,
he fucuses on what the practice of testing can achieve.
What it can achieve, is, with reference to Dijkstra and also
with reference to IEEE speak, finding bugs. Bugs mean all of

"an unsatisfactory program execution is a "failure", pointing
to a "fault" in the program, itself the result of a "mistake"
in the programmer's thinking."

So when you are testing, you try to make a program fail in order
to notice these phenomena. What does it mean in relation to specs?

Stating what might be obvious to some, he points out (as the
second of 7 "principles"), that specifications may be much more
inclusive and precise than a test can be. "X squared" says more
than a list of pairs such as (1, 1), (2, 4), (3, 9), the
latter being a number of test cases.
Hence you can construct test cases from specs but not specs
from test cases.

An oracle is described as the instance that decides whether
a test has succeeded or has failed. DbC specs can serve as
an oracle.
</extr>

When I write tests, run them and watch the results, then
another thing happens: May the tests succeed or fail,
I notice that some of my specs are wrong.

It might seem pointless to run a test against an incorrect spec.
What would you be concluding? What kind of oracle is this?
However, these seemingly pointless tests still help me to
get the assumptions right.

- Have I assumed the correct hypotheses?
- Have I copied the mathematical structure that best
matches my problem?
- Have I reworked the hypotheses and structures into
DbC specs correctly?

In conclusion, can't I use unit testing to see whether
the specs accurately reflect the problem definition?