Requesting critique of a C unit test environment

Erik Wikström said:

<snip>

> Testing is used to find errors, while formal methods are used to prove
> that there are no errors, at least that's the goal. So if you can
> prove that there are no errors why test for them?

"Beware of bugs in the above code; I have only proved it correct, not
tried it." - Donald E Knuth.

--
Richard Heathfield <http://www.cpax.org.uk>
Email: -www. +rjh@
Google users: <http://www.cpax.org.uk/prg/writings/googly.php>
"Usenet is a strange place" - dmr 29 July 1999

Erik Wikström wrote:

> Testing is used to find errors, while formal methods are used to prove
> that there are no errors, at least that's the goal. So if you can prove
> that there are no errors why test for them?

I use testing as a formal method, to prevent errors. I have not tried
the "proof" systems (and please don't try to tell the mathematicians I used
to hang out with that they are really "proofs").

You are describing writing and running tests in isolation from the
development process. Don't do that.

--
Phlip
http://www.oreilly.com/catalog/9780596510657/
^ assert_xpath
http://tinyurl.com/23tlu5 <-- assert_raise_message

Richard Heathfield <rjh@see.sig.invalid> writes:

> Erik Wikström said:
>
> <snip>
>
>> Testing is used to find errors, while formal methods are used to prove
>> that there are no errors, at least that's the goal. So if you can
>> prove that there are no errors why test for them?
>
> "Beware of bugs in the above code; I have only proved it correct, not
> tried it." - Donald E Knuth.

Proof of Correctness depends very much on how and when "correct" is
used. It is a crock of shit in most day to day SW development.

Phlip wrote:

> Erik Wikström wrote:

>>Testing is used to find errors, while formal methods are used to prove
>>that there are no errors, at least that's the goal. So if you can prove
>>that there are no errors why test for them?

> I use testing as a formal method, to prevent errors.

1. As Dijkstra said : a test can only prove the presence of an error, not
their absence.

2. As Myers said : a test is an input with the intent of finding a defect.


Feel free to show how whatever "testing" method you use can ever equate to
a formal method (or actually for that matter address 1 or 2) ...


> You are describing writing and running tests in isolation from the
> development process. Don't do that.

1. There is *nothing* in the text quoted ">>" that states anything about
"writing and running tests" . Therefore you have mis-represented the
posters' words.

2. The act of "writing and running tests in isolation from the development
process" is something that has been done before. Now what was that approach
called ... ah yes ... the test method for the Cleanroom process.

How does whatever "testing" method you use (which we can only assume does
*not* involve "writing and running tests in isolation from the development
process" ) compare to the Cleanroom defect detection/prevention rate ...??


Steven Perryman

Ian Collins wrote, On 27/08/07 08:14:
> Ark Khasin wrote:
>> Ian Collins wrote:

<snip>

>>>> And finally there is a port issue (it's an embedded type talking ). I
>>>> am proposing something that requires only the compiler.
>>>>
>>> Shouldn't matter for unit testing, develop and test on a hosted system.
>>> If you require bits of the target environment, mock (simulate) them.
>>>
>> The farthest I can go away from the target is a software simulator of
>> the instruction set. Same compiler, same version, perhaps, more
>> "memory". I think I am not alone in this...
>>
> Why?

There are many possible *valid* reasons for this. One is that if you are
not using the same version of the same compiler with the same switches
then the code you are testing is not the same as the code that will be
run. Since compilers *do* have bugs it is possible that the bug will be
triggered in the real environment but not in the test environment unless
you ensure that they are the same.

If I was doing QA for a product I would insist than you either use the
same version of the same compiler or you provide testing to the same
level that the deliverable SW requires of *both* the compiler used for
test *and* the compiler used for the final SW. The more critical the SW,
the more insistent I would be on this, and the more testing you would
have to do on the compilers, for safety critical SW this would probably
kill the project dead if you did not use identical SW to build for test
and build for delivery. BTW, I *have* rejected SW and documentation at
review, and even told developers that there was no point in putting it
in for review because I would fail it.
--
Flash Gordon

Richard Heathfield <rjh@see.sig.invalid> writes:

> Erik Wikström said:
>
> <snip>
>
>> Testing is used to find errors, while formal methods are used to prove
>> that there are no errors, at least that's the goal. So if you can
>> prove that there are no errors why test for them?
>
> "Beware of bugs in the above code; I have only proved it correct, not
> tried it." - Donald E Knuth.

But this was a "by hand" proof in 1977. A machine assisted proof of
the actual code could be expected to inspire a little more confidence.

--
Ben.

Ben Bacarisse said:

> Richard Heathfield <rjh@see.sig.invalid> writes:
>
>> Erik Wikström said:
>>
>> <snip>
>>
>>> Testing is used to find errors, while formal methods are used to
>>> prove that there are no errors, at least that's the goal. So if you
>>> can prove that there are no errors why test for them?
>>
>> "Beware of bugs in the above code; I have only proved it correct, not
>> tried it." - Donald E Knuth.
>
> But this was a "by hand" proof in 1977. A machine assisted proof of
> the actual code could be expected to inspire a little more confidence.

Why? Presumably the machine that is doing the assisting is itself a
computer program. What makes you think the assistance program is
correct?

--
Richard Heathfield <http://www.cpax.org.uk>
Email: -www. +rjh@
Google users: <http://www.cpax.org.uk/prg/writings/googly.php>
"Usenet is a strange place" - dmr 29 July 1999

Flash Gordon wrote:
> Ian Collins wrote, On 27/08/07 08:14:
>> Ark Khasin wrote:
>>> Ian Collins wrote:
>
> <snip>
>
>>>>> And finally there is a port issue (it's an embedded type talking
>>>>> ). I
>>>>> am proposing something that requires only the compiler.
>>>>>
>>>> Shouldn't matter for unit testing, develop and test on a hosted system.
>>>> If you require bits of the target environment, mock (simulate) them.
>>>>
>>> The farthest I can go away from the target is a software simulator of
>>> the instruction set. Same compiler, same version, perhaps, more
>>> "memory". I think I am not alone in this...
>>>
>> Why?
>
> There are many possible *valid* reasons for this. One is that if you are
> not using the same version of the same compiler with the same switches
> then the code you are testing is not the same as the code that will be
> run. Since compilers *do* have bugs it is possible that the bug will be
> triggered in the real environment but not in the test environment unless
> you ensure that they are the same.
>
But one has to differentiate between developer unit testing (the subject
of this post) and QA (customer) acceptance testing. The former can be
performed on any environment the developer chooses, the later must be
run on the target.

> If I was doing QA for a product I would insist than you either use the
> same version of the same compiler or you provide testing to the same
> level that the deliverable SW requires of *both* the compiler used for
> test *and* the compiler used for the final SW. The more critical the SW,
> the more insistent I would be on this, and the more testing you would
> have to do on the compilers, for safety critical SW this would probably
> kill the project dead if you did not use identical SW to build for test
> and build for delivery. BTW, I *have* rejected SW and documentation at
> review, and even told developers that there was no point in putting it
> in for review because I would fail it.

Again, this is different from developer unit testing, I don't think
anyone would be daft enough to release a product that hadn't been
through acceptance testing on the target platform.

--
Ian Collins.

Richard Heathfield <rjh@see.sig.invalid> writes:

> Ben Bacarisse said:
>
>> Richard Heathfield <rjh@see.sig.invalid> writes:
>>
>>> Erik Wikström said:
>>>
>>> <snip>
>>>
>>>> Testing is used to find errors, while formal methods are used to
>>>> prove that there are no errors, at least that's the goal. So if you
>>>> can prove that there are no errors why test for them?
>>>
>>> "Beware of bugs in the above code; I have only proved it correct, not
>>> tried it." - Donald E Knuth.
>>
>> But this was a "by hand" proof in 1977. A machine assisted proof of
>> the actual code could be expected to inspire a little more confidence.
>
> Why? Presumably the machine that is doing the assisting is itself a
> computer program. What makes you think the assistance program is
> correct?

What do you test your software with if not more software?

If you think that a machine assisted proof would not inspire "a little
more" confidence than a hand proof, then I won't try to persuade you
(it was a modest enough claim) but the fact that a proof system is
software does not invalidate the method any more than testing is
invalidated by being done in software.

--
Ben.

Ian Collins wrote, On 27/08/07 21:51:
> Flash Gordon wrote:
>> Ian Collins wrote, On 27/08/07 08:14:
>>> Ark Khasin wrote:
>>>> Ian Collins wrote:
>> <snip>
>>
>>>>>> And finally there is a port issue (it's an embedded type talking
>>>>>> ). I
>>>>>> am proposing something that requires only the compiler.
>>>>>>
>>>>> Shouldn't matter for unit testing, develop and test on a hosted system.
>>>>> If you require bits of the target environment, mock (simulate) them.
>>>>>
>>>> The farthest I can go away from the target is a software simulator of
>>>> the instruction set. Same compiler, same version, perhaps, more
>>>> "memory". I think I am not alone in this...
>>>>
>>> Why?
>> There are many possible *valid* reasons for this. One is that if you are
>> not using the same version of the same compiler with the same switches
>> then the code you are testing is not the same as the code that will be
>> run. Since compilers *do* have bugs it is possible that the bug will be
>> triggered in the real environment but not in the test environment unless
>> you ensure that they are the same.
>>
> But one has to differentiate between developer unit testing (the subject
> of this post) and QA (customer) acceptance testing.

You have missed out internal formal testing which in many environments
is far more complete than acceptance testing. For example, I've worked
on projects where a formal test literally takes a week to complete but
the customer acceptance testing takes only a few hours.

Unit test can also be formal, and in a lot of environments, including
the afore mentioned safety critical projects, you are *required* to
perform formal unit tests.

> The former can be
> performed on any environment the developer chooses,

Informal testing can be run in any environment the developer has
available. Formal testing, which is the only sort of testing that you
can guarantee will be available and working for those maintaining later,
is another matter.

> the later must be
> run on the target.

All formal testing, whether unit testing or testing at a higher level
has to be run on code compiled with the correct compiler, although not
always on an identical target.

>> If I was doing QA for a product I would insist than you either use the
>> same version of the same compiler or you provide testing to the same
>> level that the deliverable SW requires of *both* the compiler used for
>> test *and* the compiler used for the final SW. The more critical the SW,
>> the more insistent I would be on this, and the more testing you would
>> have to do on the compilers, for safety critical SW this would probably
>> kill the project dead if you did not use identical SW to build for test
>> and build for delivery. BTW, I *have* rejected SW and documentation at
>> review, and even told developers that there was no point in putting it
>> in for review because I would fail it.
>
> Again, this is different from developer unit testing, I don't think
> anyone would be daft enough to release a product that hadn't been
> through acceptance testing on the target platform.

Acceptance testing has very little to do with proving whether the system
works, it is just to give the customer some confidence. The real worth
while formal testing has to be completed *before* doing customer
acceptance testing and done with the correct compiler. At least, this is
the case in many environments, including all the projects where I have
been involved in QA, and on the safety critical project I was involved in.

If your customer acceptance testing is sufficient to prove the SW is
sufficiently correct then your customer has either very little trust in
your company or a lot of time to waste. If your customer acceptance
testing is the only testing done with the correct compiler and it is not
sufficient to prove your SW is sufficiently correct then your SW is not
tested properly. At least, not according to any standard of testing I
have come across.
--
Flash Gordon