Unit Testing and Test Cases

Here's my understanding as of now.

If I were writing a function

bool IsValidContact(Offerer objOfferer, Accepter objAccepter,
TermsAndConditions objTermsAndConditions);


Before writing the function, I'd enlist all the conditions that must be
met for a contract to be valid. Something along the lines of:

1. There must be a valid offer made by an offerer;
2. There must be an unconditional, voluntary acceptance of the offer;
3. The offerer and accepter must be above the age of minority;
4. There must be a valid consideration (quid pro quo).

Though there are many more conditions, for the sake of simplicity, let
us stick to only the above four conditions.

Going from the above four, I'd translate them into code precondition
checks as follows:


1. Assert (objOfferer != null)
2. Assert (objAccepter.Acceptance.IsVoluntary() &&
objAccepter.Acceptance.IsUnconditional())
3. Assert ((objOfferer.Age >= ObjLawOfMinority.MinimumAge) &&
((objAccepter.Age >= ObjLawOfMinority.MinimumAge))
4. Assert (objTermsAndConditions.Consideration.Value > 0)


These four would be the basis of unit tests after writing code.

In the implementation, I'd use Design By Contract (DBC) style
assertion:

if (objOfferer == null)
return false;

if (SomeOtherCondition Not Met)
return false;



Or I might use the straight-forward-one-return-path style design like,


if (objOfferer)
if (SomeOtherCondition Met)
return true;



After writing the implementation of the above function, I'd write unit
test scripts to test each of the above conditions. These would be
seperate functions that would call the function IsValidContract with
invalid and valid values against each of the above four conditions we
outlined.

Now, let me in on some terminology. Which of the above are test cases
and what is the unit tests here?

> Now, let me in on some terminology. Which of the above are test cases
> and what is the unit tests here?

You are coding a function returning boolean - a predicate.

Your immediate concern is: "how do I know that the function is
correct" ?

A test case is a specification of inputs to and outputs from your
function. If, when supplied these inputs, your function returns
something other than the specified output, then the function is
*incorrect*.

A unit test is code that supplies the inputs, calls the function, and
checks the result against the output specified by the test case.

I think you are confused about preconditions. It is *not* a precondition
on the isValidContract() function that the offerer should be an adult.
Rather, it is the job of the function, when passed a parameter which
denotes a minor, to return FALSE indicating that the contract is not, in
fact, valid.

If I was writing unit tests for this function, I'd probably come up with
something along the following lines:

void testIsValidContract() {
// given these inputs...
offerer = makeMajorOfferer();
acceptance = makeVoluntaryAcceptance();
terms = makeTermsWithValidConsideration();
accepter = makeMinorAccepter(acceptance);
// ...should return false because accepter is a minor
assertFalse( isValid( offerer, acceptance, terms) );
}

(There would be many more such tests, to cover the different cases. How
many tests depends on your approach to test design. The fewer tests you
need to specify for equivalent guarantees of correctness, the better.)

Laurent

Thanks for your diligent reply, Laurent. It has been very helpful and
interesting.


I think I follow you and believe that my understanding has been correct
so far. What still intrigues me is:

1. What then, are pre-conditions?

My initial assessment showed they were conditions to test inside the
function's implementation, before the function returned valid input. If
they're not that, what are they?


2. I understood the unit-test script you wrote called
testIsValidContract. You tested it for one condition/validation, and as
it concurs with my understanding, you imply that it must extend for all
such validations. Fine. But what about test-cases. You said, they are a
specification of what input the function gets and what it spits out. My
question precisely is, "what is the format in which they are written?"
Could you please produce a list of test-cases for this example we've
taken? That would give me an outline.


Thanks.

In article <1141051642.754991.221520{}u72g2000cwu.googlegroup s.com>,
"Water Cooler v2" <wtr_clr{}yahoo.com> wrote:

> Thanks for your diligent reply, Laurent. It has been very helpful and
> interesting.
>
>
> I think I follow you and believe that my understanding has been correct
> so far. What still intrigues me is:
>
> 1. What then, are pre-conditions?
>
> My initial assessment showed they were conditions to test inside the
> function's implementation, before the function returned valid input. If
> they're not that, what are they?

A pre-condition is something about the input such that, if the condition
doesn't hold, the function's result is undefined. In other words, if the
precondition doesn't hold, the function has no way of knowing what to
return. There may be no preconditions in your function.

To put it another way, preconditions are conditions that your function
doesn't have to test, because they must hold *before* the function can
be called.


> 2. I understood the unit-test script you wrote called
> testIsValidContract. You tested it for one condition/validation, and as
> it concurs with my understanding, you imply that it must extend for all
> such validations. Fine. But what about test-cases. You said, they are a
> specification of what input the function gets and what it spits out. My
> question precisely is, "what is the format in which they are written?"
> Could you please produce a list of test-cases for this example we've
> taken? That would give me an outline.

The format for the unit tests depends on the language and test harness
used, but they are written *outside* the function. A test case should
create some input, call the function with that input, then test the
output to see if it is correct.

The tests can also be used by implementers of code that use your
function as examples of proper use.


--
Magic depends on tradition and belief. It does not welcome observation,
nor does it profit by experiment. On the other hand, science is based
on experience; it is open to correction by observation and experiment.

> My initial assessment showed they were conditions to test inside the
> function's implementation, before the function returned valid input.

That's right, as long as you're careful with the implications of that
definition. Preconditions are conditions that must prevail for the
function to have a meaningful result. If preconditions are not met upon
entry, that means someone is using the function wrongly. In other words,
someone wrote a bug, and the bug isn't inside the function.

A major benefit of DbC in my opinion is that it makes it easier, in just
that way, to reason about "where the bug is". A precondition violation
implies a bug in the caller; a postcondition violation imples a bug in
the function called.

Thus,
objOfferer != null
is a precondition. It is incorrect to call the function with a null
offerer.

However,
objTermsAndConditions.Consideration.Value > 0
cannot be a precondition, since you *expect* that the function will
sometimes be called with terms that do not comply with this criterion.
The point of writing the function is that it should tell you when that
is the case, by returning FALSE.

> But what about test-cases. You said, they are a
> specification of what input the function gets and what it spits out. My
> question precisely is, "what is the format in which they are written?"

It depends.

I'm a practitioner of Test-Driven Development. I write down test cases
in the format of unit tests.

So, the testIsValidContract function I sketched is both things:
- a test case (it specifies inputs and outputs)
- an executable unit test

You can also write down test cases in some other format. On paper, or on
the whiteboard (I do that on occasion), in Excel tables... The precise
format depends on what it is you're coding.

Laurent

"Water Cooler v2" <wtr_clr{}yahoo.com> wrote in message
news:1141051642.754991.221520{}u72g2000cwu.googleg roups.com...
>
> 1. What then, are pre-conditions?
>
> My initial assessment showed they were conditions to test inside the
> function's implementation, before the function returned valid input. If
> they're not that, what are they?

There was an argument about this earlier in the newsgroup, so I'm going
to qualify my answer with "this is my opinion; not everyone agrees with it":

A pre-condition is a condition which must be true for your function to
behave correctly. If the pre-condition is not true, then there is no
guarantee as to how the function will behave.

When you're implementing a function, the pre-conditions are things which
you can assume to be true. When calling a function, the pre-conditions are
things which you must ensure are true in order for the function you're
calling to actually be useful.

Here's an example with trivial (perhaps satirical) pre-conditions:

/*
Pre-conditions:
(*) God will not change the laws of logic.
(*) a gamma ray will not strike the computer, flipping bits randomly.
(*) the compiler will not mess up the translation of the function into
machine code.
*/
public boolean getFalse() {
return false;
}

Here's a somewhat more practical example:

/*
Pre-conditions:
(*) x is not zero
*/
public double getMultiplicativeInverse(double x) {
return 1.0 / x;
}

- Oliver

Water Cooler v2 wrote:

> Thanks for your diligent reply, Laurent. It has been very helpful and
> interesting.
>
> I think I follow you and believe that my understanding has been correct
> so far. What still intrigues me is:
>
> 1. What then, are pre-conditions?
>
> My initial assessment showed they were conditions to test inside the
> function's implementation, before the function returned valid input. If
> they're not that, what are they?

They are the client's part of the routine's contract.

It is up to the client's to ensure a routine precondition
*before* a call is attempted (however, a wise supplier
should protect itself from erroneous preconditions, as
explained below).

In return, the client will be ensured *after* its
execution - if the routine is correct - that
the routine's postconditions will hold.

Another very important assertion in the OO world
are class invariants. Invariants are conditions required
to hold immediately before and immediately after any
use of an object (those periods are named: object
stable times).

Together these three assertions, if used correctly,
allow the programmer to express the (approximate)
semantics of the object's Abstract Data Type.

Ideally all those assertions should be ensured
statically (at compile time), situation in which
the program would be correct (regarding
those assertions).

However, such ideal and desirable goal is not
possible for most practical programs.
One is then faced to what to do when there
is the possibility (hopefully, very remote)
of a run-time assertion failure.

We can identify four possible approaches to this
problematic run-time situation:

1. Ostrich approach: Ignore the problem.

2. Defensive programming approach: convert the
problem to the normal program control flow (if's).

3. "All-or-nothing" approach: decorate the program
with C like "asserts" in the place of runnable assertions,
ensuring that either the program meets all of its
assertions at run-time, or else terminates with
an appropriate error message.

4. DbC approach: make all those assertions part
of the programming language and relate them
with the exception mechanism. A failure of
an assertion raises an exception.

Only the last two approaches are appropriate
to properly handle run-time contracts.

BTW, it makes no practical sense to discuss what
postcondition should a routine "ensure" when a
precondition fails. When a precondition fails we
are in the presence of a program error, of the
responsibility of the caller. The routine should
never be executed in those cases.

Nevertheless, the routine *implementation* should
always assume the the precondition holds (and, after
the routine execution, the caller's code should always
assume the routine's postcondition holds).

Class assertions are not part of the class implementation,
they are part of its interface (and it is there that they should
be tested, whenever it is possible and there is no static
guarantee that they are always verified).

(Being part of the class interface they are required
to be properly inherited by descendant classes: a
very difficult behavior to ensure in C++ or in
standard Java).

When a run-time assertion fails, the (wise) programmer(s)
should only take one of two paths:

1. The program error should be identified and
corrected to ensure its inexistence next time
the program is to be executed ("all-or-nothing",
or DbC approaches);

2. In critical applications, an appropriate safe fault tolerant
technique should be devised and implemented within the
program itself (DbC approach).

> (...)
>
> Thanks.

Best regards,

-miguel

--
Miguel Oliveira e Silva

In article <44035117.A32AC86E{}det.ua.pt>,
Miguel Oliveira e Silva <mos{}det.ua.pt> wrote:

> Water Cooler v2 wrote:
>
> > Thanks for your diligent reply, Laurent. It has been very helpful and
> > interesting.
> >
> > I think I follow you and believe that my understanding has been correct
> > so far. What still intrigues me is:
> >
> > 1. What then, are pre-conditions?
> >
> > My initial assessment showed they were conditions to test inside the
> > function's implementation, before the function returned valid input. If
> > they're not that, what are they?
>
> They are the client's part of the routine's contract.
>
> It is up to the client's to ensure a routine precondition
> *before* a call is attempted (however, a wise supplier
> should protect itself from erroneous preconditions, as
> explained below).

A nice write up except for this one problem. If the supplier is able to
protect itself from erroneous inputs, then it can define proper outputs
for those inputs, thus making them no longer erroneous.

Someone else's example:

double inverse( double x ) {
return 1/x;
}

The precondition of course is that 'x != 0' however, thats an easy thing
to check, and as such one can define a particular output for that
condition:

// returns NaN if x == 0
double inverse( double x ) {
return 1/x;
}

On the other hand, some functions have preconditions that can't be
checked. For example C++'s std::copy takes two input iterators that must
be in a range (ie incrementing the first iterator must eventually make
it equal to the second.) This cannot be checked from inside the routine,
even theoretically. Now that's a *real* precondition.

> In return, the client will be ensured *after* its
> execution - if the routine is correct - that
> the routine's postconditions will hold.
>
> Another very important assertion in the OO world
> are class invariants. Invariants are conditions required
> to hold immediately before and immediately after any
> use of an object (those periods are named: object
> stable times).
>
> Together these three assertions, if used correctly,
> allow the programmer to express the (approximate)
> semantics of the object's Abstract Data Type.
>
> Ideally all those assertions should be ensured
> statically (at compile time), situation in which
> the program would be correct (regarding
> those assertions).
>
> However, such ideal and desirable goal is not
> possible for most practical programs.
> One is then faced to what to do when there
> is the possibility (hopefully, very remote)
> of a run-time assertion failure.
>
> We can identify four possible approaches to this
> problematic run-time situation:
>
> 1. Ostrich approach: Ignore the problem.
>
> 2. Defensive programming approach: convert the
> problem to the normal program control flow (if's).
>
> 3. "All-or-nothing" approach: decorate the program
> with C like "asserts" in the place of runnable assertions,
> ensuring that either the program meets all of its
> assertions at run-time, or else terminates with
> an appropriate error message.
>
> 4. DbC approach: make all those assertions part
> of the programming language and relate them
> with the exception mechanism. A failure of
> an assertion raises an exception.
>
> Only the last two approaches are appropriate
> to properly handle run-time contracts.

For real preconditions, none of the above approaches will work. You
can't ignore the problem, and you can't write code to protect yourself
from it.


--
Magic depends on tradition and belief. It does not welcome observation,
nor does it profit by experiment. On the other hand, science is based
on experience; it is open to correction by observation and experiment.

"Daniel T." <postmaster{}earthlink.net> wrote in message
newsostmaster-B27730.19331527022006{}news.east.earthlink.net...
> In article <44035117.A32AC86E{}det.ua.pt>,
> Miguel Oliveira e Silva <mos{}det.ua.pt> wrote:
>
>> Water Cooler v2 wrote:
>>
>> > Thanks for your diligent reply, Laurent. It has been very helpful and
>> > interesting.
>> >
>> > I think I follow you and believe that my understanding has been correct
>> > so far. What still intrigues me is:
>> >
>> > 1. What then, are pre-conditions?
>> >
>> > My initial assessment showed they were conditions to test inside the
>> > function's implementation, before the function returned valid input. If
>> > they're not that, what are they?
>>
>> They are the client's part of the routine's contract.
>>
>> It is up to the client's to ensure a routine precondition
>> *before* a call is attempted (however, a wise supplier
>> should protect itself from erroneous preconditions, as
>> explained below).
>
> A nice write up except for this one problem. If the supplier is able to
> protect itself from erroneous inputs, then it can define proper outputs
> for those inputs, thus making them no longer erroneous.

Right, except keep in mind that sometimes the person designing the
contract and the person implementing the code are two different people. That
latter person may not have the authority to change the contract.

One reason you might not want to make certain inputs no longer erroneous
is to give you flexibility to change your implementation later on.

>
> Someone else's example:
>
> double inverse( double x ) {
> return 1/x;
> }
>
> The precondition of course is that 'x != 0' however, thats an easy thing
> to check, and as such one can define a particular output for that
> condition:
>
> // returns NaN if x == 0
> double inverse( double x ) {
> return 1/x;
> }

In another thread, someone mentioned that in some situations (e.g.
games) it might be worth treating 0 instead as a tiny positive non-zero
value; so the implementation might become:

double inverse(double x) {
if (x == 0) {
x = 0.000001;
}
return 1/x;
}

By not prematurely tightening your contract, you leave yourself open to
make changes like these, whereas if you enforced the behaviour that if x =
0, NaN is returned, then you wouldn't have this freedom.

Of course, if the desired behaviour is to have NaN returned, then by all
means, specify it in the contract! But don't make the contract overly
restrictive if the calling client doesn't care about those corner cases, or
can ensure that those corner cases will never happen, or else you're just
making the implementor's job more difficult for no reason.

>
> On the other hand, some functions have preconditions that can't be
> checked. For example C++'s std::copy takes two input iterators that must
> be in a range (ie incrementing the first iterator must eventually make
> it equal to the second.) This cannot be checked from inside the routine,
> even theoretically. Now that's a *real* precondition.
>

Agreed.

[snip: various approaches to dealing with pre-conditions which do not hold]

>
> For real preconditions, none of the above approaches will work. You
> can't ignore the problem, and you can't write code to protect yourself
> from it.

I'd phrase it as "you (the implementor) are forced to ignore the
problem", while the calling client is forced to cross their fingers and hope
the problem never comes up. This is often the case when there are
uncomputable pre-conditions on the input, but the input comes from the user
via the command-line or a GUI or something similar, e.g.:

/*
pre-condition: String describes valid C source code for a program that
eventually halts.
*/
String void determineOutput(String programSourceCodeInC) {
/*Implementation of a C interpreter goes here*/
}

> --
> Magic depends on tradition and belief. It does not welcome observation,
> nor does it profit by experiment. On the other hand, science is based
> on experience; it is open to correction by observation and experiment.

OT, but I thought that magic is a lot closer to science than to
religion. That is, if you do something using magic (drink a potion, chant an
incantation, wave a wand, etc.) exactly the same way, you will get exactly
the same results.

If you do something using religion (pray, sacrifice an animal, etc.)
exactly the same way, you will get different results, depending on the moods
of the deities involved.

That is, science is all about independently repeatable experiments, and
if magic were real, you could perform independently repeatable experiments
in magic as well.

As a magician, you may or may not want to share the results of your
magical experiments with others, but that would be a characteristic of each
particular magician, and not of magic as a whole (similarly, some scientists
don't share their results, and some do).

- Oliver

"Daniel T." wrote:

> In article <44035117.A32AC86E{}det.ua.pt>,
> Miguel Oliveira e Silva <mos{}det.ua.pt> wrote:
>
> > Water Cooler v2 wrote:
> >
> > > Thanks for your diligent reply, Laurent. It has been very helpful and
> > > interesting.
> > >
> > > I think I follow you and believe that my understanding has been correct
> > > so far. What still intrigues me is:
> > >
> > > 1. What then, are pre-conditions?
> > >
> > > My initial assessment showed they were conditions to test inside the
> > > function's implementation, before the function returned valid input. If
> > > they're not that, what are they?
> >
> > They are the client's part of the routine's contract.
> >
> > It is up to the client's to ensure a routine precondition
> > *before* a call is attempted (however, a wise supplier
> > should protect itself from erroneous preconditions, as
> > explained below).
>
> A nice write up except for this one problem. If the supplier is able to
> protect itself from erroneous inputs, then it can define proper outputs
> for those inputs, thus making them no longer erroneous.

That is not correct.

Most electronic circuits - for example the components used
to build computers - protected themselves against incorrect
connections (through appropriate interface physical shapes)
and even against out of bound voltage signals (through
appropriate protecting circuits). That protection is *not part*
of the normal component behavior, neither is part of the
component's contract to the outside world: it is simply
a way for the component to protect itself against *incorrect*
uses (thus increasing its fault tolerance and life span).
Nothing - other than malfunction - is expected from an
electronic circuit when it is incorrectly used (thus one
cannot "define proper outputs for those inputs" as you
said). Nevertheless, a wise electronic engineer (eager
to keep its job) protects, as much as reasonable, its
circuits against incorrect external uses.

Same thing with assertions.

It seems to me, that you are incorrectly mixing the world
of normal program behavior, and the world of exceptions
and exceptional behavior. Contracts aim at correctness
in the normal program behavior. Exceptions exist as a
mean to deal with (detectable) incorrect programs
(in DbC it is as simply as this).

> Someone else's example:
>
> double inverse( double x ) {
> return 1/x;
> }
>
> The precondition of course is that 'x != 0'

It should be part of the function interface as in Eiffel:

inverse(x: DOUBLE): DOUBLE is
require -- precondition
x /= 0
do
Result := 1/x
ensure -- postcondition
(Result * x - 1).abs <= Maximum_acceptable_near_zero_value
end;

> however, thats an easy thing
> to check,

Indeed it is.

> and as such one can define a particular output for that
> condition:
>
> // returns NaN if x == 0
> double inverse( double x ) {
> return 1/x;
> }

You surely can, but you surely shouldn't (or else
you will be doing defensive programming and not
DbC).

That said, of course the programmer is free to
establish the contracts we wants to its classes
and routines.

Of course, in this particular (mathematical) case
it would not be wise to remove the precondition
(and extend the postcondition with a NaN result
value when x equals zero).

I don't see the interest of dividing anything by zero.
There are some very important mathematical and
engineering uses for "extracting" square roots out
of negative values, but none to my knowledge out
of x/0.

99.99...99% of 1/0 uses are programming errors
(and should be treated as such).

> On the other hand, some functions have preconditions that can't be
> checked.

Yes, that situation can occur (the world is not perfect).
Manytimes times we can only make a practical
runnable approximation of the required precondition.

Why do you think that, somehow, this practical limitation
contradicts anything that I have written (the "all-or-nothing"
and the DbC's exceptional behavior was clearly identified as
applying to *runnable* assertions)?

Are you, by any chance, defending that a program should
not have runnable preconditions?

If you are, then that is not DbC: it is defensive programming.

> For example C++'s std::copy takes two input iterators that must
> be in a range (ie incrementing the first iterator must eventually make
> it equal to the second.) This cannot be checked from inside the routine,
> even theoretically.

So?

Are you saying that we should not use iterators unwisely
or in critical code? If you are, I agree (at least the C++'s
iterators you refer).

Of course there can exist situations in which
we cannot express runnable assertions. However,
as in the example you give, many times the
programmer is free to choose the library that
better fills its needs (why should he use a blind
copy of unsafe iterators?).

> Now that's a *real* precondition.

What do you mean by "real"?

(Surely you are not stating that runnable
preconditions are not real!)

A better name would be hard or difficult.

> > In return, the client will be ensured *after* its
> > execution - if the routine is correct - that
> > the routine's postconditions will hold.
> >
> > Another very important assertion in the OO world
> > are class invariants. Invariants are conditions required
> > to hold immediately before and immediately after any
> > use of an object (those periods are named: object
> > stable times).
> >
> > Together these three assertions, if used correctly,
> > allow the programmer to express the (approximate)
> > semantics of the object's Abstract Data Type.
> >
> > Ideally all those assertions should be ensured
> > statically (at compile time), situation in which
> > the program would be correct (regarding
> > those assertions).
> >
> > However, such ideal and desirable goal is not
> > possible for most practical programs.
> > One is then faced to what to do when there
> > is the possibility (hopefully, very remote)
> > of a run-time assertion failure.
> >
> > We can identify four possible approaches to this
> > problematic run-time situation:
> >
> > 1. Ostrich approach: Ignore the problem.
> >
> > 2. Defensive programming approach: convert the
> > problem to the normal program control flow (if's).
> >
> > 3. "All-or-nothing" approach: decorate the program
> > with C like "asserts" in the place of runnable assertions,
> > ensuring that either the program meets all of its
> > assertions at run-time, or else terminates with
> > an appropriate error message.
> >
> > 4. DbC approach: make all those assertions part
> > of the programming language and relate them
> > with the exception mechanism. A failure of
> > an assertion raises an exception.
> >
> > Only the last two approaches are appropriate
> > to properly handle run-time contracts.
>
> For real preconditions, none of the above approaches will work.

("Real"?)

There are three kinds of assertions:

1. Those that can be ensured statically (the ideal goal);

2. Those that can be checked at run-time (the practical
compromise);

3. All the remaining (the "comment" ones).

A wise programmer attempts to put assertions, as much as
is possible (and reasonable), in the group 1; then he attempts
to put the remaining in the group 2 (even if only some
practical approximations); only then, he should use
comments to informally express the remaining ones
(this last group of assertions will only serve as a
documentation aid).

> You can't ignore the problem, and you can't write code to protect yourself
> from it.

Perhaps not completely (sometimes), but some protection is surely
possible. Even if only *after* the problem as manifest itself.
(The world of runnable assertions is not bound only to preconditions.)

Of course it will be much harder to establish clear responsibilities
between the caller and the callee and to bring objects to stable
usable states (verifying its invariants) if the (precondition) error
is found after the routine starts its internal execution.

All the problems you identify do exist in the practical use
of DbC, but I fail to understand in what way such problems
contradicts anything that I have stated about DbC.

> --
> Magic depends on tradition and belief. It does not welcome observation,
> nor does it profit by experiment. On the other hand, science is based
> on experience; it is open to correction by observation and experiment.

Best regards,

-miguel

--
Miguel Oliveira e Silva