Requesting critique of a C unit test environment

[First, I apologize for cross-posting. I just think a wider audience can
critique from different vantage points.]

Unit testing is an integral component of both "formal" and "agile"
models of development. Alas, it involves a significant amount of tedious
labor.

There are test automation tools out there but from what limited exposure
I've had, they are pricey, reasonably buggy, and require compiler/target
adaptation.

Out of my frustration with two out of two of them came my own. Its
instrumentation approach is based solely on profound abuse of the C
preprocessor (and in this respect it is equally applicable to C++).

I would like to ask to evaluate the approach
- whether it has gaping holes in ideology or implementation
- whether in your opinion it has merits

A preliminary draft description is at
http://www.macroexpressions.com/dl/C...shoestring.pdf

A reference implementation (with a C99 accent) with a runnable example is at
http://www.macroexpressions.com/dl/maestra.zip

Please reply to a newsgroup or via email as you find convenient.
Thank you for your anticipated feedback,

-- Ark

Ark Khasin wrote:
> [First, I apologize for cross-posting. I just think a wider audience can
> critique from different vantage points.]
>
> Unit testing is an integral component of both "formal" and "agile"
> models of development. Alas, it involves a significant amount of tedious
> labor.
>
> There are test automation tools out there but from what limited exposure
> I've had, they are pricey, reasonably buggy, and require compiler/target
> adaptation.
>
> Out of my frustration with two out of two of them came my own. Its
> instrumentation approach is based solely on profound abuse of the C
> preprocessor (and in this respect it is equally applicable to C++).
>
> I would like to ask to evaluate the approach
> - whether it has gaping holes in ideology or implementation
> - whether in your opinion it has merits
>
Why not just use one of the free frameworks such as CppUnit?

It works well with both C (with a little fiddling like you do in your
paper for "static") and C++. I'm sure the same applies for other
frameworks.

--
Ian Collins.

Ian Collins wrote:
<snip>
> Why not just use one of the free frameworks such as CppUnit?
>
> It works well with both C (with a little fiddling like you do in your
> paper for "static") and C++. I'm sure the same applies for other
> frameworks.
>
Ian,
Thank you for your response.

Please correct me if I am wrong, but AFAIK CppUnit doesn't provide a
code execution trace, so it's pretty darn hard to prove code coverage.
[There must be reasons why testing tools vendors command big money.]

Also, if I use C in non-C++ compatible way (e.g. tentative definitions),
my source won't even compile for CppUnit.

And finally there is a port issue (it's an embedded type talking ). I
am proposing something that requires only the compiler.

Regards,
Ark

Ark Khasin wrote:
> Ian Collins wrote:
> <snip>
>> Why not just use one of the free frameworks such as CppUnit?
>>
>> It works well with both C (with a little fiddling like you do in your
>> paper for "static") and C++. I'm sure the same applies for other
>> frameworks.
>>
> Ian,
> Thank you for your response.
>
> Please correct me if I am wrong, but AFAIK CppUnit doesn't provide a
> code execution trace, so it's pretty darn hard to prove code coverage.
> [There must be reasons why testing tools vendors command big money.]
>
If you develop your software test first, you get all the code coverage
you need.

> Also, if I use C in non-C++ compatible way (e.g. tentative definitions),
> my source won't even compile for CppUnit.
>
If you mean K&R style prototypes, don't use them. Write and compile
your tests in C++ and your code in C. Don't attempt to compile your C
with a C++ compiler.

> And finally there is a port issue (it's an embedded type talking ). I
> am proposing something that requires only the compiler.
>
Shouldn't matter for unit testing, develop and test on a hosted system.
If you require bits of the target environment, mock (simulate) them.

--
Ian Collins.

Ian Collins wrote:
<snip>
>> Please correct me if I am wrong, but AFAIK CppUnit doesn't provide a
>> code execution trace, so it's pretty darn hard to prove code coverage.
>> [There must be reasons why testing tools vendors command big money.]
>>
> If you develop your software test first, you get all the code coverage
> you need.
>
Test first is a nice model but not of a universal applicability.
Besides, I need to demonstrate test coverage to the certifying/auditing
entity.
OTOH, I wonder if the proposed instrumentation can be made a part of
CppUnit. I think, there is nothing in either that would prohibit it.
>> Also, if I use C in non-C++ compatible way (e.g. tentative definitions),
>> my source won't even compile for CppUnit.
>>
<snip>
> Write and compile
> your tests in C++ and your code in C. Don't attempt to compile your C
> with a C++ compiler.
Right. It just didn't occur to me
>
>> And finally there is a port issue (it's an embedded type talking ). I
>> am proposing something that requires only the compiler.
>>
> Shouldn't matter for unit testing, develop and test on a hosted system.
> If you require bits of the target environment, mock (simulate) them.
>
The farthest I can go away from the target is a software simulator of
the instruction set. Same compiler, same version, perhaps, more
"memory". I think I am not alone in this...

-- Ark

Ark Khasin wrote:

> Besides, I need to demonstrate test coverage to the certifying/auditing
> entity.

It sounds like learning what their requirements are and meeting them is more
important than guessing if TDD will incidentally meet their requirements.

> Test first is a nice model but not of a universal applicability.

You can keep the TDD thing a secret...

--
Phlip
http://www.oreilly.com/catalog/9780596510657/
^ assert_xpath
http://tinyurl.com/23tlu5 <-- assert_raise_message

Colin Paul Gloster wrote:

> Testing is not an intgeral component of formal methods intended to
> reduce testing.
>
> Colin Paul Gloster

Why would a formal method intend to reduce a Good Thing??

--
Phlip
http://www.oreilly.com/catalog/9780596510657/
^ assert_xpath
http://tinyurl.com/23tlu5 <-- assert_raise_message

Ark Khasin wrote:
> Ian Collins wrote:
> <snip>
>>> Please correct me if I am wrong, but AFAIK CppUnit doesn't provide a
>>> code execution trace, so it's pretty darn hard to prove code coverage.
>>> [There must be reasons why testing tools vendors command big money.]
>>>
>> If you develop your software test first, you get all the code coverage
>> you need.
>>
> Test first is a nice model but not of a universal applicability.
> Besides, I need to demonstrate test coverage to the certifying/auditing
> entity.

You are not alone in that, I'd suggest you take this to a TDD list for
advice.

>>
>>> And finally there is a port issue (it's an embedded type talking ). I
>>> am proposing something that requires only the compiler.
>>>
>> Shouldn't matter for unit testing, develop and test on a hosted system.
>> If you require bits of the target environment, mock (simulate) them.
>>
> The farthest I can go away from the target is a software simulator of
> the instruction set. Same compiler, same version, perhaps, more
> "memory". I think I am not alone in this...
>
Why?

--
Ian Collins.

On 2007-08-27, Ark Khasin <akhasin@macroexpressions.com> wrote:

|--------------------------------------------------------------------------------------|
|"[..] |
| |
|Unit testing is an integral component of [..] "formal" [..] |
|models of development. [..] |
| |
|[..]" |
|--------------------------------------------------------------------------------------|

Testing is not an intgeral component of formal methods intended to
reduce testing.

Regards,
Colin Paul Gloster

On 2007-08-27 09:08, Phlip wrote:
> Colin Paul Gloster wrote:
>
>> Testing is not an intgeral component of formal methods intended to
>> reduce testing.
>>
>> Colin Paul Gloster
>
> Why would a formal method intend to reduce a Good Thing??

Testing is used to find errors, while formal methods are used to prove
that there are no errors, at least that's the goal. So if you can prove
that there are no errors why test for them?

--
Erik Wikström