what is best way to capture customer escalations/requirements andfeed back to test efforts

I work for a test team for a popular flavor of UNIX and we are
constantly asked "how do you know your testing is effective??"

Well testing is only as effective as the tests you execute...and the
tests come from development teams to compliment the new features they
create

But I have always had a feeling that more testing should be driven by
what customers are finding out in the field and the test environment
should mimic the customer experience, as opposed to being dictated by
what development see as sufficient.

So I'd like to know if anyone has ideas or tips/tricks for better
keeping the testing of an operating system relevant and effective (and
dynamic -> always moving the croshairs to hit different parts of the
product during the development cycle)

Does such a paradigm exist and if so how bst can I start working
twoard this goal

any advice welcome
sincerely
terry

> I work for a test team for a popular flavor of UNIX and we are
> constantly asked "how do you know your testing is effective??"

How do you answer them?

> Well testing is only as effective as the tests you execute...and the
> tests come from development teams to compliment the new features they
> create
>
> But I have always had a feeling that more testing should be driven by
> what customers are finding out in the field and the test environment
> should mimic the customer experience, as opposed to being dictated by
> what development see as sufficient.

I agree, in a not-only-but-also sense. That is, the development team
might have some very interesting and appropriate ideas for testing
that would never occur to a customer. In addition, customers will
also test the product inadvertently, unintentionally, accidentally,
incidentally, as part of their routine. Often they won't be aware
they're testing it until they see some evidence of failure.

> So I'd like to know if anyone has ideas or tips/tricks for better
> keeping the testing of an operating system relevant and effective (and
> dynamic -> always moving the croshairs to hit different parts of the
> product during the development cycle)

> Does such a paradigm exist and if so how best can I start working toward this goal.

One way to do it is to relax the notion of "best", since as soon as
you identify one approach as "best", you'll start to diminish the
attention that you pay to other approaches. Another is to consider
Ross Ashby's Law of Requisite Variety: "In order to understand
something complicated, you have to complicate yourself." In Rapid
Testing, we talk about diversity--diversify your test team and your
tactics--and broaden your testing to include as many different
customer roles and tasks and business domains as you can. Often this
starts with friendly, helpful, dedicated, and skilled technical
support. When I started working at Quarterdeck, it was considered
axiomatic that technical support people were like antennae for the
developers--that we were the principal sources of information on what
customers wanted from the product, and what problems they were finding
with it. When that attitude was no longer emphasized, our products
suffered for it. So: another approach: don't outsource technical
support. Make tech support, to the greatest degree feasible, part of
your test team. Do it informally, if need be; don't wait for
management support or encouragement, because you may die of old age
before it arrives. Encourage tech support to collect stories about
how people actually use the system. Get in touch with people with
interesting tasks and stories.

Consider a large variety of test coverage models, too: we talk about
structure (how it's put together), functions (what it does), data
(what it does stuff to), platforms (what it depends on), operations
(how people use it), and time (how the system and the things around it
are affected by time) as elements to consider. James Whittaker talks
about a diversified set of user models, where to him, "users" of a
program include the operating system, the file system, the API, and
the human customer. Consider a wide variety of quality criteria--we
think about capability, reliability, usability, security, scalability,
performance, installability, compatibility, supportability,
testability, maintainability, portability, and localizability.

Hope this helps...

---Michael B.

Nicely done michael. I'm reading whittaker right now.
Agree with your analysis
"Michael Bolton" <michael.a.bolton@gmail.com> wrote in message
news:549cced2-451e-4f60-8766-09bad8c8f01b@i76g2000hsf.googlegroups.com...
>> I work for a test team for a popular flavor of UNIX and we are
>> constantly asked "how do you know your testing is effective??"
>
> How do you answer them?
>
>> Well testing is only as effective as the tests you execute...and the
>> tests come from development teams to compliment the new features they
>> create
>>
>> But I have always had a feeling that more testing should be driven by
>> what customers are finding out in the field and the test environment
>> should mimic the customer experience, as opposed to being dictated by
>> what development see as sufficient.
>
> I agree, in a not-only-but-also sense. That is, the development team
> might have some very interesting and appropriate ideas for testing
> that would never occur to a customer. In addition, customers will
> also test the product inadvertently, unintentionally, accidentally,
> incidentally, as part of their routine. Often they won't be aware
> they're testing it until they see some evidence of failure.
>
>> So I'd like to know if anyone has ideas or tips/tricks for better
>> keeping the testing of an operating system relevant and effective (and
>> dynamic -> always moving the croshairs to hit different parts of the
>> product during the development cycle)
>
>> Does such a paradigm exist and if so how best can I start working toward
>> this goal.
>
> One way to do it is to relax the notion of "best", since as soon as
> you identify one approach as "best", you'll start to diminish the
> attention that you pay to other approaches. Another is to consider
> Ross Ashby's Law of Requisite Variety: "In order to understand
> something complicated, you have to complicate yourself." In Rapid
> Testing, we talk about diversity--diversify your test team and your
> tactics--and broaden your testing to include as many different
> customer roles and tasks and business domains as you can. Often this
> starts with friendly, helpful, dedicated, and skilled technical
> support. When I started working at Quarterdeck, it was considered
> axiomatic that technical support people were like antennae for the
> developers--that we were the principal sources of information on what
> customers wanted from the product, and what problems they were finding
> with it. When that attitude was no longer emphasized, our products
> suffered for it. So: another approach: don't outsource technical
> support. Make tech support, to the greatest degree feasible, part of
> your test team. Do it informally, if need be; don't wait for
> management support or encouragement, because you may die of old age
> before it arrives. Encourage tech support to collect stories about
> how people actually use the system. Get in touch with people with
> interesting tasks and stories.
>
> Consider a large variety of test coverage models, too: we talk about
> structure (how it's put together), functions (what it does), data
> (what it does stuff to), platforms (what it depends on), operations
> (how people use it), and time (how the system and the things around it
> are affected by time) as elements to consider. James Whittaker talks
> about a diversified set of user models, where to him, "users" of a
> program include the operating system, the file system, the API, and
> the human customer. Consider a wide variety of quality criteria--we
> think about capability, reliability, usability, security, scalability,
> performance, installability, compatibility, supportability,
> testability, maintainability, portability, and localizability.
>
> Hope this helps...
>
> ---Michael B.