On Software Quality Improvement

Hello,

I am a programmer who is currently a member of team which I believe
could benefit from many improvements to its development process. I
have made some attempts to discuss the issue with other colleagues and
management, but I am getting a fair share of indifference and
resistance.

The goal of this post is to promote a discussion on how to go about
improving the development process and the quality of the software
product. I would appreciate some feedback from people with experience
in this area. I want to take small steps and many of them. I discuss
here a potential first step. I must say that I am having a hard time
thinking of further steps which may be an indication that this "first"
step may not be too small after all.

As an introduction, I should say that over the last few years I have
significantly improved my abilities as a software engineer through
consistent study, experimentation, and hard work. I recently started
working for a large investment bank in the New York City area. The
team that I joined writes software for in-house use. It is comprised
of very talented people but with very poor software engineering
skills. They have plenty of business knowledge (many have been at
their current positions for many years) which is planted in plenty of
spaghetti code -hardly useful to the newcomer; a maintainer's
nightmare. Also, their development infrastructure (compilers, build
system, deployment system) is in tatters and there is plenty of manual
error-prone work as part of the daily routine.

I now spend much less time coding than I would like to. The rest of my
time is spent battling bug caused by duplication or needless
complexity (there is very little abstraction in the code base. The
code is fairly flat and implementation details are mixed with business
logic as if there is no tomorrow)

Finally, I am truly considering quitting my job. If the resistance
becomes unbearable or my abilities as team motivator are not up to the
task, then leaving are a viable solution. In fact, I am pursuing that
option at the same time I try to get some traction with the team to
improve the process and raise the quality of development. According to
my experience, however, there is a lack of software engineering skills
on the majority of software teams out there. That's why finding
another job, while apparently an easier solution, it is not without
problems. It is not easy to find a team of motivated software
developers who are interested in and put an effort in improving their
skills, with the support of their immediate or corporate managers -in
the end, this benefits the organization a great deal.

If someone patient enough to read this post wants to offer me a job, I
would seriously consider it (just having the patience and curiosity to
read through this is an indication of motivation, regardless of
whether or not there is agreement on the solutions)

And now to the substance ...

I am proposing to management that, as a first step, the collaboration
between the team members increases. I am suggesting that two people be
assigned to projects and they are both equally responsible for the
project: one more senior than the other. I tend to believe that the
natural coaching-mentoring that will go on will be healthy and
knowledge (business and software engineering knowledge) will be spread
faster through the team that way. I am also suggesting that code
reviews be done much more frequently (perhaps as frequent as several
times a day for a few minutes each time).

I put together a list of guidelines to be considered while doing
development and while doing frequent code reviews (one more detail:
development is primarily done in C++, so there are some references to
that fact below):

~ A little introspection goes a long way. Pay close attention to your
feelings and instincts while programming. Turn off the automatic pilot
for most of the day.

~ Write small functions with very clear names. You want to be able to
know what a function does by its name. You want to be able to grasp
how it does it by "glancing" at its implementation. We, humans, are
able to keep somewhere in the vicinity of 7 (+- 2) items in our short
term memory at the same time. Keep the number of things your functions
do within those limits.

~ pass few parameters to functions. This goes along the same line of
the item above. Use aggregates (structs, class, etc) to group
conceptually related parameters when passing them around. There should
be no need to pass more than 3 or 4 parameters to a function.

~ Use exceptions liberally; avoid returning error codes (ala C).
Exceptions are a great tool to remove the error handling code from the
first line of sight. As a general rule, consider throwing an exception
in a function when its work cannot be completed and all the calling
function can do is terminate if an error is detected. Exceptions are
computationally expensive (take longer than average), but …

~ "Premature optimization is the root of all evil" -Donald Knuth.
Local optimizations (without an objective, measurable idea of the
impact of the optimization) are generally a waste of brain cycles.
Furthermore, and most importantly, code readability degrades quickly
with ad-hoc premature optimizations. (code examples are required here
to show how not to reuse local variables; how to group together blocks
of code that perform a related task; how to create functions that
compute a value that would otherwise be stored in a local auxiliary
variable for the sake of "optimizations".)

~ Avoid creeping duplication. As you find yourself writing the second
copy of the same piece of code, wink at the duplication you are
creating. Also, mark the two copies with a comment stating the
duplication in such a way that makes it easy later to find the two
copies. But do not write a third copy of the same code, refactor the
two previous copies (they were easy to find because someone marked
them as such) and use the new tool in all three places.

~ Duplicating goes beyond identical lines of code. If you get to a
programming crossroad and make a decision, encapsulate it using a
function/class/template/macro/you-name-it. That way you expose the
essence of the problem you decision set out to solve. As you reuse
your you-name-it, the code that calls it becomes loosely dependent on
your decision; a major achievement indeed, based on the fact that you
increased clarity and, in case you made the wrong decision, you only
need to change one place in the code.

~ Practice ego-less programming. Have your code reviewed often and
review others' code often (as often as a couple of times a day). That
means that you need to think hard about the code you will have your
peer scrutinize. You do not want to waste her time nor yours by
showing a litany of if/then/else, for loops, and error-checking code.
It would be interesting instead to show a more abstract view of the
situation and how the code is written to tell a story about it. Do
take ownership of code but don’t get attached to it. That way it
becomes easier to learn from others.

~ Write code that is as simple for the client code to use as possible,
but not simpler come up with a CCT-related example For that purpose,
it may help to think how one would do it from scratch. It is healthy
to dream of freeing oneself from the constraints of the existing
design, for the practice can only give you insight into the essence of
the problem at hand. You would be exercising your beginner's mind.

~ the saying goes like this: "write one to throw away". Clearly at a
large scale, this does not make much sense, but at a smaller scale you
may reap some benefits. First you should focus on getting the stuff
done by sweating your interaction with that API or tools that are new
to you or you barely remember how it all works (you can safely argue
that they are not intuitive; that's an argument hard to counter and
you will sound sophisticated).
But once you managed to implement what you wanted in some spaghetti-
like form, you could put on the maintainability hat and give some
better shape to that code. What's better? Whatever is closer to the
domain; whatever is simpler for the client code (but not simpler);
whatever is not optimized prematurely and not done in a dumb manner;
whatever can be almost understood by glancing at it (I am repeating
myself). Taking that second step is crucial; it helps with the 6-
months-later effect of not remembering anything about the thing. The
good thing is that it helps not having to remember anything because
you could easily read the code.

~ Get acquainted with programming techniques and best practices. Make
the habit of always reading a book on programming. Major goal, but
easier to achieve than you think.

~ try out stuff that you learned. If you don’t, it fades away.

Belebele wrote:
> Hello,
>
> I am a programmer who is currently a member of team which I believe
> could benefit from many improvements to its development process. I
> have made some attempts to discuss the issue with other colleagues and
> management, but I am getting a fair share of indifference and
> resistance.
>
> The goal of this post is to promote a discussion on how to go about
> improving the development process and the quality of the software
> product. I would appreciate some feedback from people with experience
> in this area. I want to take small steps and many of them. I discuss
> here a potential first step. I must say that I am having a hard time
> thinking of further steps which may be an indication that this "first"
> step may not be too small after all.

First of all, you will surely (probability approaches 100%)
get more than one reply espousing the "one true way" they
believe in. Take these with a grain of salt. Pick and
choose.

>
> As an introduction, I should say that over the last few years I have
> significantly improved my abilities as a software engineer through
> consistent study, experimentation, and hard work. I recently started
> working for a large investment bank in the New York City area. The
> team that I joined writes software for in-house use. It is comprised
> of very talented people but with very poor software engineering
> skills. They have plenty of business knowledge (many have been at
> their current positions for many years) which is planted in plenty of
> spaghetti code -hardly useful to the newcomer; a maintainer's
> nightmare. Also, their development infrastructure (compilers, build
> system, deployment system) is in tatters and there is plenty of manual
> error-prone work as part of the daily routine.

What you describe is a MAJOR hit on productivity. Get
the build and and deployment cleaned up and automated
first before trying to tackle that spaghetti code.

If your shop does NOT do automated testing, beat the
drums for it. It may be hard as a newbie to the
organization to "carry a big stick," but try to
institute a rule that the test suite for the code
is submitted WITH the code. Both test-suite
source and results of the tests

>
> I now spend much less time coding than I would like to. The rest of my
> time is spent battling bug caused by duplication or needless
> complexity (there is very little abstraction in the code base. The
> code is fairly flat and implementation details are mixed with business
> logic as if there is no tomorrow)

That IS a tough battle to fight. The original crew has
a vested interest in the code they wrote. You will be
saying that their baby is ugly.

>
> Finally, I am truly considering quitting my job. If the resistance
> becomes unbearable or my abilities as team motivator are not up to the
> task, then leaving are a viable solution. In fact, I am pursuing that
> option at the same time I try to get some traction with the team to
> improve the process and raise the quality of development. According to
> my experience, however, there is a lack of software engineering skills
> on the majority of software teams out there. That's why finding
> another job, while apparently an easier solution, it is not without
> problems. It is not easy to find a team of motivated software
> developers who are interested in and put an effort in improving their
> skills, with the support of their immediate or corporate managers -in
> the end, this benefits the organization a great deal.
>
> If someone patient enough to read this post wants to offer me a job, I
> would seriously consider it (just having the patience and curiosity to
> read through this is an indication of motivation, regardless of
> whether or not there is agreement on the solutions)
>
> And now to the substance ...
>
> I am proposing to management that, as a first step, the collaboration
> between the team members increases. I am suggesting that two people be
> assigned to projects and they are both equally responsible for the
> project: one more senior than the other. I tend to believe that the
> natural coaching-mentoring that will go on will be healthy and
> knowledge (business and software engineering knowledge) will be spread
> faster through the team that way. I am also suggesting that code
> reviews be done much more frequently (perhaps as frequent as several
> times a day for a few minutes each time).

"Pair programming often works. But there may be
personality clashes, and turf battles. Beware.

>
> I put together a list of guidelines to be considered while doing
> development and while doing frequent code reviews (one more detail:
> development is primarily done in C++, so there are some references to
> that fact below):
>
> ~ A little introspection goes a long way. Pay close attention to your
> feelings and instincts while programming. Turn off the automatic pilot
> for most of the day.
>
> ~ Write small functions with very clear names. You want to be able to
> know what a function does by its name. You want to be able to grasp
> how it does it by "glancing" at its implementation. We, humans, are
> able to keep somewhere in the vicinity of 7 (+- 2) items in our short
> term memory at the same time. Keep the number of things your functions
> do within those limits.
>
> ~ pass few parameters to functions. This goes along the same line of
> the item above. Use aggregates (structs, class, etc) to group
> conceptually related parameters when passing them around. There should
> be no need to pass more than 3 or 4 parameters to a function.
>
> ~ Use exceptions liberally; avoid returning error codes (ala C).
> Exceptions are a great tool to remove the error handling code from the
> first line of sight. As a general rule, consider throwing an exception
> in a function when its work cannot be completed and all the calling
> function can do is terminate if an error is detected. Exceptions are
> computationally expensive (take longer than average), but …
>
> ~ "Premature optimization is the root of all evil" -Donald Knuth.
> Local optimizations (without an objective, measurable idea of the
> impact of the optimization) are generally a waste of brain cycles.
> Furthermore, and most importantly, code readability degrades quickly
> with ad-hoc premature optimizations. (code examples are required here
> to show how not to reuse local variables; how to group together blocks
> of code that perform a related task; how to create functions that
> compute a value that would otherwise be stored in a local auxiliary
> variable for the sake of "optimizations".)

In addition get people to ask "how fast MUST it be" rather
than "how fast WILL it be." The second mind-set leads
to an attitude that "we'll build it and see how fast
it is." The first mind-set required someone to place
real-life requirements on the speed and the throughput
requirements. These can then be apportioned out to
individual modules, e.g. if transaction A had to call
modules B, C, and D and meet a response time requirement
of 3 seconds, a first approximation is the each of
B, C, and D is allocated 1 second to do its work. This
way each of the modules can be tested individually.
It's usually OK as long as the total is less than 3 seconds.


[ Much snipped from original here. You're
going in the right direction, but it's a long
row to hoe. Been there, done that!]

Silverlock


--
"It is impossible to make anything foolproof
because fools are so ingenious"
- A. Bloch

On Mon, 1 Sep 2008 17:05:40 UTC, Belebele <beluchin@gmail.com> wrote:
> I am a programmer who is currently a member of team which I believe
> could benefit from many improvements to its development process. I
> have made some attempts to discuss the issue with other colleagues and
> management, but I am getting a fair share of indifference and
> resistance.
>
One thing to remember is that opinion is not evidence of a need for
change. Often evidence is required to change attitudes. Prove they are
wasting resources. You have two choices in this regard: 1) Track your
own effort closely and use it as evidence. 2) Convince management to
let you run a project using your processes, and track metrics closely.
Note that both of these are contingent on existing metrics. Don't
expect outside metrics from other companies'/practitioner's experience
to count. Also expect to meet resistance from the software developers
who have been there longer. People don't like to change, but they do
like it when the work is easier.

--
Lee W. Riemenschneider
GO BOILERS!
Running eComStation (eCS)(the latest incarnation of OS/2)
Buy eCS everyone! Buy it now! http://www.ecomstation.com

On Sep 1, 8:05 pm, Belebele <beluc...@gmail.com> wrote:
> Hello,
>
> I am a programmer who is currently a member of team which I believe
> could benefit from many improvements to its development process. I
> have made some attempts to discuss the issue with other colleagues and
> management, but I am getting a fair share of indifference and
> resistance.
>
> The goal of this post is to promote a discussion on how to go about
> improving the development process and the quality of the software
> product. I would appreciate some feedback from people with experience
> in this area. I want to take small steps and many of them. I discuss
> here a potential first step. I must say that I am having a hard time
> thinking of further steps which may be an indication that this "first"
> step may not be too small after all.
>
> As an introduction, I should say that over the last few years I have
> significantly improved my abilities as a software engineer through
> consistent study, experimentation, and hard work. I recently started
> working for a large investment bank in the New York City area. The
> team that I joined writes software for in-house use. It is comprised
> of very talented people but with very poor software engineering
> skills. They have plenty of business knowledge (many have been at
> their current positions for many years) which is planted in plenty of
> spaghetti code -hardly useful to the newcomer; a maintainer's
> nightmare. Also, their development infrastructure (compilers, build
> system, deployment system) is in tatters and there is plenty of manual
> error-prone work as part of the daily routine.
>
> I now spend much less time coding than I would like to. The rest of my
> time is spent battling bug caused by duplication or needless
> complexity (there is very little abstraction in the code base. The
> code is fairly flat and implementation details are mixed with business
> logic as if there is no tomorrow)
>
> Finally, I am truly considering quitting my job. If the resistance
> becomes unbearable or my abilities as team motivator are not up to the
> task, then leaving are a viable solution. In fact, I am pursuing that
> option at the same time I try to get some traction with the team to
> improve the process and raise the quality of development. According to
> my experience, however, there is a lack of software engineering skills
> on the majority of software teams out there. That's why finding
> another job, while apparently an easier solution, it is not without
> problems. It is not easy to find a team of motivated software
> developers who are interested in and put an effort in improving their
> skills, with the support of their immediate or corporate managers -in
> the end, this benefits the organization a great deal.
>
> If someone patient enough to read this post wants to offer me a job, I
> would seriously consider it (just having the patience and curiosity to
> read through this is an indication of motivation, regardless of
> whether or not there is agreement on the solutions)
>
> And now to the substance ...
>
> I am proposing to management that, as a first step, the collaboration
> between the team members increases. I am suggesting that two people be
> assigned to projects and they are both equally responsible for the
> project: one more senior than the other. I tend to believe that the
> natural coaching-mentoring that will go on will be healthy and
> knowledge (business and software engineering knowledge) will be spread
> faster through the team that way. I am also suggesting that code
> reviews be done much more frequently (perhaps as frequent as several
> times a day for a few minutes each time).
>
> I put together a list of guidelines to be considered while doing
> development and while doing frequent code reviews (one more detail:
> development is primarily done in C++, so there are some references to
> that fact below):
>
> ~ A little introspection goes a long way. Pay close attention to your
> feelings and instincts while programming. Turn off the automatic pilot
> for most of the day.
>
> ~ Write small functions with very clear names. You want to be able to
> know what a function does by its name. You want to be able to grasp
> how it does it by "glancing" at its implementation. We, humans, are
> able to keep somewhere in the vicinity of 7 (+- 2) items in our short
> term memory at the same time. Keep the number of things your functions
> do within those limits.
>
> ~ pass few parameters to functions. This goes along the same line of
> the item above. Use aggregates (structs, class, etc) to group
> conceptually related parameters when passing them around. There should
> be no need to pass more than 3 or 4 parameters to a function.
>
> ~ Use exceptions liberally; avoid returning error codes (ala C).
> Exceptions are a great tool to remove the error handling code from the
> first line of sight. As a general rule, consider throwing an exception
> in a function when its work cannot be completed and all the calling
> function can do is terminate if an error is detected. Exceptions are
> computationally expensive (take longer than average), but …
>
> ~ "Premature optimization is the root of all evil" -Donald Knuth.
> Local optimizations (without an objective, measurable idea of the
> impact of the optimization) are generally a waste of brain cycles.
> Furthermore, and most importantly, code readability degrades quickly
> with ad-hoc premature optimizations. (code examples are required here
> to show how not to reuse local variables; how to group together blocks
> of code that perform a related task; how to create functions that
> compute a value that would otherwise be stored in a local auxiliary
> variable for the sake of "optimizations".)
>
> ~ Avoid creeping duplication. As you find yourself writing the second
> copy of the same piece of code, wink at the duplication you are
> creating. Also, mark the two copies with a comment stating the
> duplication in such a way that makes it easy later to find the two
> copies. But do not write a third copy of the same code, refactor the
> two previous copies (they were easy to find because someone marked
> them as such) and use the new tool in all three places.
>
> ~ Duplicating goes beyond identical lines of code. If you get to a
> programming crossroad and make a decision, encapsulate it using a
> function/class/template/macro/you-name-it. That way you expose the
> essence of the problem you decision set out to solve. As you reuse
> your you-name-it, the code that calls it becomes loosely dependent on
> your decision; a major achievement indeed, based on the fact that you
> increased clarity and, in case you made the wrong decision, you only
> need to change one place in the code.
>
> ~ Practice ego-less programming. Have your code reviewed often and
> review others' code often (as often as a couple of times a day). That
> means that you need to think hard about the code you will have your
> peer scrutinize. You do not want to waste her time nor yours by
> showing a litany of if/then/else, for loops, and error-checking code.
> It would be interesting instead to show a more abstract view of the
> situation and how the code is written to tell a story about it. Do
> take ownership of code but don’t get attached to it. That way it
> becomes easier to learn from others.
>
> ~ Write code that is as simple for the client code to use as possible,
> but not simpler come up with a CCT-related example For that purpose,
> it may help to think how one would do it from scratch. It is healthy
> to dream of freeing oneself from the constraints of the existing
> design, for the practice can only give you insight into the essence of
> the problem at hand. You would be exercising your beginner's mind.
>
> ~ the saying goes like this: "write one to throw away". Clearly at a
> large scale, this does not make much sense, but at a smaller scale you
> may reap some benefits. First you should focus on getting the stuff
> done by sweating your interaction with that API or tools that are new
> to you or you barely remember how it all works (you can safely argue
> that they are not intuitive; that's an argument hard to counter and
> you will sound sophisticated).
> But once you managed to implement what you wanted in some spaghetti-
> like form, you could put on the maintainability hat and give some
> better shape to that code. What's better? Whatever is closer to the
> domain; whatever is simpler for the client code (but not simpler);
> whatever is not optimized prematurely and not done in a dumb manner;
> whatever can be almost understood by glancing at it (I am repeating
> myself). Taking that second step is crucial; it helps with the 6-
> months-later effect of not remembering anything about the thing. The
> good thing is that it helps not having to remember anything because
> you could easily read the code.
>
> ~ Get acquainted with programming techniques and best practices. Make
> the habit of always reading a book on programming. Major goal, but
> easier to achieve than you think.
>
> ~ try out stuff that you learned. If you don’t, it fades away.


Overcoming resistance is not a technical problem, so you need the
skill and the evidence to convince people, to talk them to be on your
side. Of course you shall start from the management. Without
management support all your efforts will be a waste. If you cannot
sell the whole idea to your boss then you better stop hitting the wall
with your head and quit.

After you have got managerial support you will face another problem -
resistance of people who will face the fear of changes. Just find
several enthusiasts to support you in this role and keep delivering a
consistent message of why you need changes. Try using convincing facts
from the past history of the team or that of the whole company to show
that things are going bad and need to be improved. Words without facts
are a weak weapon against the resistance.

Another big issue about introducing changes is focusing on what is
really important. There is nothing that kills motivation better than
seeing no return on the efforts applied. Do one thing at a time and do
it completely before starting another one. This is very common when
important changes are implemented partially and do not serve the
purpose whereas the focus moved to something new and "more exciting".
Do not fool yourself. The goal is the result, not the process.

----
Best Wishes,
Vladimir

Responding to Belebele...

> I am a programmer who is currently a member of team which I believe
> could benefit from many improvements to its development process. I
> have made some attempts to discuss the issue with other colleagues and
> management, but I am getting a fair share of indifference and
> resistance.

This is pretty typical of ad hoc shops that buy into the notion that
software development is a purely creative endeavor and any sort of
formal process will inhibit that creativity. That's utter nonsense, but
you are not going to get anywhere trying to point that out to the people
who believe it.

This topic happens to be close to my heart and I can personally testify
that modifying such environments is a long and painful process that will
take years. There are substantial rewards at the end of the road, but
you need a lot of patience and a pretty thick skin along the way. As you
point out, the easier path is to look for a job in a shop that is
already process-oriented. Sadly, there are very few of those around.

You have apparently already read up on software process improvement
since many of the things you suggest are right out of the process
alphabet soup (TQM, CMM, 6-Sigma, et al). So those are clearly the right
things to do. But I think the iniital problem is getting people to do them.

IME the only way to get people to change the way they do things is to
demonstrate the value of the change in a controlled experiment. Each
experiment gets a new practice in place because it has demonstrated
value. But far more important is that each successful experiment builds
credibility in process improvement itself. It is tough to argue with
success. When you can say, for example, that previously there were X
defects/KLOC and with the improvement there are now Y defects/KLOC and Y
<< X, then it is pretty tough to argue that it was a waste of time.

To demonstrate value, though, you need to quantify it, which means
collecting data. But developers hate to collect data, so that is the
first hurdle to overcome. IMO, that is probably the single biggest
impediment to process improvement.

So you need management backing to try some small, controlled
experiments. That shouldn't be a problem if the problem is small enough
that it won't take a lot of time. You should initially select problems
that (a) everyone agrees are problems, (b) are small, and (c) where the
data needed to demonstrate improvement is easy to collect. IOW, don't
pick the "worst" problem; pick one that is likely to be a Poster Child
for process improvement efficacy. In the first few experiments *your*
real goal is not to make development life easier; it is to build
credibility for process improvement itself.

As I mentioned, developers initially hate to collect data because it
distracts from writing code. Ironically, though, once there is hard data
available, the developers often want to collect more. That's because the
data usually has additional implications beyond the original purpose and
those implications raise more questions that need more data for
resolution. In the end all good software developers have good analytic
skills and once one has a pile of data it makes a good target for those
skills.

However, a corollary is that you need to be very careful to make any
data collection minimally invasive. If it appears to be clumsy,
inconvenient, or time-consuming you will have militant resistance. I
cannot emphasize enough how important it is to provide user-friendly
data collection tools. Any effort in providing such tools will pay huge
dividends for buying into process improvement.

<apocryphal anecdote>
Years ago I was in a group that had already been successful with process
improvement and Management mandated that all the other software groups
should do what we did. We had just adopted a spreadsheet from TSP that
suited our needs even though it was poorly designed because you had to
scroll through tabs to get to the data entry forms that were most
commonly used.

A couple of days after we introduced another group to the spreadsheet,
an irate group manager bent my ear for half an hour about how the whole
data collection thing was a total waste of time that was trashing his
group's productivity. In reality, the data that they needed to enter on
a daily basis would take no more than a total of -- at most -- ten
minutes a week and probably a lot less per developer. What that group
had was just a perception of difficulty due to the annoying need to
scroll the tabs in the spreadsheet. That perception was all that really
mattered because they were predisposed to not wanting to collect data so
the tool provided a focal point for their objections. In contrast, our
group had already bought into the value of the data and saw the
scrolling as a minor price to pay for a huge gain.

Moral: it is not the fact that matters for data collection; it is the
perception.
</apocryphal anecdote>

Another piece of advice would be to provide control for the initial
experiments by using a predefined problem solution process. I happened
to use the 7-Step from TQM, but all the process improvement frameworks
have them and it really doesn't make any difference which one you use
since they all share the same underlying process engineering. The value
is that such processes already have consensus and team building
techniques built into them, which will help in getting buy-in through
active participation. Those processes are also self-correcting because
they provide for improving the improvement process itself (e.g., the
meetings went off on too many tangents). It you use such a process there
really isn't much for the critics to get their hands around.

You don't have to tell the developers exactly where you got it initially
and that might be desirable because just mentioning acronyms like CMM or
TQM in zoo shops can cause the developers to bring out the garlic cloves
and crucifixes even though they have absolutely no idea what those
frameworks are really about.



--
There is nothing wrong with me that could
not be cured by a capful of Drano.

H. S. Lahman
hsl@pathfindermda.com
Pathfinder Solutions
http://www.pathfindermda.com
blog: http://pathfinderpeople.blogs.com/hslahman
"Model-Based Translation: The Next Step in Agile Development". Email
info@pathfindermda.com for your copy.
Pathfinder is hiring:
http://www.pathfindermda.com/about_us/careers_pos3.php.
(888)OOA-PATH

Belebele wrote:

> I now spend much less time coding than I would like to. The rest of my
> time is spent battling bug caused by duplication or needless
> complexity (there is very little abstraction in the code base. The
> code is fairly flat and implementation details are mixed with business
> logic as if there is no tomorrow)

Google for "test driven development", and "lean software development".

> I am proposing to management that, as a first step, the collaboration
> between the team members increases. I am suggesting that two people be
> assigned to projects and they are both equally responsible for the
> project: one more senior than the other. I tend to believe that the
> natural coaching-mentoring that will go on will be healthy and
> knowledge (business and software engineering knowledge) will be spread
> faster through the team that way. I am also suggesting that code
> reviews be done much more frequently (perhaps as frequent as several
> times a day for a few minutes each time).

Google "pair programming" - and don't commit a pair to a task. Rotate pairs
every few hours - yes even in the middle of a feature. The whole crew should
eventually work on everything.

> ~ Write small functions with very clear names.

Look up "refactor mercilessly". Your post assumes that code, once written, can
never be changed.

> ~ Avoid creeping duplication. As you find yourself writing the second
> copy of the same piece of code, wink at the duplication you are
> creating. Also, mark the two copies with a comment stating the
> duplication in such a way that makes it easy later to find the two
> copies. But do not write a third copy of the same code, refactor the
> two previous copies (they were easy to find because someone marked
> them as such) and use the new tool in all three places.

Don't even allow single instances of duplication. Code should be DRY - Don't
Repeat Yourself. You should write code specifically to be refactored - sometimes
even cloning one method just to change one line in it - and you should refactor
early and often.

Next, if two routines look similar, refactor along these lines:

- move similar things next to each other
- refactor similar things to make them more similar
- merge them via Extract Method Refactor on their common lines

Note that "refactor" means "make the smallest change you can think of, pass all
the tests, and repeat". One big refactor consists of zillions of little ones.
But it ain't "refactoring" without unit tests. You _do_ have wall-to-wall unit
tests, don't you?

--
Phlip

I can't support the approach H. S. Lahman suggests strongly enough. The way
to get moving in the direction of a culture of software excellence is by
demonstration. Select some small part of the process, make some meaningful
measurements, make a change, measure again, prove that you have an
improvement.

You do need to think hard about your first few experiments. Changes early
in the process reap the greatest rewards, but it can take a long time for
the results to show. Look at situations where you can make credible
measurements and then expect to have measureable improvements fairly quickly
for your first few experiments. Start with small successes, and remember
that the only measurement management really understands is money, so
translate everything into money. Your accountants are your greatest asset
here, and also down the road identifying low hanging fruit.

The whole business of measure, adjust, re-measure isn't very sexy.
Certainly any of the latest hot new software fads are a lot cooler. But if
you really want to make a significant improvement, then you need to begin
instituting a culture of making decisions based on data. The effects will
be small at first, but programmers are basically technical people and
inclined to be swayed by facts. As you have small successes, larger ones
will follow. Since you have the data, you don't need to keep doing things
that don't work. Recognize that some ideas won't work, and walk away from
them as soon as the data tells you that. People will respect your ability
to discard things that didn't work perhaps even more than they will the
really effective ideas.

Follow the data

...

"H. S. Lahman" <hsl@pathfindermda.com> wrote in message
news:CKdvk.317$jE1.64@trnddc03...
> Responding to Belebele...
>
>> I am a programmer who is currently a member of team which I believe
>> could benefit from many improvements to its development process. I
>> have made some attempts to discuss the issue with other colleagues and
>> management, but I am getting a fair share of indifference and
>> resistance.
>
> This is pretty typical of ad hoc shops that buy into the notion that
> software development is a purely creative endeavor and any sort of formal
> process will inhibit that creativity. That's utter nonsense, but you are
> not going to get anywhere trying to point that out to the people who
> believe it.
>
> This topic happens to be close to my heart and I can personally testify
> that modifying such environments is a long and painful process that will
> take years. There are substantial rewards at the end of the road, but you
> need a lot of patience and a pretty thick skin along the way. As you point
> out, the easier path is to look for a job in a shop that is already
> process-oriented. Sadly, there are very few of those around.
>
> You have apparently already read up on software process improvement since
> many of the things you suggest are right out of the process alphabet soup
> (TQM, CMM, 6-Sigma, et al). So those are clearly the right things to do.
> But I think the iniital problem is getting people to do them.
>
> IME the only way to get people to change the way they do things is to
> demonstrate the value of the change in a controlled experiment. Each
> experiment gets a new practice in place because it has demonstrated value.
> But far more important is that each successful experiment builds
> credibility in process improvement itself. It is tough to argue with
> success. When you can say, for example, that previously there were X
> defects/KLOC and with the improvement there are now Y defects/KLOC and Y
> << X, then it is pretty tough to argue that it was a waste of time.
>
> To demonstrate value, though, you need to quantify it, which means
> collecting data. But developers hate to collect data, so that is the first
> hurdle to overcome. IMO, that is probably the single biggest impediment to
> process improvement.
>
> So you need management backing to try some small, controlled experiments.
> That shouldn't be a problem if the problem is small enough that it won't
> take a lot of time. You should initially select problems that (a) everyone
> agrees are problems, (b) are small, and (c) where the data needed to
> demonstrate improvement is easy to collect. IOW, don't pick the "worst"
> problem; pick one that is likely to be a Poster Child for process
> improvement efficacy. In the first few experiments *your* real goal is not
> to make development life easier; it is to build credibility for process
> improvement itself.
>
> As I mentioned, developers initially hate to collect data because it
> distracts from writing code. Ironically, though, once there is hard data
> available, the developers often want to collect more. That's because the
> data usually has additional implications beyond the original purpose and
> those implications raise more questions that need more data for
> resolution. In the end all good software developers have good analytic
> skills and once one has a pile of data it makes a good target for those
> skills.
>
> However, a corollary is that you need to be very careful to make any data
> collection minimally invasive. If it appears to be clumsy, inconvenient,
> or time-consuming you will have militant resistance. I cannot emphasize
> enough how important it is to provide user-friendly data collection tools.
> Any effort in providing such tools will pay huge dividends for buying into
> process improvement.
>
> <apocryphal anecdote>
> Years ago I was in a group that had already been successful with process
> improvement and Management mandated that all the other software groups
> should do what we did. We had just adopted a spreadsheet from TSP that
> suited our needs even though it was poorly designed because you had to
> scroll through tabs to get to the data entry forms that were most commonly
> used.
>
> A couple of days after we introduced another group to the spreadsheet, an
> irate group manager bent my ear for half an hour about how the whole data
> collection thing was a total waste of time that was trashing his group's
> productivity. In reality, the data that they needed to enter on a daily
> basis would take no more than a total of -- at most -- ten minutes a week
> and probably a lot less per developer. What that group had was just a
> perception of difficulty due to the annoying need to scroll the tabs in
> the spreadsheet. That perception was all that really mattered because they
> were predisposed to not wanting to collect data so the tool provided a
> focal point for their objections. In contrast, our group had already
> bought into the value of the data and saw the scrolling as a minor price
> to pay for a huge gain.
>
> Moral: it is not the fact that matters for data collection; it is the
> perception.
> </apocryphal anecdote>
>
> Another piece of advice would be to provide control for the initial
> experiments by using a predefined problem solution process. I happened to
> use the 7-Step from TQM, but all the process improvement frameworks have
> them and it really doesn't make any difference which one you use since
> they all share the same underlying process engineering. The value is that
> such processes already have consensus and team building techniques built
> into them, which will help in getting buy-in through active participation.
> Those processes are also self-correcting because they provide for
> improving the improvement process itself (e.g., the meetings went off on
> too many tangents). It you use such a process there really isn't much for
> the critics to get their hands around.
>
> You don't have to tell the developers exactly where you got it initially
> and that might be desirable because just mentioning acronyms like CMM or
> TQM in zoo shops can cause the developers to bring out the garlic cloves
> and crucifixes even though they have absolutely no idea what those
> frameworks are really about.
>
>
>
> --
> There is nothing wrong with me that could
> not be cured by a capful of Drano.
>
> H. S. Lahman
> hsl@pathfindermda.com
> Pathfinder Solutions
> http://www.pathfindermda.com
> blog: http://pathfinderpeople.blogs.com/hslahman
> "Model-Based Translation: The Next Step in Agile Development". Email
> info@pathfindermda.com for your copy.
> Pathfinder is hiring:
> http://www.pathfindermda.com/about_us/careers_pos3.php.
> (888)OOA-PATH

Hi,

The list that you posted is absolutely correct! However, in order to
make guidelines effective they must be enforced.
You could consider the use of source code analysis tools for that.
Additionally, these tools also allow you to check for the technical
quality of your source code. This can help you to tackle two important
issues brought up by Lee: evidence and management support.
By running these tools, you can have insight about your software
quality and gather evidence of potential problems. With clear facts is
easier to argument for requesting management support and to convince
others.

Another important issue that was not brought up yet is software
monitoring.
Using health as analogy, one should do medical check-up both when
having a problem and periodically. In the presence of symptoms in most
of the cases we go to the doctor. That's clear! However, we should
also go to the doctor periodically, to check-up for our health. For
instance, doing yearly blood tests allows us to check up for our
levels of fat, which if high, creates awareness that if measures are
not taken we risk having a heart attack!
The same should happen with software. You should measure technical
quality regularly, and look for quality indicators (size, complexity,
etc.) and trends. You can use those to create awareness of problems
and use them as input to base your decisions to control and manage
software quality!

You can try to do this in your company for a small project (or a small
part of your system) and if successful try to apply it for the overall
system.
Alternatively, you could also consider to contact Software Improvement
Group (SIG) (http://www.sig.eu/).
SIG is a company specialized in providing consultancy support to solve
these problems having extensive experience in banking, government,
logistical and IT industries.