On Sofware Quality Improvement

Hello,

I am a programmer who is currently a member of team which I believe
could benefit from many improvements to its development process. I
have made some attempts to discuss the issue with other colleagues and
management, but I am getting a fair share of indifference and
resistance.

The goal of this post is to promote a discussion on how to go about
improving the development process and the quality of the software
product. I would appreciate some feedback from people with experience
in this area. I want to take small steps and many of them. I discuss
here a potential first step. I must say that I am having a hard time
thinking of further steps which may be an indication that this "first"
step may not be too small after all.

As an introduction, I should say that over the last few years I have
significantly improved my abilities as a software engineer through
consistent study, experimentation, and hard work. I recently started
working for a large investment bank in the New York City area. The
team that I joined writes software for in-house use. It is comprised
of very talented people but with very poor software engineering
skills. They have plenty of business knowledge (many have been at
their current positions for many years) which is planted in plenty of
spaghetti code -hardly useful to the newcomer; a maintainer's
nightmare. Also, their development infrastructure (compilers, build
system, deployment system) is in tatters and there is plenty of manual
error-prone work as part of the daily routine.

I now spend much less time coding than I would like to. The rest of my
time is spent battling bug caused by duplication or needless
complexity (there is very little abstraction in the code base. The
code is fairly flat and implementation details are mixed with business
logic as if there is no tomorrow)

Finally, I am truly considering quitting my job. If the resistance
becomes unbearable or my abilities as team motivator are not up to the
task, then leaving are a viable solution. In fact, I am pursuing that
option at the same time I try to get some traction with the team to
improve the process and raise the quality of development. According to
my experience, however, there is a lack of software engineering skills
on the majority of software teams out there. That's why finding
another job, while apparently an easier solution, it is not without
problems. It is not easy to find a team of motivated software
developers who are interested in and put an effort in improving their
skills, with the support of their immediate or corporate managers -in
the end, this benefits the organization a great deal.

If someone patient enough to read this post wants to offer me a job, I
would seriously consider it (just having the patience and curiosity to
read through this is an indication of motivation, regardless of
whether or not there is agreement on the solutions)

And now to the substance ...

I am proposing to management that, as a first step, the collaboration
between the team members increases. I am suggesting that two people be
assigned to projects and they are both equally responsible for the
project: one more senior than the other. I tend to believe that the
natural coaching-mentoring that will go on will be healthy and
knowledge (business and software engineering knowledge) will be spread
faster through the team that way. I am also suggesting that code
reviews be done much more frequently (perhaps as frequent as several
times a day for a few minutes each time).

I put together a list of guidelines to be considered while doing
development and while doing frequent code reviews (one more detail:
development is primarily done in C++, so there are some references to
that fact below):

~ A little introspection goes a long way. Pay close attention to your
feelings and instincts while programming. Turn off the automatic pilot
for most of the day.

~ Write small functions with very clear names. You want to be able to
know what a function does by its name. You want to be able to grasp
how it does it by "glancing" at its implementation. We, humans, are
able to keep somewhere in the vicinity of 7 (+- 2) items in our short
term memory at the same time. Keep the number of things your functions
do within those limits.

~ pass few parameters to functions. This goes along the same line of
the item above. Use aggregates (structs, class, etc) to group
conceptually related parameters when passing them around. There should
be no need to pass more than 3 or 4 parameters to a function.

~ Use exceptions liberally; avoid returning error codes (ala C).
Exceptions are a great tool to remove the error handling code from the
first line of sight. As a general rule, consider throwing an exception
in a function when its work cannot be completed and all the calling
function can do is terminate if an error is detected. Exceptions are
computationally expensive (take longer than average), but …

~ "Premature optimization is the root of all evil" -Donald Knuth.
Local optimizations (without an objective, measurable idea of the
impact of the optimization) are generally a waste of brain cycles.
Furthermore, and most importantly, code readability degrades quickly
with ad-hoc premature optimizations. (code examples are required here
to show how not to reuse local variables; how to group together blocks
of code that perform a related task; how to create functions that
compute a value that would otherwise be stored in a local auxiliary
variable for the sake of "optimizations".)

~ Avoid creeping duplication. As you find yourself writing the second
copy of the same piece of code, wink at the duplication you are
creating. Also, mark the two copies with a comment stating the
duplication in such a way that makes it easy later to find the two
copies. But do not write a third copy of the same code, refactor the
two previous copies (they were easy to find because someone marked
them as such) and use the new tool in all three places.

~ Duplicating goes beyond identical lines of code. If you get to a
programming crossroad and make a decision, encapsulate it using a
function/class/template/macro/you-name-it. That way you expose the
essence of the problem you decision set out to solve. As you reuse
your you-name-it, the code that calls it becomes loosely dependent on
your decision; a major achievement indeed, based on the fact that you
increased clarity and, in case you made the wrong decision, you only
need to change one place in the code.

~ Practice ego-less programming. Have your code reviewed often and
review others' code often (as often as a couple of times a day). That
means that you need to think hard about the code you will have your
peer scrutinize. You do not want to waste her time nor yours by
showing a litany of if/then/else, for loops, and error-checking code.
It would be interesting instead to show a more abstract view of the
situation and how the code is written to tell a story about it. Do
take ownership of code but don’t get attached to it. That way it
becomes easier to learn from others.

~ Write code that is as simple for the client code to use as possible,
but not simpler come up with a CCT-related example For that purpose,
it may help to think how one would do it from scratch. It is healthy
to dream of freeing oneself from the constraints of the existing
design, for the practice can only give you insight into the essence of
the problem at hand. You would be exercising your beginner's mind.

~ the saying goes like this: "write one to throw away". Clearly at a
large scale, this does not make much sense, but at a smaller scale you
may reap some benefits. First you should focus on getting the stuff
done by sweating your interaction with that API or tools that are new
to you or you barely remember how it all works (you can safely argue
that they are not intuitive; that's an argument hard to counter and
you will sound sophisticated).
But once you managed to implement what you wanted in some spaghetti-
like form, you could put on the maintainability hat and give some
better shape to that code. What's better? Whatever is closer to the
domain; whatever is simpler for the client code (but not simpler);
whatever is not optimized prematurely and not done in a dumb manner;
whatever can be almost understood by glancing at it (I am repeating
myself). Taking that second step is crucial; it helps with the 6-
months-later effect of not remembering anything about the thing. The
good thing is that it helps not having to remember anything because
you could easily read the code.

~ Get acquainted with programming techniques and best practices. Make
the habit of always reading a book on programming. Major goal, but
easier to achieve than you think.

~ try out stuff that you learned. If you don’t, it fades away.

Belebele <beluchin@gmail.com> writes:
> I am a programmer who is currently a member of team which I believe
> could benefit from many improvements to its development process. I
> have made some attempts to discuss the issue with other colleagues and
> management, but I am getting a fair share of indifference and
> resistance.
> [... good ideas ...]

It may also help to add this rule:

~ write small programs. Better ten 20kLoC programs working together
than a single 200kLoC one.

It will be easier to rewrite a small program to improve its quality,
than having to rewrite the big one, and it will be easier to share
libraries between two small programs, then a third, then yet another
one, etc, than to do the massive refactoring that would be needed for
the big one.

Of course, the difficulty may be in spliting the big program in small
ones...


--
__Pascal Bourguignon__ http://www.informatimago.com/

"You question the worthiness of my code? I should kill you where you
stand!"

Belebele <beluchin@gmail.com> writes:
> I am proposing to management that, as a first step, the collaboration
> between the team members increases. I am suggesting that two people be
> assigned to projects and they are both equally responsible for the
> project: one more senior than the other. I tend to believe that the
> natural coaching-mentoring that will go on will be healthy and
> knowledge (business and software engineering knowledge) will be spread
> faster through the team that way. I am also suggesting that code
> reviews be done much more frequently (perhaps as frequent as several
> times a day for a few minutes each time).

I'll just comment on a few of these. In general, I like your list.

> ~ Use exceptions liberally; avoid returning error codes (ala C).
> Exceptions are a great tool to remove the error handling code from the
> first line of sight. As a general rule, consider throwing an exception
> in a function when its work cannot be completed and all the calling
> function can do is terminate if an error is detected. Exceptions are
> computationally expensive (take longer than average), but …

In C++, this can be dangerous advice. Not necessarily bad advice, but
you had better make very sure that your team knows how to write
exception-safe code in C+! (OK, so they need to know how in any
language, but C++ seems to make it particularly perilous.)

> ~ pass few parameters to functions. This goes along the same line of
> the item above. Use aggregates (structs, class, etc) to group
> conceptually related parameters when passing them around. There should
> be no need to pass more than 3 or 4 parameters to a function.

In general, I would agree. In some cases, though, a function (or its
subordinate functions) may have sophisticated behavior controlled by
several options or flags; in some cases, this can be readily
encapsulated in a class (e.g. pretty-printing). In some other cases,
perhaps not so much, but with only positional parameters available, 3 or
4 sounds like a good rule. In environments using languages that make
available keyword or labeled parameters (Common Lisp, Python, OCaml), it
may be reasonable to relax this standard to permit significantly larger
numbers of keyword parameters (but there, you have the keywords explain
the purpose of each at the point of call, so you don't have the "now
what was parameter 7 for again?" situation).
>
> ~ Avoid creeping duplication. As you find yourself writing the second
> copy of the same piece of code, wink at the duplication you are
> creating. Also, mark the two copies with a comment stating the
> duplication in such a way that makes it easy later to find the two
> copies. But do not write a third copy of the same code, refactor the
> two previous copies (they were easy to find because someone marked
> them as such) and use the new tool in all three places.

I've built essentially this rule into my development processes over
time. Copy once -- the second time you want to copy, abstract. I
haven't been commenting duplicates, but most of my stuff has been
smaller solo jobs. It's a worth suggestion, though -- I may start
putting duplication comments in my code.

> ~ try out stuff that you learned. If you don’t, it fades away.

Just so long as "stuff you learned" isn't hairy nasty tricks that make
an Intercal programmer cringe .

- Michael

--
mouse, n: A device for pointing at the xterm in which you want to type.
Confused by the strange files? I cryptographically sign my messages.
For more information see <http://www.elehack.net/resources/gpg>.

On Sep 1, 1:03*pm, Belebele <beluc...@gmail.com> wrote:
> Hello,
>
> I am a programmer who is currently a member of team which I believe
> could benefit from many improvements to its development process. I
> have made some attempts to discuss the issue with other colleagues and
> management, but I am getting a fair share of indifference and
> resistance.
>
> The goal of this post is to promote a discussion on how to go about
> improving the development process and the quality of the software
> product. I would appreciate some feedback from people with experience
> in this area. I want to take small steps and many of them. I discuss
> here a potential first step. I must say that I am having a hard time
> thinking of further steps which may be an indication that this "first"
> step may not be too small after all.
>
> As an introduction, I should say that over the last few years I have
> significantly improved my abilities as a software engineer through
> consistent study, experimentation, and hard work. I recently started
> working for a large investment bank in the New York City area. The
> team that I joined writes software for in-house use. It is comprised
> of very talented people but with very poor software engineering
> skills. They have plenty of business knowledge (many have been at
> their current positions for many years) which is planted in plenty of
> spaghetti code -hardly useful to the newcomer; a maintainer's
> nightmare. Also, their development infrastructure (compilers, build
> system, deployment system) is in tatters and there is plenty of manual
> error-prone work as part of the daily routine.
>
> I now spend much less time coding than I would like to. The rest of my
> time is spent battling bug caused by duplication or needless
> complexity (there is very little abstraction in the code base. The
> code is fairly flat and implementation details are mixed with business
> logic as if there is no tomorrow)
>
> Finally, I am truly considering quitting my job. If the resistance
> becomes unbearable or my abilities as team motivator are not up to the
> task, then leaving are a viable solution. In fact, I am pursuing that
> option at the same time I try to get some traction with the team to
> improve the process and raise the quality of development. According to
> my experience, however, there is a lack of software engineering skills
> on the majority of software teams out there. That's why finding
> another job, while apparently an easier solution, it is not without
> problems. It is not easy to find a team of motivated software
> developers who are interested in and put an effort in improving their
> skills, with the support of their immediate or corporate managers -in
> the end, this benefits the organization a great deal.
>
> If someone patient enough to read this post wants to offer me a job, I
> would seriously consider it (just having the patience and curiosity to
> read through this is an indication of motivation, regardless of
> whether or not there is agreement on the solutions)
>
> And now to the substance ...
>
> I am proposing to management that, as a first step, the collaboration
> between the team members increases. I am suggesting that two people be
> assigned to projects and they are both equally responsible for the
> project: one more senior than the other. I tend to believe that the
> natural coaching-mentoring that will go on will be healthy and
> knowledge (business and software engineering knowledge) will be spread
> faster through the team that way. I am also suggesting that code
> reviews be done much more frequently (perhaps as frequent as several
> times a day for a few minutes each time).
>
> I put together a list of guidelines to be considered while doing
> development and while doing frequent code reviews (one more detail:
> development is primarily done in C++, so there are some references to
> that fact below):
>
> ~ A little introspection goes a long way. Pay close attention to your
> feelings and instincts while programming. Turn off the automatic pilot
> for most of the day.
>
> ~ Write small functions with very clear names. You want to be able to
> know what a function does by its name. You want to be able to grasp
> how it does it by "glancing" at its implementation. We, humans, are
> able to keep somewhere in the vicinity of 7 (+- 2) items in our short
> term memory at the same time. Keep the number of things your functions
> do within those limits.
>
> ~ pass few parameters to functions. This goes along the same line of
> the item above. Use aggregates (structs, class, etc) to group
> conceptually related parameters when passing them around. There should
> be no need to pass more than 3 or 4 parameters to a function.
>
> ~ Use exceptions liberally; avoid returning error codes (ala C).
> Exceptions are a great tool to remove the error handling code from the
> first line of sight. As a general rule, consider throwing an exception
> in a function when its work cannot be completed and all the calling
> function can do is terminate if an error is detected. Exceptions are
> computationally expensive (take longer than average), but …
>
> ~ "Premature optimization is the root of all evil" -Donald Knuth.
> Local optimizations (without an objective, measurable idea of the
> impact of the optimization) are generally a waste of brain cycles.
> Furthermore, and most importantly, code readability degrades quickly
> with ad-hoc premature optimizations. (code examples are required here
> to show how not to reuse local variables; how to group together blocks
> of code that perform a related task; how to create functions that
> compute a value that would otherwise be stored in a local auxiliary
> variable for the sake of "optimizations".)
>
> ~ Avoid creeping duplication. As you find yourself writing the second
> copy of the same piece of code, wink at the duplication you are
> creating. Also, mark the two copies with a comment stating the
> duplication in such a way that makes it easy later to find the two
> copies. But do not write a third copy of the same code, refactor the
> two previous copies (they were easy to find because someone marked
> them as such) and use the new tool in all three places.
>
> ~ Duplicating goes beyond identical lines of code. If you get to a
> programming crossroad and make a decision, encapsulate it using a
> function/class/template/macro/you-name-it. That way you expose the
> essence of the problem you decision set out to solve. As you reuse
> your you-name-it, the code that calls it becomes loosely dependent on
> your decision; a major achievement indeed, based on the fact that you
> increased clarity and, in case you made the wrong decision, you only
> need to change one place in the code.
>
> ~ Practice ego-less programming. Have your code reviewed often and
> review others' code often (as often as a couple of times a day). That
> means that you need to think hard about the code you will have your
> peer scrutinize. You do not want to waste her time nor yours by
> showing a litany of if/then/else, for loops, and error-checking code.
> It would be interesting instead to show a more abstract view of the
> situation and how the code is written to tell a story about it. Do
> take ownership of code but don’t get attached to it. That way it
> becomes easier to learn from others.
>
> ~ Write code that is as simple for the client code to use as possible,
> but not simpler come up with a CCT-related example For that purpose,
> it may help to think how one would do it from scratch. It is healthy
> to dream of freeing oneself from the constraints of the existing
> design, for the practice can only give you insight into the essence of
> the problem at hand. You would be exercising your beginner's mind.
>
> ~ the saying goes like this: "write one to throw away". Clearly at a
> large scale, this does not make much sense, but at a smaller scale you
> may reap some benefits. First you should focus on getting the stuff
> done by sweating your interaction with that API or tools that are new
> to you or you barely remember how it all works (you can safely argue
> that they are not intuitive; that's an argument hard to counter and
> you will sound sophisticated).
> But once you managed to implement what you wanted in some spaghetti-
> like form, you could put on the maintainability hat and give some
> better shape to that code. What's better? Whatever is closer to the
> domain; whatever is simpler for the client code (but not simpler);
> whatever is not optimized prematurely and not done in a dumb manner;
> whatever can be almost understood by glancing at it (I am repeating
> myself). Taking that second step is crucial; it helps with the 6-
> months-later effect of not remembering anything about the thing. The
> good thing is that it helps not having to remember anything because
> you could easily read the code.
>
> ~ Get acquainted with programming techniques and best practices. Make
> the habit of always reading a book on programming. Major goal, but
> easier to achieve than you think.
>
> ~ try out stuff that you learned. If you don’t, it fades away.

Your principles are excellent. But you're in the process of learning
that there is a difference between _you_ knowing the right ways to
move forward and getting other people to follow you.

You're also forming a correct conclusion: Your remaining problem is
one of leadership. The technical merit of your ideas isn't going to
get them accepted. _You_ as a person must get them accepted.

Moving to another job may take you to a setting where the leadership
problems are smaller. That's okay, if that's what you want.

On the other hand, if you approach the motivation of people in the
same way you've done with software engineeinrg, you could end up with
new valuable and fulfilling skills, not to mention far greater promise
for a great career. Good software engineers are not so hard to find.
Good software engineers who can transform software development
organizations are rare and extremely valuable. If you want to be one
of the latter, consider reading, studying and practicing--as much as
you have on software development--the best practices for persuading
people and leading teams.

The first step in effecting change as a leader is to guide the
organization to a common vision of where it wants to be. Working on
small, low level changes is not going to get you very far. You might
earn some initial credibility by instituting a couple of low level
changes that have obvious payoffs. Maybe you force a version control
system into your processes, and overtime requirements go down because
stupid version-related bugs go away. Now you have everyone's attention
for a while. This is nice, but in the longer run, the main task is to
get everyone agreeing what the organization can change to be. For
software deveopment, you should always define this "end state" in
measurable terms. Motivation follows when people buy into your
arguments for how their lives will be better when those end state
measurements are achieved: competitive advantage, job security,
bonuses, prestige, professional pride, yada yada. In turn, you'll
only be able to figure out the best motivating factors if you get to
know the people you're working with at a personal level and help them
learn to value each other. This is just the beginning. Read, study,
and practice. You're in a whole new ball game, which can be
incredible fun.

Pascal J. Bourguignon wrote:
> It may also help to add this rule:
>
> ~ write small programs. Better ten 20kLoC programs working together
> than a single 200kLoC one.
>
> It will be easier to rewrite a small program to improve its quality,
> than having to rewrite the big one, and it will be easier to share
> libraries between two small programs, then a third, then yet another
> one, etc, than to do the massive refactoring that would be needed for
> the big one.
>
> Of course, the difficulty may be in spliting the big program in small
> ones...

You mean splitting a program into *modules* right?


August

August Karlstrom <fusionfile@gmail.com> writes:

> Pascal J. Bourguignon wrote:
>> It may also help to add this rule:
>> ~ write small programs. Better ten 20kLoC programs working together
>> than a single 200kLoC one.
>> It will be easier to rewrite a small program to improve its quality,
>> than having to rewrite the big one, and it will be easier to share
>> libraries between two small programs, then a third, then yet another
>> one, etc, than to do the massive refactoring that would be needed for
>> the big one.
>> Of course, the difficulty may be in spliting the big program in
>> small
>> ones...
>
> You mean splitting a program into *modules* right?

Of course, but if possible, also into separate programs. Unix-like.
For example, instead of having one big program to manage files, we
have a lot of little commands: ls, cp, mv, rm, ...
Other examples would be mh
http://en.wikipedia.org/wiki/MH_Message_Handling_System
or postfix http://www.postfix.org/

--
__Pascal Bourguignon__ http://www.informatimago.com/

PLEASE NOTE: Some quantum physics theories suggest that when the
consumer is not directly observing this product, it may cease to
exist or will exist only in a vague and undetermined state.

Michael Ekstrand wrote:
> Belebele <beluchin@gmail.com> writes:
[...]
>> ~ pass few parameters to functions. This goes along the same line of
>> the item above. Use aggregates (structs, class, etc) to group
>> conceptually related parameters when passing them around. There should
>> be no need to pass more than 3 or 4 parameters to a function.
>
> In general, I would agree. In some cases, though, a function (or its
> subordinate functions) may have sophisticated behavior controlled by
> several options or flags; in some cases, this can be readily
> encapsulated in a class (e.g. pretty-printing).

See also http://en.wikipedia.org/wiki/Option-operand_separation.


August

August Karlstrom <fusionfile@gmail.com> writes:

> Michael Ekstrand wrote:
>> Belebele <beluchin@gmail.com> writes:
> [...]
>>> ~ pass few parameters to functions. This goes along the same line of
>>> the item above. Use aggregates (structs, class, etc) to group
>>> conceptually related parameters when passing them around. There should
>>> be no need to pass more than 3 or 4 parameters to a function.
>>
>> In general, I would agree. In some cases, though, a function (or its
>> subordinate functions) may have sophisticated behavior controlled by
>> several options or flags; in some cases, this can be readily
>> encapsulated in a class (e.g. pretty-printing).
>
> See also http://en.wikipedia.org/wiki/Option-operand_separation.

This reads much like the maxim: "pass data, not control" (to functions).

--
Ben.

Belebele wrote:
>
> And now to the substance ...
>
> I am proposing to management that, as a first step, the collaboration
> between the team members increases. I am suggesting that two people be
> assigned to projects and they are both equally responsible for the
> project: one more senior than the other. I tend to believe that the
> natural coaching-mentoring that will go on will be healthy and
> knowledge (business and software engineering knowledge) will be spread
> faster through the team that way. I am also suggesting that code
> reviews be done much more frequently (perhaps as frequent as several
> times a day for a few minutes each time).
>
A good start, you might even be able to nudge them onto full on pair
programming and collective code ownership.

> ~ A little introspection goes a long way. Pay close attention to your
> feelings and instincts while programming. Turn off the automatic pilot
> for most of the day.
>
Pairing helps here.

> ~ Write small functions with very clear names.

Good.

> ~ pass few parameters to functions.

Good.

> ~ Use exceptions liberally; avoid returning error codes (ala C).

Good so long as your team understands exception safe coding techniques.

> ~ "Premature optimization is the root of all evil" -Donald Knuth.

We all agree with that one!

> ~ Avoid creeping duplication. As you find yourself writing the second
> copy of the same piece of code, wink at the duplication you are
> creating.

Good, but see my closing comment.

> ~ Practice ego-less programming. Have your code reviewed often and
> review others' code often (as often as a couple of times a day).

Good, part way to collective code ownership.

> ~ Write code that is as simple for the client code to use as possible,
> but not simpler.

Good.

> ~ the saying goes like this: "write one to throw away".

> But once you managed to implement what you wanted in some spaghetti-
> like form, you could put on the maintainability hat and give some
> better shape to that code.

Good, refactor towards a better design.
>
> ~ Get acquainted with programming techniques and best practices. Make
> the habit of always reading a book on programming. Major goal, but
> easier to achieve than you think.

The problem there is what constitutes "best practice" varies..

One important point you omitted: Write comprehensive, fully automated
unit and acceptance tests. Without them, you risk breaking things when
you refactor code.

--
Ian Collins.

> One important point you omitted: *Write comprehensive, fully automated
> unit and acceptance tests. *Without them, you risk breaking things when
> you refactor code.

In my experience, writing effective test cases is far from trivial. I
also believe it is one of the most important tools in a programmer's
kit; with practice, test-first development helps you write good code
naturally, as opposed to having to fight your way through it.

Getting the right level of semantics to test is a skill that I have
been working on for a long time. The skill is related to the
understanding that finding the right abstractions is key. I struggled
through a (long) period in which my tests were too fine or too coarse
most of the time and I had to refactor them too often to the point
that they felt somewhat impractical. It was only with plenty of
perseverance and study that I started to get a feel for what the
comfortable semantic level to put under test was (the book Domain
Drive Design by Evans helped immensely).

For those reasons, I decided not to recommend test-driven development
initially. I felt that the amount of spaguetti code is too large, the
functions too long, the abstraction level too flat that the subtleties
of such fine tool could be missed irreparably.