Read this post by Alistair Cockburn on Design as Knowledge Acquisition.

Developing application software, which is what I’ve done for a decade, always involves learning — acquiring knowledge.  We have to learn about the problem space (the ‘domain’), proposed solutions, the goals of the software, and design technical architectures that will support the goals.  It is frequently the case that some areas are new, and involve lot of unknowns.  The old fashioned waterfall approach tried to tackle the problem of these unknowns by a long period of study of the problems, prior to making attempts to design solutions.  The idea was that more knowledge up front would lead to better designs, earlier.  This is actually mistaken.   Implementation design should begin with only just enough initial knowledge to get going.  Assumptions can be tested against reality in a more rapid fashion.  This leads more quickly to satisfactory solutions at lower costs.  Alistair’s post describes it in more detail.  Do read it!

I was re-reading appendix B in Alistair Cockburn’s book, Agile Software Development, 2nd edition.  He has posted the entire appendix on his own blog.

Peter Naur thinks it important to consider the sort of activity that programming is.  Because if it is misconceived, we will not be as successful at it as we could be.

… the main point I want to make is that programming in this sense primarily must be the programmers’ building up knowledge of a certain kind, knowledge taken to be basically the programmers’ immediate possession…

Naur calls this kind of knowledge a ‘theory’, in the sense that the philosopher Gilbert Ryle used the term. A theory in this sense is a kind of tacit knowledge that a person can acquire, that – here is the strange part – cannot be expressed, i.e. cannot be put down as so many rules or principles.  However, one in possession of a theory about a program can, by working closely with others, induce them to acquire a similar theory, one that is serviceable for carrying on the development of a computer program in accord with the theory of it.

What does this mean for practice?  It is often recommended that a team of programmers who have developed a program remain closely connected with it throughout its life.  In Naur’s sense, they possess a theory of the program that other people cannot acquire by reading the program source code and any amount of documentation, no matter how well written.  There will be objection to this idea.  It once was, and still is in some quarters, a standard practice to hand over a program to a maintenance team after the program sees production.  The idea is that the original designers, presumably expert developers, should move on to new projects where their skill in, shall we say ‘theory building’, can best be utilized.  Meanwhile, the maintenance team, usually comprised of members with less skill and experience than the original developers, will take responsibility for program extensions and bug fixes.  A major point of Naur’s essay is that this is misguided.  Without a period of transition, where at least some of the original developers work on the maintenance team, the group picking up a program having only the source and documentation will be trying to ‘resuscitate’ a dead program, a program whose original development team has dispersed and the theory forgotten.  They will be attempting to build a new theory of the program and how it relates to the world.  And it is almost guaranteed they will get it wrong.  This is not a diatribe against documentation.  But it must be realized that the point of documentation is to be an aid to memory in the building of a theory of the program.  Using a document alone is not sufficient to acquire an adequate theory of the program.  Getting that requires speaking directly – and at length – with someone in possession of it.

You will have to digest the Naur essay and see if you agree or not.  I believe that Naur is right, and that a tacit knowledge, a theory of a program and its relation to the world (domain) is created by the programmer.  This is the real activity of what we call computer programming.  It is about gaining an understanding of a certain kind and becoming an expert in that program and its domain.  This turns the programmer, as person, into a value asset for the employer.  Naur concludes that as such, programmers should be recognized for what it is they actually do, which is to become knowledgeable.  This, and not the texts they produce as source code and documents, constitutes their real contribution.  This is why programmers should be kept together as working teams and kept working on software they have developed.  Because they are the ones in possession of the program’s theory and cannot give it easily over to others, but only over time in working with new team members.

A retrospective is a team meeting conducted at the end of each iteration (sprint) in which the team talks about how to improve how they work.  I believe that the effectiveness of an agile team strongly correlates with their ability to engage in productive retrospective discussion.

The topics in retrospectives I’ve been involved with have been concerned to ask and answer the questions, What worked?, What didn’t work?, What can we do better?  One thing easily overlooked is to examine how well the retrospective meeting itself is working.  If it becomes rote, it may become less effective.  But why is it so important?

An agile team is supposed to be self-managing.  To do that it needs to observe itself!  People are not usually very good at self-observation, and neither are groups.  The goal of continuous improvement requires taking stock of where we are, as a team.  It’s not obvious how to go about this.

I must interject a point I learned from Alistair Cockburn in his book, Agile Software Development.  Comparing agile approaches to the Capability Maturity Model, Cockburn notes that in CMMI optimization of the ‘process’ does not take place until the highest level – five- is reached.  That’s because in level four the project keeps metrics about the process and then in level five uses the metrics to feed back and improve.  Agile teams don’t need to gather metrics before they begin optimizing.  Why?  Because agile teams make use of the subjective judgments of members.  Collectively, a team can agree on ways to improve without needing objective measurements.

How team members are feeling is important.  In their book, Agile Retrospectives, Esther Derby and Diana Larsen give many exercises useful to elicit feelings, as well as ideas about how things are going.  “Coming into this retrospective, if you were a car, what kind of car would you be?” is one of their questions.  The answers will give a clue about how people are feeling and will enliven the talk.  One of the goals of an agile coach or scrum master is to help foster a safe space where people are free to express contrary opinions without fear.  By getting issues into the open in discussion, things that everyone knows are problems can be brought out and addressed.

An agile team takes responsibility for the outcome of its projects.  They can do this because they are empowered to make decisions about many aspects of development.  This way of working together asks the members to engage in a high level of social cooperation. To work well and to improve continuously, the team needs to be able to take a frank look at themselves.  Only if they can gain insights will they be able to put these into play.  That is why I say that without effective retrospective meetings, a team can’t be effectively agile.

Don’t give short shrift to your retrospective meeting!

Just a short post about what collaboration means in agile software development.  It means people working together at the same time on the same task, with continual discussion going on.  Pair Programming is the epitome of collaboration.  The people do not have to be co-located.  I have had very good pair programming meetings using screen sharing technologies, such as Skype.  You need to share a screen and audio.

Collaboration can involve more than two people, of course, but two is ideal.  It can be any activity, not just programming.  Pair story writing, pair test plan writing, pair domain modeling, and so forth, can all be collaborative.  The key is that two people are working on one problem together at the same time.  Those are the conditions for rapid, rich communication.  The opposite model is people sitting in cubicles working on pre-decomposed problems by themselves.  Some independent thinking is worthwhile.  But complete independence or dependence on written materials simply isn’t as effective.  Look up some of the research.  Collaboration, as I’ve defined it, is very productive.  The idea that people working separately on different parts of a complex task will be more effective, due to a ‘division of labor’, was taken apart years ago by Fred Brooks in The Mythical Man Month.  The reason is clear:  communications requirements reduce the efficiency of division of labor for highly complex tasks, such as developing software.

The idea of software craftsmanship seems to be gaining buzz recently.  Just now, Martin Fowler has also written a blog post about it.  The idea of software development as a craft vs. an engineering science isn’t at all new.  The book by Pete McBreen, pictured, was published in 2001.  (I’m not completely endorsing that book.  But it is a part of the history.)  The Wikipedia article has more history, so I’m not going to repeat that.

“Writing code is no longer the hard part of development; the hard part is figuring out what to write,” says McBreen.

I certainly agree with the second part of this statement, but I disagree that writing code is no longer hard.  Uncle Bob Martin is with me on this.  The industry abounds with laments about poor software quality, overly costly and failed projects.  You can find plenty of explanations and proposals to improve this state of affairs.  This discussion has been going on for decades without apparent progress in making software better.  It seems to me there is a divide between the engineering and ‘best practices’ approach vs. the craftsmanship and evolutionary practices approach.  The first approach starts from the assumption that software development can in principle be captured in the form of universal practices and standards, which if followed will lead to the best outcomes.  It also seems to me that while historically this has not yet been the case (this can be argued, of course), nevertheless adherents cling to a hope that it still can be brought about or at least be more widely attempted.

The second approach, the craftsmanship movement, wants to reframe the problem as not a technical or research-oriented one, but as one of recognizing that human beings make software, and that only humans can grasp what it’s for.  And recognizing that despite decades of attempts to make it easy for the users to write their own software, it really does take technical experts to do it well.  The challenge then is to find those individuals who combine the specialization of a software analyst having the ability to enter into the minds and language games of the customers, with the specialization of the programmer, who knows how to write code.  But this highly versatile type of person is a kind of rarity.  And there is the practical problem of how to find and educate them.  The craftsmanship adherents (I am one) are those who take it upon themselves to become proficient in all aspects of development.  It’s not an easy task, but we have help in the form of many conferences that are centered on practitioners needs.  We recognize the need to collaborate as paramount.  And I think this is where the more traditional engineering approach falters, in that it presumes that people are a fungible resource, and that highly specialized individuals can be operated like cogs in a machine, which machine is the modern organization.  It is only necessary to give people the proper training in areas where knowledge already exists and can be transmitted by educational institutions.  Those people will communicate using traditional methods, specifically written documents.   The limitations of these ideas is becoming more and more apparent to those developing software.

If good software depends on the availability of versatile and exceptional people, and such people are not readily available, what hope is there for widespread improvements in software outcomes?  Not much.  The only avenue for improving software by following the craftsmanship movement is to find ways to promote the recognition and education of exceptional people.  But can exceptional developers ever become the norm?  Probably not.  Thus the reluctance to depend on exceptional individuals leads many organizations to emphasize standards of practices, in hopes that if people of average ability can be employed and made to conform to the rules, the outcomes will be good in proportion to that conformance.  Is this faith well grounded is the hard question.

In conclusion then, it is not surprising to find widespread problems in software cost and quality, if achieving these goals takes exceptional people and such people are rare, or that professional standards of practice either are not well known or not actually used.   It is also not surprising to find persistent arguments for standards of practices and training that would enable average workers to produce superlative results.  So the debate seems to be whether to find and encourage exceptional developers, or to research and promote standards of practice.  Why not do both?  Are they really that antithetical?  I am not sure, but it is clear that the understanding of human beings is quite different in each.  And I believe that that difference lies behind much of the strident arguments.

Steve Denning has a new book out. The Leader’s Guide to Radical Management. See also Steve Denning’s blog, and this interview with him on InfoQ. Denning’s seven principles of management resonate with the principles of the Agile Manifesto.  Is Radical Management yet another management fad?  I don’t think so.  But I also think that it’s not that new of an idea.  It may rather be an emergence into practice of ideas being developed more than 100 years ago in response to problems of knowledge and authority created by modernism.

I am currently reading a fine book by John Patrick Diggins, The Promise of Pragmatism: Modernity and the Crisis of Knowledge and Authority.  Diggins was a historian who delved deeply into the history of ideas.  His explication of the philosophies of such founders of American pragmatism as Henry Adams, William James, C.S. Peirce, and John Dewey is clear and uncluttered by jargon.

As I was reading Diggins on Dewey, I was struck by similarities between some of Dewey’s ideas and the principles propounded by software agilists trying to be ‘pragmatic.’  I’m going to quote from Diggins about Dewey.  “Thought is prompted by the specific conditions of a given environment, whose discordance and instability confront men with ‘problematic situations’ that activate the mind and initiate processes of ‘inquiry.’…  Dewey (saw) the ‘problematic situation’ as requiring cooperative effort (my italics) to resolve.” … “Dewey makes clear that genuine knowledge involves not what is worth believing from the individual’s standpoint but what society judges useful, a judgment that must be continually reexamined.”  OK.  Now substitute ‘software project’ for ‘problematic situation’ and consider what cooperative effort is becoming these days in that problem space.

Diggins book addresses the problem of authority that arose with modernity.  Again quoting, “In classical philosophy knowing and doing represented two discrete activities, and the political implications of this epistemological dualism could only be undemocratic: authority resided in knowing and obedience in doing.”  Twentieth century bureaucracy developed hierarchical systems of authority based on the classical model.  (One of Diggins’s extreme examples is the Soviet Union.)  The ‘managers’ are bureaucrats who have knowledge and give orders to workers who must obey.  But Dewey ‘turned upside down the assumptions of classical thought: authority… is neither given by nor revealed to the theoretical intellect, but instead is produced by human activity.”

I think what’s happening with ‘radical management’ and ‘agile software development’ is related first to the gradual collapse of the myth of the knowledge-possessing manager, what Steve Denning calls the ‘old paradigm of command and control,’ and second on the ascension of practice based on principles rooted in American pragmatism, as exemplified by Dewey.  A key  principle is the ‘self-organizing team’ which has no central authority.  This will seem crazy to people stuck in the old paradigm.  Why?  Because the old paradigm presumes that knowledge is needed prior to action (e.g. a full specification, perhaps, before coding), whereas Dewey argued that knowledge and action (doing) are not separate.  And that success is not predefined, but emerges in collective activity.   For software, now, success is increasingly not measured by conformance to a specification, but by ‘delighting the customer’.

Quoting from the Denning interview, “Radical management is thus part of a larger story, an emerging process of societal change, in which the structures that we build are adjusted to enhance rather than strangle the living part of our lives.”

The current global economy, increasing the pace, has stressed traditional management structures which are too rigid to respond to rapid change.  This has opened up a field for innovation and even ‘paradigm change’ in corporate organization.  So now ideas developed eighty to a hundred years ago can find fertile soil to sprout in.  Radical management may be a new buzz word, but I think it’s not that new of an idea.  It just may be an idea that’s found the right conditions to flourish.

I’m reading the book, Continuous Delivery.  It addresses a very important and widely overlooked problem with information systems:  The deployment of a software application into an execution environment is fraught with peril.  Especially if it’s the production environment.  The authors note what every development team knows well, that rolling out a deployment is typically a time of stress and anxiety.  And they remind us why.  It’s most often a manual process that’s not repeatable.  It involves people working in silos:  Developers, DBAs, OS Administrators, Network Administrators, Application Administrators, who all must coordinate.  Reams of documentation, usually out of date, detailing manual procedures are thrown over walls in the hopes that the instructions will be followed.  Sound familiar?  It does to me!  My attitude towards this problem in the past is probably typical for application developers.  “I provided some documentation, and I’m available to answer questions.  Other than that, it’s not my job.”

This attitude can often be explained by organizational bureaucracy. Division of labor and assignment of tasks to people with specialized skills has been typical of the way I.T. businesses work.  The authors identify these anti-patterns:

• The production of extensive, detailed deployment documentation
• Reliance on manual testing to confirm that the application is running correctly
• Frequent calls to the development team with problems on release day
• Frequent corrections to the release process during the course of a release
• Clustered environments, differing in configuration across cluster members
• Releases that take more than a few minutes to perform
• Releases that are unpredictable, have unforeseen problems, need to be rolled back manually
• Sitting bleary-eyed in front of a monitor at 2 A.M. the day after the release, trying to make it work

Shouldn’t we move forward to address these problems and apply the same standards we use to develop quality software to the process of deploying that software?  This is the final step in automation that’s been successful with continuous integration where on every change to the software, data, or configuration:

1. The full set of automated tests:  unit, functional, integration, acceptance, are run.
2. Any test failure is immediately reported to the development team, and is quickly addressed.
3. The successful build is deployed to a testing server for review by humans.

The ideal of the authors is that in addition to all of the source code, test code, database scripts, and build scripts, the project revision system holds configuration information for every environment, allowing automatic deployment on command of any version of the software to any defined environment.  A human need only specify the version of the software and a target environment and press the ‘make it so’ button.  But why haven’t we been doing this?

Well, it’s hard!  There are technical challenges.  But there are also organizational challenges, turf wars, and role issues.   Resistance to change often takes the form of, “This is how we do things here.”  I bought the book to dig into the technical solutions the author have discovered.  However, the author state, “It’s axiomatic that most projects fail due to people problems, rather than technical problems.”  They argue that the Operations group needs to be engaged with Development much better than typically found.  Everyone shares the goal of delivering business value.  I’m looking forward to reading more about the author’s experiences in this dimension.  And I want automated deployment for continuous delivery to become part of my job, and not just a job for someone else.

I’ve been playing around with Spring Roo.  Roo is a Java developer tool.  Essentially it’s a code generator for Java Spring projects that generates not only Java code, but can scaffold entire web database applications.  Maybe you’ve seen one of those Ruby On Rails or Grails demonstrations that start with a new directory and create a fully functional web database app in about five minutes.  You can do the same thing with Roo.  And at about the same level of basic functionality.  Except you wind up with an application based on Maven, Spring Core, AspectJ, a JPA provider such as Hibernate, and Spring MVC or Google Web Toolkit.  Here’s a neat demo video of Roo that features GWT.

After doing the ten-minute tutorial, I used the roo command tool to build a new application from scratch.  My application would have a database table of users, roles, and a mapping of users to roles.  The idea was to have roles that are editable by an admin, a basic user management system.

I found it easy to create the domain model classes with roo commands, including the many-to-many mapping.  Integration tests against a MySQL database were generated automatically.  With one command I created the entire web tier.   The app ran with the mvn tomcat:run command and allowed me to edit roles and users.  Great!  Next, one command added Spring Security.  But there the automation stopped.  I would have to do manual coding to integrate my persistent users and roles with Spring Security.  Still, not bad.

Things to know about Spring Roo

• Relies heavily on AspectJ.  Most of the generated code is in .aj files.
• Does not use DAO or service layer patterns.  Model and Controller classes, mostly as aspects, contain all the logic.
• Depends heavily on Spring annotations.
• Uses REST URL patterns.  For example, update uses HTTP PUT.
• Does round-trip code generation.  You can add a field in a Java model class and roo detects this and automatically updates related aspects and web forms related to the entity.
• Uses hbmtoddl to generate the DDL.  I’m not keen on defining my DDL in Java model classes, though.
• All artifact dependencies use Maven repositories maintained by Spring Source.  Dependencies are automatically added to the pom.xml file.  One benefit is getting OSGi compatible jars with manifests.

I’m undecided if I would try to use Roo for a new project.  My concerns include:

• There’s very little Java code, and you’re not allowed to edit the aspect source.
• Unit tests are written as aspects.  How do I write my own, say with Groovy?
• I would need to create a service layer for some things.
• I’m uncomfortable with code-generated custom SQL queries.  How do I take back some control without a DAO?
• I would need to provide a way to load test data to the database.
• I would need to remove hbm2ddl.

My main concern is the risk involved in learning which parts I can customize by hand without breaking things.  In short, as with many slick tools it’s easy to get started, but will take time to master.

Roo, like Rails or Grails, is opinionated.  It starts out with standard approaches to a Java web database project, approaches that you have to learn — and with which you may not always agree.  The key is understanding where lock-in could occur, such that your hands are tied by having used Roo. It’s not really clear what that risk is going in.

Here’s my PDF resume.

Richard Brewster Resume

I gave a talk on September 7, 2010, at the Neon Guild meeting.

http://www.neonguild.org/

I spoke about Ward Cunningham’s concept of technical debt, weaving in some ideas I got from Alistair Cockburn about the cost of making changes to software and practices such as refactoring and automated testing as a way to reduce ‘debt’.

The key idea is that software change becomes increasingly costly (exponentially as the number of changes).  I related technical debt to a buildup of complexity and confusion in the software design over time.  And I spoke of ‘building knowledge in’ as a way to reduce this problem.  Building knowledge in means improving the design, as new understanding is acquired.  New understand always occurs, but it is not always built back into the software, and thus risks getting lost in documents and in the heads of people who leave the project.  Knowledge is built in by continually improving the design, as well as adding automated tests.  The activities for doing this are paid back in cost savings over time.

Here’s a link to a PDF of the slides I used in the talk.

Technical_Debt