Software Craftsmanship and the Matching Washer and Dryer

I’ve enjoyed Ted Neward’s recent posts on software craftsmanship.  I 100% agree that software craftsmen need to be vigilant about coming off as holier than thou and condescending.  I say this as someone who takes pride in my code.  Though I’ve never identified myself as a software craftsman, the title probably fits how I approach my work in software.  What I hope to contribute to the conversation is perhaps a reason to be a little more humble about it: Condescending to people from a position of software craftsman is akin to condescending to people about winning the consolation prize of the matching washer and dryer on Jeopardy.

I think that’s an appropriate analogy for many (most?) software craftsmen.  I have a hunch that almost nobody got into software with a burning desire to be a software craftsman, there was some other driving force.  Nobody goes on Jeopardy to try and get second place, they are trying for the big prize.  If someone was condescending about how intelligent they are, and offered as proof their 2nd place finish on Jeopardy, you would probably chuckle, at least on the inside.  Similarly, condescending about software craftsmanship is in all likelihood bragging about making the best of your 2nd, 3rd, or maybe 4th choice in life.  People get into software for all kinds of reasons, but a burning desire to be a craftsman was probably not one of them.

Some of you probably got into software to be the next Jobs, Woz, Gates, Brin, etc.  You wanted to build a software empire and change the world.  Guess what?  You failed, at least so far.  Keep that in mind when touting software craftsmanship.

Some of you were really excited about the science fiction-y aspects of computers.  You were enthralled by artificial intelligence, robotics, etc.  If you are touting software craftsmanship instead of your latest advances in AI or robotics, then you failed at your original intention.

I know for a fact that some of you got into computer science as a second choice after not gaining entrance into a computer/electrical engineering department.  At my undergraduate institution, the CE/EE requirements were really difficult and I heard more than one person complaining about this during my stint as a physics TA at my university.  Just remember, computer science was your second choice.

Or maybe your first choice in careers was totally unrelated to software.  That’s my situation.  In 1999 I failed at getting into the graduate schools I wanted to attend, in a field unrelated to software.  I did manage to get into my last choice school, the utter fallback position.  But it wasn’t even in the field I wanted to study.  It was related to the field I wanted to study, but getting a Masters degree in that field would largely have been an exercise in resume padding in the hopes of getting into one of the schools I had already failed in attending.  I attended one day of classes, realized there was no way I could go though 2-3 years of that on the possibility that it might lead to my eventual goal, and dropped out.  Luckily this was the height of the dot com boom and anyone with a pulse and who could spell HTML got a job in software.  So even though I now take craftsmanship seriously, I always try and remember that I got here after failing and because the entrance requirements to the guild of programming were incredibly lax at the time I started.

Anyway, there are lots of other reasons people get into software.  However, I have never heard people say that they got into this field to write unit tests, write elegant code (whatever that means), use agile methodologies, refactor code, write correct object oriented code, write correct functional code (we can’t even agree on what type of code to write), use a continuous integration server, or any of the other dozens of things that constitutes “software craftsmanship.”

But that’s o.k. in my book.  Almost nobody gets their first choice.  The good life does not consist in getting your first choice, it consists in being a good person.  Making the best of what you have, and being a software craftsman certainly can be a part of that, is noble and honorable.  But being a prick about craftsmanship negates that, which turns a noble pursuit into something petty.

Just remember, software craftsmanship is a matching washer and dryer.  It does no good to condescend about your matching set because everyone you know already has one too.

Threading in Web Containers

One question was asked at the December 2012 Groovy/Grails meeting that I don’t think I answered fully.  I was running out of time and it wasn’t something I had thought about previously.  My answer was I believe correct, but incomplete.  The question concerned threading inside a JEE container.  The concern was that threading should not be done inside the container because it is discouraged and may be contrary to certain JEE specs.  By threading I mean the programmer manually launching threads, not consuming container provided threads.  By “JEE container” I mean both application servers and servlet containers.

My answer was that there shouldn’t be a problem with creating new threads in a web container, as long as one doesn’t go crazy creating threads, as that might affect the performance of the web container.  A JVM can only create and manage so many threads.  While this is correct, this is only part of the reason that manual thread creation is discouraged inside JEE containers.

The other main reason is that a JEE container is going to provide lots of services to the programmer in the form of JNDI, JDBC connection pooling, HTTP Request/Response management, JMS, entity management, etc.  The problem with manually creating threads is that you will lose any thread local information that the container needs to manage the resources necessary to provide these services.

Take a hypothetical example of JDBC connection pooling.  One way a container might implement this is to check out a Connection object from a connection pool at the beginning of performing some work.  Once it is check out, the connection object is stored in a thread local variable and used by that thread as necessary.  Once the work is completed, the Connection object is returned to the pool and the thread is scheduled to run again in the container’s thread pool.

In this situation, creating your own thread effectively disables all of this.  The container expects to provide services based on having a Connection object in thread local storage.  But, since you started the thread, the services are not available.  This is likely to cause confusion and be perceived as bugs in the container.  The easiest way to avoid all of this is to simply forbid thread creation.

If you keep these things in mind, I don’t see a problem with creating threads inside a container.  To prevent these problems, use a thread pool to limit the number of threads created (it’s good practice anyway) and don’t use any container provided services from inside threads you create.

DFW Groovy/Grails User Group Meeting

On December 5, 2012 I had the pleasure of presenting on threading in Groovy at the DFW Groovy/Grails Users Group meeting.  I thought the presentation went pretty well, there were a number of questions/comments from the audience, so at least some people stayed awake.  All of the materials I used in the presentation are at my GitHub GroovyThreading repo.

All of the code samples should run on Groovy 2.0.x on a Java 7 JVM.  In the presentation folder are the LibreOffice presentations and the notes in both odt and pdf formats for the presentations.  At the December meeting we only covered the groovy presentations, but we did not have time to get to the gpars presentations.  We’ll be covering gpars at the February DFW Groovy/Grails meeting.

Coders At Work – Final Thoughts

When I read the book Coders at Work I had one main goal: to find programming wisdom.  This may not have been the goal of the author, to communicate that wisdom, so this may not have been the best goal.  But, I was willing to be an active reader and glean wisdom from what the authors said, so Seibel’s goals did not need to match up with mine for this to work.  So the question is, did I find wisdom in the book?

Unfortunately, the answer is not as much as I expected.  This is not a dig against Seibel, I thought his book was well written and accomplished the goals Seibel set for himself.  The problem is that I don’t think that the world of software has been all that conducive to building up a store of wisdom, not in encouraging the development of software wisdom thus far.  I think there are real reasons for this, but that’s a topic for another blog post.

I do however think that the software world is becoming more conducive to the development and acquisition of wisdom.  Here are a few reasons why this is the case.

The end of the easy money software strategy.  Basically, software startups are not the quick cash grabs that they have been in the past.  Empirically, this has been the case for several years now, for an example of this, see Cringely’s column on the slowdown in startups.  But I think there also may be some realization within the industry that startups are basically the purview of big money and that we developers are cannon fodder who have a small chance of hitting the jackpot.  This is leading to at least some developers realizing that it’s healthier and more fulfilling to grow a company organically, rather than to try and grow quick.  37signals is a great example of the former, rather than the latter.

The end of the death march.  I commented in my review of the Zawinski chapter that it seems that death marches are a thing of the past, or at least much less prominent than before.  You simply can’t acquire much wisdom while experiencing death marches (other than perhaps the truism that death marches are not a good idea).  This means much more ability to reflect on the software one makes and to learn from it.  In the end I think this is a net positive because reflecting on the software one has made can then lead to improvements in that software and in future software as well.

The long term stabilization of platforms.  It’s hard to acquire wisdom about a platform that is constantly changing.  It makes no sense to do so when a given platform has a very short lifespan.  However, I see that plaforms are starting to mature and stick around for a while.  The best example of this is the Java platform.  It has been around for over 17 years now.  For the most part, it has been very stable and this means that one can acquire wisdom in working with the Java platform.  This leads to a base set of skills and knowledge that it makes sense to acquire, because it is usable and transferable.  This holds even as more and more developers use JVM based languages other than Java, such as Groovy, Scala, and Clojure.  There are a lot of skills and know how that transfer from coding in Java and using the JVM to using another JVM based language.  This is a good thing and leads to software wisdom.  Plus, it looks like the Java ecosystem may be around for quite a few more years.  One could make arguments for other ecosystems such as Ruby, Python, .NET, etc.  The jury is still out on Perl, who knows when Perl6 will be usable (is it usable now?, I lost track).

There are other reasons as well, but those are big movements in the software world that are encouraging to me.  I think the software world will be a much better place to be, and will produce much better software once we have moved on from the idea that writing software is something that one does for 5-10 years, then burns out, then manages other programmers until they also burn out.  Imagine if lawyers, engineers, and doctors also had the same approach to their careers and industries.  They don’t and the world is a better place because of it.

Coders At Work – Donald Knuth

I know next to nothing about literate programming, but I think I know why most programs are not literate programs: they don’t deserve to be and they can’t be.

Literate programming seems to be about producing a well thought out piece of software that is done in the way a book is done.  You don’t publish a book and then decide to “upgrade” the book.  Well, maybe someone comes out with an expanded edition.  Or there are college textbooks that every few years come out with a revised edition.  But each one is a coherent whole, and there’s always a bug chunk of time between editions.  The problem with most software today is that 1) No one knows what the hell they do and 2) They are never done and are constantly expanding. Maybe I’m misunderstanding the whole point of literate programming, but that seems to be the main problem as I see it.

This is too bad because I was intrigued by a passage where it was remarked that Knuth’s literate code doesn’t use a lot of functions, much fewer than a normal program would.  This is because literate programming uses different abstraction mechanisms than does the average program.  Since the point of computer programming is largely about creating abstractions, this really intrigues me.  It’s almost an article of faith among programmers that abstraction necessarily involves creating classes, objects, and functions, but Knuth does it a different way.  I like it when people challenge fundamental assumptions.  I really need to check it out.

The question I ask myself is: Why is abstraction almost universally seen as creating classes, objects, and functions? I think the answer is what I said before, code never stops growing and no one knows what it does.  The only way to keep growing a system, while still keeping abstractions, is to produce abstractions in small chunks.  The small chunks are of course classes, objects, and functions.  The universality of this derives not from some universal obviousness of those mechanisms, but from our undeniable cluelessness.

I also thought it was nice to finally see someone praise C.

Finally, I thought his distrust of black box abstractions was spot on.  When you force programming to be about using black box abstractions, it becomes a job in gluing stuff together.  When you can look at and modify code, it gives you an outlet to be more creative.  From page 581:

The problem is that coding isn’t fun if all you can do is call things out of a library, if you can’t write the library yourself. If the job of coding is just to be finding the right combination of parameters, that does fairly obvious things, then who’d want to go into that as a career?

The thing that struck me about that is how this reduces programming to a search problem and programmers to little executors of search algorithms.  Black box reuse is all about searching for something that is already done and writing a bit of code to wire it all together.  But the crux of the problem is to find things.  Find the right library, find the right configuration parameters, find the objects your need, find the correct methods on the objects, and so forth.  In some ways reducing programming to search is a throwback to the days of good old fashioned AI, when AI was largely about search, except now it’s not using AI to search, but using real intelligence to search.

And in some ways that’s almost an abdication of computer science altogether.  Instead of real insight into how to solve a problem, we are offloading the real processing to real intelligence, maybe because we have no idea what we are doing.

Recommended Reading: None that I recall.

Coders At Work – Bernie Cosell

More than any other chapter this one made me be in awe of just how much wizardry it took to be a programmer back in the 1960′s.  Patching binaries, examining memory using LED displays, using paper tape, etc. I’m in awe.  That stuff is just scary.  This of course is ironic because Cosell remarks at the end, “I don’t envy modern programmers, and it’s only going to get worse.”  This is one definitive way in which programming has changed.  It’s gone from solve intricate smaller puzzles where you know how all of the little pieces fit together to solving huge puzzles, usually not so intricate, but you don’t know how all of the little pieces fit together.

The chapter was good for seeing lots of war stories and seeing a legend in action.  I also had to chuckle that the book finally found another happy Perl user.  Like I said before, I think Perl is unfairly maligned, though I don’t like it myself and tend not to use it.

Recommended Books: The Art of Computer Programming

Coders At Work – Fran Allen

These last three are going to be real short.  It’s not that they didn’t have much to say, had I read them first I probably would have gotten more out of them.

The most interesting thing to me about this chapter was how C has done a lot of harm.  The interesting part is why.  Not because it makes complicated, buggy, or insecure software, but because you can’t produce smart compilers for C.  In a very real sense that’s true because the compiler doesn’t know what in the world you are doing.  As you start going bananas with pointers, a compiler really doesn’t have a lot of ways of knowing what you are doing or optimizing it to be what you should have done.  See for example Steve Yegge’s explanation of why OCaml compilers can make faster code than C compilers.

But I also find myself asking, In what ways do a lot of programs need to be written in C, or in some equally compiler killing language?  Is C creating accidental complexity or is it trying to solve essential complexity?  Surely in lots of cases it’s the former.  But my guess is that in many cases, probably in the cases of most of the big pieces of software that use C, it’s the latter.  And if that’s the case then there really isn’t any magic pixie dust that compiler writers can sprinkle on people’s code to make it good.  I mean if the essential aspect of a piece of code is to manage chunks of memory, point at lots of things in memory, optimize memory access, etc., then C is the right tool for the job.  A smart compiler could probably figure out how to optimize in some cases or in some problem domains.  But if the problem is essential, then compilers don’t have a chance to do these optimizations in general.

Recommended Reading: None

Coders At Work – Ken Thompson

I was really excited to read this chapter.  I am not excited to report on it.

On giving advice to his son to go into biology instead of computers:

Seibel: In a 1999 interview you talked about how you had told your son he should go into biology instead of computers because you thought computers were played out. That was almost ten years ago. How do you feel about that now?


Thompson: I feel the same, Nothing much new has happened in computers that you couldn’t have predicted.

Wow, that’s depressing.

He says he still enjoys making small programs, the kind you can do in a month or less.  He also says that he did a large chunk of Unix (or was it all of it, it wasn’t too clear) in the month his wife and kids were out of town without him.  I think there is something interesting in this.

One of the my theories about the early days of computing is that it was exciting because it was mostly low hanging fruit.  You didn’t have to know much or work very hard to pluck low hanging fruit.  I think this is why computing has stagnated, and I do agree with Thompson that it has stagnated.  It has stagnated because we keep using the strategies, attitudes, and expectations that were used in plucking the low hanging fruit.  But since there is no more low hanging fruit, there is no more harvest. So those of us in software have two choices: leave (like Thompson advised his son to do) or stop trying to pick low-hanging fruit.

Book Recommendations: None

Coders At Work – L Peter Deutsch

I don’t remember much about the chapter, even though I just read it this afternoon.  The one thing that did stick out to me is that Deutsch is an example of what I am trying to avoid.  Basically in his mid-50′s he left the world of software out of sheer boredom with the whole thing.  I don’t say this to judge Deutsch, if he wanted to leave and had the financial resources to leave, good for him.  But, I doubt in my mid-50′s I will have those resources, and I would like to make a career of some kind out of software.

If there is one thing that really saddens me about software, it’s this.  Deutsch’s story is not uncommon.  In fact, if anything lasting until your mid-50′s might mean he lasted longer than the average programmer/developer.  In almost every other profession, mid-50′s should put you at the pinnacle of your career.  You have accumulated lots of experience, plus dementia hasn’t set in yet.  It’s this group that should really be driving the profession forward through hard won wisdom and insight.  Instead, people tend to quit or move into management.  This is sad. Plus, I think this is one thing that is really holding software back.  Instead of old developers really changing things, they quit and the younger programmers continue to reinvent the proverbial wheel.

Recommended Books: None

Coders At Work – Dan Ingalls

Smalltalk, kernels, and dynamic environments were the main points Ingalls hit on.  I have to confess that I don’t have much interest in the first two.  In theory Smalltalk does interest me, but because there is so much in computer science that I would like to learn, I have to actively avoid learning certain things to keep my sanity in tact.  Unfortunately, Smalltalk is a member of that dubious list.  As for kernels, while I appreciate the practicality of having a small core, it would be way down the priority list for me if I am implementing a system.

I do enjoy working in a dynamic environment, having implemented a small linear algebra library in common lisp.  But, I think the dynamicity of a system should mainly be at the programmer level.  Although providing power to end users seems nice, in my experience end users prefer to be constrained by system.  People generally want to use software so they don’t have to think about something, and most people really have no patience to play with a system.

Recommended Books: None