Patterns of Enterprise Deployment: Versioned Contract

This is part of the Patterns of Enterprise Deployment series.

Summary

A contract is my word for anything that forms an agreement between two systems. It could be formed by database schema, message format or API.

You have a new version of a system that uses the same technologies as before but in the new version there are changes of usage or naming that makes the old version and the new version incompatible and confusing to compare. In order that you don’t have to do a BigBang deployment, your system continues to support both contracts. The new system maybe distinguished by a new name, version number, address, port number, GUID etc. Under these contracts, you may or may not decide to have one code base supporting all the requests. Ideally, there would be a single source for the data if the same data is accessible  via both contracts.

Many systems for doing remote invocation support VersionedContract as part of their architecture; e.g., DCOM, RMI, CORBA, .Net Remoting, .NET WCF.

Difficulty

Low

Advantages

• Simple, quick to do
• maintains old interface, callers don’t need to update

Disadvantages

• leads to cluttered system interfaces
• sometimes it just isn’t possible to clean out the old callers; that leaves you with version 2 and 3 as well
• support burden can be onerous as the “old way” holds you back

Use when

• You have a small new feature to implement and you can re-use the old contract as the implementation
• it’s time to make a clean break with an old contract that is being stretched too thin
• you just don’t know who calls your system or when

Don’t use

• When you have already used it once.

My experiences

The classic instances of this, for me, come from the windows API. The one that stands out in my mind is the DCOM call CoCreateEx. The windows AP is full of this sort of thing because they fall under the “you don’t know who is calling you” advice. That is the one thing that MicroSoft do better than anyone else: backwards compatibility. They can support so many flavours of OS and the same damn code keeps on working. But you also see the burden of all those changes!

Patterns of Enterprise Deployment: Automated Deployment

This is part of the Patterns of Enterprise Deployment series.

Summary

An application to take the pre-built executable deployment package and dump it onto the target system in the correct state. The AutomatedDeployment may be as simple as a shell script to copy the binaries from your development machine to the test server; or, it might be consuming the output of a system that does a full clean build and migrates it onto the production server. This is somewhat different from an InstallerApplication as there is no interactive element and the emphasis is on rapidity and simplicity.

Difficulty

Medium

Advantages

• repeatable
• automation means you can concentrate on something else, especially if it is time consuming

Disadvantages

• a fair amount of upfront work
• slow to react to changes to the architecture of the system
• can be dangerous; manual changes take time to be destructive; a deployment script that goes awry can wreck servers in seconds
  

Use when

• there are several steps in the process of moving built executables to the running server
• you perform the deployment many times; you may be deploying to a test server 20 times a day

Don’t use

• When your project is changing so rapidly that you don’t know what it will produce
• if you haven’t tested the automation very well and the target environment is unstable (e.g., many target machines and many versions of OS or other people are also using manual deployments)

My experiences

Coupled with a DeployFromSourceControl this is a good thing to have for agile developers who like to build the head of the trunk and stick if on the server. I also use it a lot when I’m working with a test server; i just want to grab the binaries that I just build and restart the service on the remote machine. It takes a few minutes to knock up a powershell script to do this, but it saves so much frustration and really, drag-n-drop is very slow! But, this really isn’t the same as a production quality script. If you have to use a script to deploy it is best if that is the only method to deploy the application; otherwise minor changes will creep in when people don’t follow the same pattern and the script won’t work properly. Worst case it will half-deploy and leave you with a system that you will have to recover by hand.

Patterns of Enterprise Deployment: Deploy From Source Control

This is part of the Patterns of Enterprise Deployment series.

Summary

DeployFromSourceControl is used when you are wish to create deployed packages only from source controlled code. Normally the pattern will be implemented by a system that will ensure that a known version of the source is obtained – often the latest – and then build into a deployable package. Many levels of sophistication are employed from batch scripts to full-blown event-driven service centres with web front-ends and deep integration with work-tracking project management systems that produce full audit trails.

Difficulty

Medium up to a full life’s work

Advantages

• Repeatable
• if it isn’t in source control then it won’t get built so there is no danger of losing changes when you re-deploy
• you will, with varying levels of effort, be able to reconstruct any package that you build by examining source control timestamps

Disadvantages

• a fair amount of upfront work
• slow to react to changes to the architecture of the system; you need to make a second change to the deployment package builder
• can be slow; developers must check in then wait for the changes to be build from source control

Use when

• building the deployment package from source control is almost always essential on a serious enterprise project. In any case, this is valuable intellectual property; check it in!

Don’t use

• When your project is changing so rapidly that you don’t know what it will produce

My experiences

This is a crucial one for any project, but don’t feel that you need to be constrained by the hoary old “build script”. As long as you are only building your deployment packages from something that is in source control you can do anything you like.  One nice twist that I’ve always wanted to try is to use a continous integration system – like CruiseControl.Net – to provide the built artefacts. You know that the stuff is in source control, just that it is a little fresher than normal! If you don’t like this and think that it is too unstable for releasing then you can run the CI build on a very stable branch that you only check into by merging in from the development brnaches.

If you can get a dedicated system for this then when you start integrating with the rest of your development process tools then things really start to look sweet. If you can get the whole TFS stack running well, then Team Build provides a huge amount of integration with work items and changesets/checkins and produces reports for each build in the staging area and means of tagging the build that is ready for deployment.

See also

AutomatedDeployment

Patterns of Enterprise Deployment: Just Copy The Binaries

This is part of the Patterns of Enterprise Deployment series.

Summary

JustCopyTheBinaries is what you do when you don’t have an enterprise deployment problem at all. The binary – or other executable files like javascript files – are simply constructed on the developer machine and copied using the file system to the machine where they will be used.

Difficulty

Very Low

Advantages

• quick
• easy
• always works; even if you have to craft a special hack on your laptop when the power is off in the server room

Disadvantages

• little or no control, auditability, repeatability
• development machines are not the place to make clean builds
• When the developer leaves the project or their box blows up, then deployment is difficult

Use when

• you don’t need anything else and you are working on an uncontrolled or or pre-alpha prototype project
• all the work is done on one machine

Don’t use

• When you are creating a real system, even for version 1.0

Patterns of Enterprise Deployment: Introduction

Deployment is a neglected topic. It sits on the project diagram as the line that separates development work from support work. It is rarely given any time in development and even less for the testing. Just as Continuous Integration stresses that if you can’t build your project, then you don’t have a project; if you can’t deploy your project then you don’t have anything.

This will be a set of articles discussing the topic of enterprise deployment; not the kind of thing that you do on your own website where you JustCopyTheBinaries. That might work for the first version where you don’t have any users but it won’t be any good when you have a mesh of consuming systems and users in 4 times zones. It just isn’t good enough to tell people who have systems that have just stopped working on quarter-end that “Oh yes, that was a breaking change”.

Often, in the enterprise, you don’t even get the luxury of a trivial v1.0 deployment. More often than not, you have to replace an existing system; but support its bugs for a while even though your new system is event-based and the old system runs as an overnight batch. You just have to make it work, real people doing real work are depending on these systems to actually make the money that pays for your servers. Shocking, I know.

And deployment is something that never stops. You can’t “finish” the project with a deployment. In the enterprise world there are always new features and new servers in new offices that need a deployment; if you don’t have the deployment under control then you will be wasting your time and your customer’s time; with downtime and debugging.

Maybe, downtime isn’t the problem for you; maybe you don’t have a lot of timezones so you have all night, every night to do what you want; but you do have a set of users who want new features every month. Maybe you work on a non-critical application like a social web site; you are on 24×7 but you can tolerate downtime but the website can’t be off for more than 10 minutes; you have to ship new features every day and you have a development team who demand that the new work goes live from source control to the production system. If you have to do that, you have to not only have a system that allows you to DeployFromSourceControl but you need an AutomatedDeployment.

So, this set of articles will deal with a vague set of problems from the actual “go live” moment to how to architect your applications so that they can smoothly take over from a predecessor system. The systems under consideration will range from databases to web services to client applets.

This set of articles was inspired by the Martin Fowler article on a type of deployment that he called the StranglerApplication. It seemed to me that there were many such good tricks that could do with codifying.

First things first, second things never

The title says it all. Everyone is trying to show they are good at Getting Things Done. People invented the waterfall system of project management because it was a hell of a lot better than no system at all. What agile does is cope with a piece of software that is never done, but always improving.

It works by forcing the users to decide what things are first and should be done before anything else. It not only puts first things first; it allows the things that are first this month to be different to the things that are first next month. Of course it isn’t easy,  it requires software developers to give up the pretence that they run the software projects. After all, the person who truly runs the project is the one that gets to decide if it was a success, not the one doing the work; but you can pretend otherwise – for a while – when you are the “project manager”; it sounds like you are in charge. It also requires developers to learn how to make software that is flexible and changeable and testable; but if they can’t do that then the whole future is buggered anyway because whether you are doing agile or not, your requirements will change.

The title is a quote from the 1970’s household productivity classic SuperWoman,  by Shirley Conran. She also said “Life is too short to stuff a mushroom”; but that seemed slightly less relevant.

C.S. Lewis, sly Christian propagandist that he was *, put it slightly less clearly, but more elegantly:

You can’t get second things by putting them first; you can get second things only by putting first things first.

You can’t always get people to believe that they can’t have everything, but at least with agile they get to have something now and something else later, rather than having to wait 8 months to find that the whole thing is missing the point as the new business is now over there <points to the other side of the world/>.

* I must say that the whole Christian metaphor of The Lion, the Witch and the wardrobe didn’t pass me by when I read the book, but it didn’t occur to me to be offended; after all, it was only a story and the whole bible thing, well that was just another story.

My first real agile experience

I had a spontaneous agile moment and I can confirm to those at the back: Agile Works.

Working with two other guys on a project, we agreed with the manager that in one month from now (ooh agile, a sprint) we would demo it (ooh agile, a sprint demo). The demo would consist of putting simple data into a lightweight  front-end and going all the way through to the database and back (ooh agile, a “full slice of cake” like a user story). During that month, we evolved (ooh agile) a way of working that was very effective; we would meet up each day at about 9:30 (ooh agile, daily standup meeting) and talk about the results of yesterday and what we would do today. At least 25% of the time was spent working together at the same computer (ooh agile, pairing). We had a continuous integration build using CruiseControl.Net (ooh agile) running unit tests (ooh agile) and some integration tests (ooh agile). The CI build was also doing some static code analysis (ooh agile) using Simian.

And it was really working. OK we didn’t really have the code coverage from the tests, and the code was, in places, not only a bit dirrrrrty but it was a little heavy on the singletons (the hackers favourite). And we had a few code smells like interacting tests due to all the tests running off the same database. However, we kept each other honest with the pairing and even if we weren’t doing TDD then we were writing fairly OO code that had at least a few tests per class. And, we were really cranking it out. 3 people,  1 month and nearly several thousand lines of code written. It was classic “three guys in a garage” stuff.

And then, of course, the wheels come off.

I got pulled onto something else, and the project got split in 2 so everyone now working on their own decoupled piece. And when I checked back after a few days I find that; quelle horreur it’s all gone wrong. 80% casualty rate on the automated tests; 20% progress on a deliverable and the rest of the time spent changing the structure and putting out the resulting fires. OK the changes were legit; required to get the foundations right and even agilistas like that when there is a business case for it (and there was).

So what did I learn:

• Focus matters: One person doing one task is much more productive than one person doing two tasks. It sounds obvious but I really believe that the difference is huge.
• Teamwork matters: Two people doing one task are much more productive that two people doing two tasks. In this “controlled” experiment with the same people doing the same tasks the difference was significant. The extra person keeps you honest and keeps you on track. They stop you getting distracted by other non-project tasks but more subtly they give you moral support when you get stuck.
• Test-first matters: working alone it is easy to slip and not test first; or even have a test in mind. Headphones on and in the zone you can be hundreds of lines of code in trouble before you realise that you aren’t even going to be able to test this. If you can’t make a test, you don’t have working code.
• Deadlines matter: A 4-person team working on a one-week deadline work harder than one person on a one-month deadline. The tight deadlines of agile are important; they Keep It Real and keep it scary. You won’t get scared when you have a 3 month deadline until you run the numbers and see that – even if it all goes to plan – you have 100 days of work to do. And even then you won’t be scared until you realise that in each calendar week you probably only get 3.5 days of solid work in.
• Mood matters: when we were cooking with gas, it felt good. We were all upbeat and had a great positive feedback loop going; the more productive we were the more we wanted to get done.

The beginning of the end?

So. Last week I had an experience which I think was a significant step in my programming life/career/whatever. I realised that I was about to rewrite my first big application. And when I say re-write, I don’t mean v2.0. I mean: I was faced with a problem; I came up with a solution to it that was quite specific; after a few seconds of thought about how to implement it I realised that I had already implemented it years before. As far as I know that system is still in 24×7 operation with the kernel of the code untouched by other developers. So it was clearly good, so why not just re-use it? That is good, right? We want to re-use code. Or even if we can’t re-use the code we want to re-use the pattern (in this case it was using .NET attributes to decorate strongly-typed classes so that they could be turned into loosely-typed data-transfer objects). All good, right? Even if I couldn’t use the code, I knew that the concept would solve the problem and solve it well enough to last years of subsequent tweaks.

Well, in spite of that, it didn’t feel good. It felt like the chill of old age. It felt like this is the beginning of the end. There are no new problems to solve, they have already been done, even by me. All the rest of programming is just repetition, automation and abstraction. Of course, this is probably what it has always been; ever since Turing became complete people have probably been sighing over the fact that they are doing the same thing as they were last year only faster. The shocking thing is that my father used to say the same thing; having worked on minicomputers in the 60s for doing critical path analysis he was forever telling me that “computers haven’t changed”. He was wrong of course; the scope of the problems had changed because there was 100,000 times more processing power, but I suspect that the programming problems were just the same.

The question is, do I still want to keep doing the same thing over and over? Just a little bit faster with less bugs than before. I wrote another application this week. It was, I have to say, well structured using all the tweaks that ReSharper could throw at it. The methods were tiny and readable; the code was flexible enough to cope with change but it wasn’t over-designed; I cranked it out in 4 days but it ran first time (or maybe second..). There are, as I’ve said before, no points for difficulty and the business people who wanted it would probably be happy that it got done in a week with no problems. Next patient, the doctor is IN!

But how long can it go on? The same tired old business problems; more reporting tools; more transaction processing; more stupid little GUIs with one more button to handle special case X. I mean, you can’t avoid that. The world moves on and programs rot. You can’t be writing earth-shattering middleware every week, and as I’ve said before, the stuff that becomes the new backbone of an architecture doesn’t get written, it gets extracted from a running application (e.g., Rails). But seriously, how long can people stand to do the same thing over and over. A little more abstraction isn’t enough; to get from c++ to MFC to c# in, what? 20 years?

All I can say is that I won’t be doing this in 20 years time.

http://en.wikipedia.org/wiki/Critical_path_method

The problem is not the problem

When someone starts to tell you about something that they need to get done, already you are thinking about how they can solve the problem. You are thinking about technology and all the things you can do with it; they are thinking about the task that they need to perform.

Actually, they really aren’t thinking about that. They are thinking about what they need to do to get a pay rise; they want to get a promotion; they want to improve the performance of their team so that their team doesn’t get taken over. So with all these things in mind, is it any wonder that they can’t turn out a decent spec, or a decent set of requirements; they can’t even answer questions like “what do you want it to do?”. They do know what they want it to do: they want it to get their boss fired so they can get their job; they want it to get their kid a dog; they want it to enable them to leave on time.

As my old metalwork (*) teacher used to say, first define the problem. The real issue here is getting people to tell you what they already know, then capturing it in a structured way that you can turn into a list of the nouns and verbs of the system; and while you are at if, figure out where the gaps are. And you have to do this in a language that you can both understand and you have to do it while carrying on a normal conversation.

Mostly, people fall into two groups; those who think that computers are amazing things made of spells and spiderwebs that can do anything and the people who program them are able to get the computer to do anything. The other group is still in the dark ages of desk calculators and any time the application does anything they want it to spit out everything that is related and not do anything until it has a good few clicks to encourage it; they are the ones who always want another “are you sure?” dialog box.  There is more than a grain of truth in both camps, so be gentle with them.

Actually, the real, real problem is getting them to engage with you at all. The real problem is making them see that if you don’t know what you want, you definitely won’t get it. The problem is, as all interesting problems are, a people business. Getting people to tell you what they want and then explaining that they can’t have that, but they can have this other really interesting thing that they didn’t even know that computers could do is all part of the fun. Actually, it is all part of the fun working on any kind of agile team. I pity the poor suckers whose only contact with the client is the bug reports that are fired at them by level 2 support (and they don’t even get to see level 2 support, they don’t even get an email from them; they just get a bug feed).

As with all people problems, there is no no simple solution and rarely is there some sort of break point where it all becomes clear. With people problems it’s uphill both ways and the wind is in your face. We should be grateful that all we need to do is get people to talk about what they want out of their computer systems and not what they want out of life. Anyone who is a real manager – or even better an HR manager – knows that is the real secret sauce of a productive organization.

In any case, the problem is not the problem; the problem is getting people to realise that unless they face their problems and take responsibility for the solution instead of handing it over the IT butler to sort out, then they won’t get what they really want.

* – Actually, he wasn’t my metalwork teacher; by the time I was doing it, the subject was called Design & Technology. He was a rather odd chap was Mr Tooze. He preferred to sharpen his pencils with a piece of sandpaper and then wipe it on his sock; oh and it isn’t called sandpaper it’s glasspaper; and he had a tic in his face tha made him look like he was blinking; and he spread his glue using carpet cutoffs that he kept for the purpose. And whenever he heard car tires screeching he would always say “I know that driver is under 21 years old” and then proceed to tell a story about how he had been the first on the scene of an accident where a motorcycle had hit a sportscar head on; the motorcycle had cut the car in half and the irst par of the motorcyclist he found was his foot.

Waterproofing is waste

Following on from previous rants on developers doing work for the users and not spending time working on things that are just for their own satisfaction (see Productivity Over Perfection and Quality is a Feature). I’m now turning my attention to people who are coding their applications as if they were writing the core of the .NET runtime. All the endless perfection of the object model and obsession over performance; all those little things add up. And what do they add up to? They add up to features not delivered. They add up to people not getting the software they need to do some real work. You know real work; it’s the stuff that people do when they don’t work on software. You know like getting people’s pensions paid properly; or getting those reports out in an hour so the accounts team can go home on time; or adding that extra link to the mapping widget so people can track their orders.

We all know that developers love developing; what they particularly love is developing little things that no one needs. This goldplating is not really working, it’s playing and you don’t do crossword puzzles in work, do you? However, small things are waste as well; waterproofing something that will never see water is a waste. So is endlessly checking input parameters for being null or endlessly perfecting the layout and naming of classes. You know what: write a test, make it pass. If you have some spare time when there is literally no other work then you can come back and play. Or more likely, go home on time yourself, for once.

In the end though, when the project is over, will people remember that it was delivered late or will they remember that it was perfect when it finally showed up? It may depend on how often you do it. If you are always late then people might start getting the message and asking someone else who can deliver something good enough, but deliver it on time.

The polishers will tell you that their stuff is so much better that it justifies the extra time. This is pure sleight of hand. You just can’t tell how a piece of software will turn out, especially if you are developing business software and running to catch up with a changing set of requirements. If you don’t deliver approximately on time, then no one will care about how perfect it was; they will only care that the requirement has now gone away as the business has moved on. And frankly, to anyone who isn’t a nitpicking nerd, perfection is simply irrelevant, uninteresting and unobtainable since it all changes next year when the new law/manager/technology comes in; having working software is much more important.

Remember this isn’t hacking; this is being good enough, but no more. If you are writing cryptography software then good enough has to be very good indeed. If you are writing a shell script to copy some files then copy ‘n’ paste is probably your friend. Of course, the real trick is to know how good is good enough? If you are working on a stable problem then you can take your time and keep it clean, but there are always little corner cases that won’t lie down so perfection will keep retreating anyway.

If you are spending excessive time waterproofing your code then you are wasting time and money, and if you are a paid employee, neither are yours to waste .