Moral Map of the Web

Enjoy this.

I liked this part, where MS is a dark, satanic mill, vomiting poison into the pastures of the web. Harsh.

And I also learned a new phrase: Paving the cow path

Mostly I think of technology as a job. Then when I read something like the UK government digital task force blog, it seems like a noble quest. Making the world a better place, one crappy government department at a time.

PowerShell: quick custom object

TL;DR: I parse some filenames and pack the results into a quick custom object.

I found that I had a lot of files to go through, they were all dated very nicely:

Sadly, some of the dates in the filenames didn’t match the date modified, so I can’t use that to sort them.

And I can’t use the filename because of the date format doesn’t sort correctly. If only they had nicely formatted them as 2014-02-28, then all would be good.

PoSh it is then:

#get the files into some data 
$files = gci "\\folder"

# grab the names
# pick out the date with a regex, remember to out-null to get rid of the True, True, True, True... that will come out of the pipeline
# then take up the matches using the $matches object, packing them into a little hash table, the hash will work as a hobo object
$result = $ |% {$_ -imatch ".+-(\d\d)\.(\d\d)\.(\d\d\d\d)\..+" | `
Out-Null; @{ "date" = (get-date -Day $Matches[1] -Month $Matches[2] -Year $Matches[3] ); "filename"= $_}}

#select that out, like a boss, find the file you've been dreaming of
$result | select {$_.filename}, @{e={$}; l="date"} | sort date

PowerShell : slow data in a file

Say you have a script that gets data from a lot of different places, it’s slow to run when you are developing but you probably want to connect to them all when you run for real. If that’s you, then you can save the results of your slow connections in a file with just one line of script and keep using that file unless you have to re-get fresh data.

I’ve found using PoSh’s Export-CliXml and Import-CliXml to store the results from a slow operation nice and easy, but it can output any PoSh object.

 = (join-path $PSScriptRoot
 $resultsFileRedo = $false
$bigListOfThings = get-ListOfThings
 write-host "about to get the slow data from $($bigListOfThings.count) things"
 foreach($eachThing in $bigListOfThings)
 write-host "doing $eachThing"
 $eachThingData = Get-SlowDataThatTakesAges $eachThing
 $slowData.add($eachThing , $eachThingData)
 del $resultFile
 $slowData | export-clixml $resultFile
 $slowData = import-clixml $resultFile

Essence of Agile

You can, of course read the 12 principles of the Agile alliance; sometimes I find them hard to swallow. I also can’t remember more than 5 things.

Here’s my take:

  • Smaller packages of work delivered more frequently
  • Subject matter experts working directly with technology builders reduces the risk of waste

And that’s it.

Of course, it isn’t. I can’t say anything succinctly.

Smaller packages of work, delivered more frequently

“smaller” is a relative term, and Agile is a relative concept. The idea is that you try to make and deliver the smallest thing you can. The smallest piece of work you can do independently of other pieces of work. Why do we do that? Well firstly and most importantly is that it seems to work; and works much better than any other approach. Smaller packages of work means that there is less overhead in tracking and staying focussed, less problems of scope creep, shorter meetings and shorter documents.

Also, builders don’t try and solve problems which you aren’t trying to solve, so code can stay lean. People who are smart may start solving other problems that they “know” will come up; in Agile where the problem is evolving then you don’t “know” anything, you are doing the wrong work.

So here’s where some people get lost. They think: “whoah. You should stop the problem from evolving and nail it down before you start to work!” Well that sounds great, but it’s proven not to work in practice.

The problem is evolving for 2 reasons:

  1. The world is changing and you can’t keep up by doing everything up front, you just can’t.
  2. People actually don’t know everything about the problem when they start

You might think that it’s the best thing to dive in deeper and deeper until they do understand it all, but you are wrong. Making people wait for easy and urgent parts of the system while you analyse the unimportant but very complex parts of the system means two things:

  1. You end up with a design that is bloated and complex and heavily clipped to the problem as it is today, it’s not going to evolve to vNext.
  2. Your company will go bust because you don’t have a system that allows them to track orders or trades or whatever.

“delivered” is also a subjective term. What you “deliver” in your project as scoped, designed, coded and tested might not be what goes live this week. You might have 2 different definitions of “delivered”, with 2 different sizes of a package of work. Really what might be happening there is that there are 2 back to back agile teams, one making features and one rolling them out. Or it might not; see Kanban.

“delivered” also means like really delivered, you have to understand that being Agile doesn’t mean cutting corners. It doesn’t mean doing less design or less testing or less documentation of those things. You just do them in a smaller cycle. It’s not just about being done, it’s about being done done. Which means really cleaning up after you, so when that piece of work has to change and adapt, as it probably will, they aren’t flying blind. The absence of a spec and a BDUF means that you have to back fill as you go.

“more frequently” is also a relative term, and it doesn’t mean faster (see next point). It means that you “ship early”, yes; but you also “ship often”. You have to repeatedly deliver, only the act of actually delivering something means you understand the process of delivery and didn’t just get lucky. If you don’t deliver repeatedly you aren’t removing risk. You have to give yourself a chance to learn from your mistakes. And, repetition makes you automate and automate and automate.

Work that is in the pipeline and not delivered is the same as Amazon having 5 million TVs in the warehouse; Amazon had to pay to get them but they aren’t paid for them while those goods sit there, depreciating in value.

Subject matter experts working directly with technology builders reduces the risk of waste

Really this is where Agile is faster; it isn’t any faster to design and build software incrementally. It isn’t slower either, not really. The real advantages are in the lack of waste. You never build the wrong thing, even if you need to redesign it.

“subject matter experts” are not analysts or testers, though they have to do a bit of both in the micro scale. In the classic waterfall model, you would have subject matter experts (what we sometimes call “the business”) working with analysts who are skilled at analysing things (e.g., MECE, 5 whys, etc) but aren’t software people. The analysts work with architects and testers, and they work with developers… (and no one works with the sysadmins who run it in production, of course J). What we change in Agile is to shorten the supply lines and bring the work suppliers (the business) closer to the consumers of work (the builders). That’s great as there is less confusion and better communication.

It’s harder though because everyone needs to be better at communication, more multi skilled, they need to be “requirements engineers” and not be drawn into too many corner cases, they must be their own best architect, a solid tester and they must cut code as well. Not everyone needs to be a master in them all, but you need awareness and close proximity to someone who can help.

“the risk of waste” is high in a waterfall project. You build too much without checking it works together and in live. What makes Agile work better is the small package of work. There is less waste because you stay focussed on what’s important and that smaller package of work is easier to analyse and easier to code and test correctly.

In technology all decisions are reversible*, it’s just a matter of how fast you can reverse and how much you waste. Rework is essential to Agile, and not something to avoid. Rework isn’t waste, it’s sharpening the saw.

That’s the thing: software work isn’t like building it’s like designing. And if you’ve ever worked with a house architect or a graphic designer you’ll know that you repeat as often as you need to and feel happy that you are getting what you want!

“working directly” means direct, face to face conversations, ideally in the same room. Sitting together doesn’t mean that you get the trust, openness, candour, belief in each other and sense of fun that you need to make an outstanding Agile team. But being together and repeatedly delivering small pieces of work gives that confidence in each other, and its real confidence. Not acquired on a rock-climbing course or a boozy night out, but real trust in each other’s ability.

What goes wrong?

Well nothing’s perfect is it?

Working in the microscale can cause some problems. If you are just starting a project, operating with a team that is unskilled at Agile and there is little to guide you technically, then you will make some wrong turns, for sure. Doing Agile design is a skill, and when you do it for the first time you might get it wrong and design too much or too little. But if you pick a tech and a design that is “normal” and composable, you will probably be able to fix it.

You might not need an overall architecture, though that will help if you are in steady state agile delivery. You might not need an overall “guardian” of the quality, but then again, you might. It depends on the relative pressure for delivery from the business, the overall skills mix of your builders (i.e., their tendency to under- or over-design).

One of the things that can go wrong when the team is new to Agile and the tech/builder team is politically much weaker than the business team is that the business team thinks that this is a licence to do whatever they want and get something for nothing. Separating out tech work into separate stories that never get done is a sign of this. Plus all the normal bad smells of imbalance: project crunches, tech people of low status, blame games, etc. Just like waterfall projects, you will not know that you have a problem until the end and you look back over many cycles with code quality declining and support issues draining the team.

Correcting that balance can’t be done with a process only with communication and the building of trust. But the iterative process gives you a chance to revisit the start of work over and over again, and start fresh.



*as opposed to medicine, where limbs don’t grow back. Let’s also assume that the bad decision doesn’t bankrupt the company.


There are 2 types of meeting. Discuss.

Meetings. Terrible things; but like democracy they are the least-worst way of solving the problem*. They can be a terrible waste of time and they are the only way of really getting the best from those terrible, frustrating, wonderful, inspiring, brilliant people that are our co-workers. They are the best high-bandwidth, bidirectional, audio-physical media that works for our 3d simian brains. How can we get the best out of our meetings? I think that the answer lies in that excellent reasoning device: the barbell (i.e., there’s something at either end and nothing in the middle). There are 2 types of meeting:

  • The Decision meeting
  • The Discussion meeting

Once I’ve said the names, there isn’t much more to say, but let’s grind through it.

1. The Decision meeting

The Decision meeting is pointy. The ultimate example I’ve seen is the decision meeting for an investment buy/sell. There are defining characteristics of the decision meeting:

  • The point of the meeting is to make a decision, the decision makers should know who they are before the meeting
  • In order to stop debate, the decision should be laid out in a document, before the meeting
  • In order to stop groupthink, each decision maker should make and submit their decision before the meeting
  • There must be strict record-keeping; minutes and decision notes must be recorded consistently
  • In IT, we like data. Recorded decisions should be mine-able and have a confidence level. Or use the Apache voting system.
  • There must be rules of engagement; the format of the meeting should be known and every point covered.

If all this is done before the meeting, what is the point of getting together? Well, you establish the agenda, set your own opinions and then drill into those opinions and see if anyone needs to change their mind. If you’ve ever watched The Apprentice, you know what happens when you get people together to make a decision without a strong, reasoned opinion. You get a shouting match and then vicious reversals and selective memory after the fact. Don’t do this. Of course, all this is great but you must do a lot of work to get there. The hardest thing is building a decision to decide on. You have to put together an exhaustively documented plan of action/architecture/strategy to vote on. That must be close to what people want or you’ll be lost in debate. PGMs are good because the answer is always buy/no buy. The other thing is, this decision meeting doesn’t have to be a democracy: the decision maker might be one person who ignores everyone else, but if things go bad then the documentation is there to fall back on. Meeting minutes are a cleft stick; vital and yet timewasting they deserve a post of their own. One thing is certain, publishing meeting minutes is not a way to communicate what you are doing, you have to do something to make them digestible.

2. The Discussion meeting

The Discussion meeting is schmoozy, smooth and free-flowing. In order to get the best out of this you really need an agenda, if you don’t have an agenda you better all know the problem well. You might be having some early stage meetings where you don’t even know what the agenda might be, but you better have an opinion before you walk in the room. These meetings will be useful when you are hammering out what the options are, or for finding out what the problems are. I’ve found that if the meeting group is always the same and you keep talking about the same focussed subject-area, you soon develop a team spirit and a shared context and vocabulary that enables a productive sharing of ideas; if the meeting attendance changes or meetings are infrequent productivity drops drastically as we need to reset each time.

There is a special case of the discussion meeting that is, in my opinion, the best type of meeting: the one-to-one.

These are the meetings where you share ideas freely and inspire each other. This is one case where it might be OK to not have an agenda, the very act of having to talk it out will clarify things and throw up angles and aspects that you would never find alone. See also Programmer’s dog. This is where I and the core values split: I think that brainstorming without focus is a disaster, but balanced, thoughtful, egoless collaboration, where you abandon your idea when a better one comes along… for a certain type of creativity,  there is nothing like it.

And of course, finding out how people tick will only happen in this type of meeting.

Types of meeting to avoid

  • The status update: Utter fail. Send me an email. Or don’t, even better.
  • The boxing match: One person talks, someone else argues with them. Everyone else listens and checks email on blackberry.
  • Any meeting that DOESN’T START ON TIME
  • Any meeting where people check emails other than to make the point that “this meeting is pointless”. TBH, sometimes I do send myself reminders from a berry, but until we have in-office multi-user Siri in every room…

Conclusion Know your meeting, value your colleagues time. Choose one or the other and make sure they know which it is.

*  Churchill said that. Amazingly here are the Hansard minutes that say so. Meeting minutes are a wonderful thing. (see first few lines in the linked page for comments on democracy)

Want better ideas? Stop snacking between meals

I have an observation to share with you: the invention of eating meals is the cornerstone of civilisation and all its works.

A bold claim but I’m sure it’s true. As the frequently shamed owner of several children I can tell you that when you are in contact with an environment where you are distracted by something bleating at you, the quality of thoughts really goes down a long way. There is a point at which you are being distracted so frequently that you cannot even remember what you were…umm..Anyway. So, being distracted by the constant bleats of hunger and the need to pick berries, be alert for tiger growls, children falling into streams etc. means you never really come up with anything new. So, hence the invention of mealtimes; once you have mealtimes you start to enter into a rhythm where you do one thing (gather food) then another (sit down and eat) and then another (sleep, with full bellies and empty heads). Once you are down that road, five minutes later Adam Smith turns up with a pin factory and we are all happy.

So what’s the point? That the working environment that we have built in this generation is heading for the hunter-gatherer start of constant alert with IMs and emails taking the place of tiger growls and status update meetings taking the place of passing wolfpacks.. And we can’t think straight when we are out in the rain with tigers after us. So what do we do?

Here’s my idea: stop snacking on ideas and start eating meals.

Need some time to innovate? Then try booking some time to do it and make sure you commit to it. Make a meeting booking and block out the dreaded calendar. When you are in that time, make sure that you don’t do anything else. Don’t answer the phone, don’t answer IM and, my goodness, finally take the blue pill and unplug that email.

Personally, I think that the only way to innovate is without a computer. The computer is just rails for my mind to run along, and personally I just can’t think unless I have a pen in my hand. I like to book a meeting room and sit in it with no blackberry and either alone or with one other person draw pictures of what winning looks like.

If you really do need your computer then I suggest that you get some noise cancelling headphones and/or some J. S. Bach (or Gregorian chants, or whalesong or surf sounds or this excellent stuff: which has melody but no form, try to ignore that it is birthing music) and cancel out those tiger growls.

What you also need is a sign that says “Touch me not!” Wasps have their stripes and you need some too. My own option is the “thinking cap”; when i wear it people know not to talk to me. I used to share a desk with someone who liked to chat and sometimes she didn’t see the hat. However, when I simply didn’t answer she would look up, see the hat and shut up. And actually that worked, I was able to keep focus for the 5 seconds until she realised that I wasn’t going to answer, I couldn’t advance, but equally I didn’t drop any balls (I was a Thread.Sleep but didn’t lose any state…) and could continue from where I left off.

This is really the key: if you want great creativity, you need a flow state and that takes time.

If you think you don’t need to innovate just more execution? Well, maybe that is your current problem, but we will spend our whole lives in the future, we should prepare for it. Make time.