Turning applications into middleware

In my post on building stable applications I referred to  RDBMS as an example of a stable problem that has a durable solution. Even though I also provided spreadsheets and double entry accounting systems as other examples, it occurs to me that this kind of “durable problem” is most often encountered in middleware. For example: network stacks, messaging systems (MQ, MSMQ, Tibco, 29West etc),  object-relational mapping systems or component-hosting environments (e.g., COM+, servlets like Tomcat or even “web” servers like WAS/IIS7).

Of course, saying “durable” just means “boring”; these aren’t real applications, they are middleware. That means that they don’t change and don’t get any thrilling new features. If they do get new features then people complain about they are destabilizing everything. But isn’t this a good thing? Aren’t we always talking about re-use of components, and how – in the future – software will be built by slapping together existing components? Well that concept has been in the future for at least 20 years, and it looks like it is staying in the future forever (and thank goodness, or we will all be out of work). We aren’t anywhere near clicking on some objects and making the backend for a bank and never will be.

Isn’t middleware the example that breaks that rule? It really is reusable and ends up in all sorts of interesting stuff and generally we don’t have to worry if the database is going to lose that data and we can save the testing effort and make sure we are saving the right data. And middleware is just the “right size” too: exposing enough to be flexible and hiding enough to be usable; unlike an abstract enterprise dependency-injection software-factory framework component that needs so much XML configuration that you worry about it becoming sentient. By their very existence, reliability and re-usability these components have shaped architectures. Indeed I’d go so far as to say that they have shaped architects. We have changed the way that we make software, because these stable problems have been solved and have enterprise-strength inmlpementations that are easily available.

Sometimes we need to be super-agile: providing just that one button the user needs or those little shell scripts you knock up just for the day, knowing that if you ever need it again it will be quicker to cut and paste. Mostly we need to remember our YAGNI and don’t solve all the problems in the world by solving the generic case with a abstract class. But sometimes, we can encounter a problem that has a genuinely reusable core that is so stable that it will last and last. This kind of problem is nearly always rooted in the real-world and connected to the way that human beings think and interact. For example: relational databases exist because humans organise information in lists that have references to other lists and indexes and so on; XML exists because even though humans can comprehend unstructured documents, computers can’t; IP networking exists because sometimes bits of the world get disconnected by physical events.

If we can turn our application into middleware, we can really start to live the reusable software dream. However, if you were at a retirement party and someone stood up and said “I’d like to give a massive vote of thanks to Susan because of her incredible lifetime of work on FooEngines. When they started out, they were just a solution to our problem with Foo, but we soon found that the FooEngine was solving a problem that thousand of developers had, and, in fact, the recurring nature of the problems of Foo were so wide that FooEngine became the backbone of many architectures of people working with Foo. More than that, people have begun to force their Foo problems into forms that can be solved by FooEngines, because they are so ubiquitous, well-understood and high-quality. It is no exageration to say that Susan has changed the way we work with Foo forever.”. Wouldn’t that seem like an achievement? But you notice that the praise sounds implausible even though we know examples where that has happened. These problems and their solutions must be very, very rare.

Again, the issue comes down to identifying a problem that is so stable that it recurs thousands of times and the problem is fundamental and intrinsic to the real world and can’t be removed. For instance, when doing business with a counterparty we use a contract, and when that contract is enforced we behave according to its rules and know that the counterparty will not respond if we don’t. We also know that if the contract represents the carrying out of a transaction – like purchasing a house – there is a well-defined moment when we are committed to the contract. Lawyers have ways of making these things work that have evolved over thousands of years, these problems are durable and highly recurrent. If we make a system that addresses them, we can be sure that the system will also be durable and reusable; if the problems are durable and recurrent enough, our system will evolve over years into middleware.

Should we try and make all our applications middleware? Of course not. Should we try and design middleware? No, I think that they are patterns and patterns are not invented, they are discovered. If the problem has a stable core, then we have opportunity to turn our application into middleware and we have a golden chance to make a piece of software that will last for decades, or even generations.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s