It’s not big and it’s not clever
This is not groundbreaking stuff. You’re not going to read anything particularly clever or complicated here. It might even seem too simple. It’s possible that you’ll conclude there’s no difference.
However, I urge you to look more closely at how this simple change can help you avoid the issues that plague IT estates. Issues that include multiple sources for the same information, spaghetti integrations, vendor lock-in to full stack enterprise solutions, and complex interdependencies to name but a few.
All issues that make change difficult and reduce the agility of your organisation.
Going underground
If you look at any set of systems in any organisation, you’ll usually find a common architecture has been applied. There are many variations, and advocates of each will extol the virtues of their approach over the others. However, when you step back you’ll see they’re all very similar in one key respect.
All the plumbing is underneath (or in the walls), hidden from the user, just like the plumbing in your house.
A user interacts with a system to achieve a task. This system exchanges information with others and the user gets what they want. Just like in your house. You turn on a tap in the kitchen and hot water comes out. You don’t particularly care where it came from – it’s just there when you need it. The fact that it was heated up in advance, piped to a tank then piped back down to you later on in the day is not something you have to worry about.
When you look at IT solutions from this perspective, everything seems to make sense. There’s even a whole industry dedicated to it. We call the companies in this industry system integrators. These are the plumbers of the software world.
But, does it really make sense?
Brewing up
We often accept things simply because they’re familiar, but that familiarity can prevent us from seeing other, potentially better ways. The plumbing analogy might make it feel right, but what happens when we look at it from a different perspective.
Imagine now, that the task at hand is for Bob to make a cup of tea. If we apply the IT analogy in reverse, Bob will be provided with a “tea tap”. Turn it on and out flows tea, just the way Bob likes it. Or maybe Bob wants milk in his tea this time, or his drink of choice today is coffee. What now? Different taps or one very clever configurable tap with several input controls?
Of course not. What Bob needs is the flexibility to do a wide variety of things using the same basic household systems. In the case of the tea, he fills a kettle with water from the plumbing system, then heats up the water by powering the kettle from the electrical system. What he doesn’t need is for the plumbing system and the electrical system to interact.
The systems, in this scenario, are providing the raw materials. The low level components that make up the solution. It is Bob who combines those systems together using a process to create his drink of choice. Bob is the integration. If he could get a robot to do it for him, all the better, but right now that’s not available.
So, how does that translate into an IT architecture, and what’s different about it?
Off with his head!
Probably the biggest difference in this architecture is that your complex subsystems just aren’t that complex any more. They don’t talk to each other, and they don’t handle any business processes. All they do is store information and handle local transactions. From now on, I’ll refer to them as Domain Systems. The real strength of a Domain System is its single minded stupidity.
A domain system is headless. It provides no user facing functionality (other than for system administrative purposes). It is fronted by an API through which everything it does is accessed, and it is the master datastore for all the information that belongs within its domain. No other system contains any of this data.
A Domain System is purely transactional. You ask it to do something and it does it, without talking to any of the other domain systems. That means it can only act on the parameters you pass it and the data it holds. You can put data into it and get it out again, but if you want to combine that data with information from other domains, it’s up to you to do that. You can’t ask the domain to get it from elsewhere. Domain systems don’t do aggregation.
For example, if a Domain System exists to manage customer information, that’s all it contains. Information such as name, address, and contact details are inside its domain, but things related to the customer through your business aren’t. If you’re a bank, the Customer system won’t contain account details. Those will be in the Account Domain System.
Let’s take payments as another example. A payment Domain System will do just that – payments. Ask it to make a payment from one account to another and it will do it, and nothing else. It can’t go to the account system for account details – those will have to be supplied to it via the API. It won’t update accounting systems or generate invoices, nor will it send information elsewhere or generate events that drive other business processes. It just makes the payment.
So who does all that other stuff?
Tea, but in a business process way
This is where the separation of responsibilities takes place in this architecture. Remember our tea example? The systems were stupidly simple – a system to supply water, a system to supply power, a kettle to boil water, storage for the ingredients and a process handler to bring it all together to make tea – also known as Bob.
Away from the Kitchen now and back to the world of IT, what does this mean for our architecture? As we’ve already stated that Domain Systems don’t handle process or aggregation the implication is that all of that takes place “above the line”. In other words, we have a second layer containing user facing activities. An activity is initiated by a user and concludes when the user is ready to step away from that activity. These activities are grouped together into user systems (or Products).
Products are responsible for the interaction with the user. They handle the flow of the user activity, the martialling of information to support that flow, and the triggering of any required transactions. The information and the transactions they use are provided by the Domain Systems. Products do not have their own data stores.
There is no middle layer. There is no magic component to handle orchestration of services or aggregation of data. All that happens in the product. More importantly, there is nowhere for vendors to put their integration suites. They simply aren’t necessary.
Here’s a simple picture, based on a generic insurance company, to give you a sense of what I’m describing:
In this example, the horizontal (green) boxes are the user facing (or automated process) products and the vertical boxes are the domain systems. To reiterate a previous point, domains do not interact with each other in any way, and there is no duplication of data across domains. All interaction is driven by the product systems which access the domains via simple APIs. The only processing a domain can do is on the data it exclusively contains. It is the strict adherence to these rules that is fundamental to making this approach work.
To bring this to life with a simple example, the picture below shows a basic product feature to purchase a piece of downloadable media and add it to the customer’s product holdings. The blue boxes below represent the domain systems accessed during that user flow, and the labels on the connecting lines represent the API calls used.
The model has now been inverted, with all integration happening in the user facing products, and all information storage delegated to the domain systems.
Inverse Conway’s Law
Conway’s law states that “organisations design systems that mirror their own communication structure”. The implication of this law is that if you look at the systems in an organisation you can fairly accurately predict the team structure for which they were created.
Understanding this law also leads to its equally useful converse. Essentially, if you want your systems to follow a particular pattern, organise your teams that way. If you follow this rule, the system pattern will be maintained even in the absence of governance. It simply becomes too difficult for teams to do it any other way, and the simplest approach is the one you want.
In the case of this model, the team structure is therefore as simple as the model. It consists of two types of team with two ways of working.
Firstly, you have Product teams. These teams are genuinely multidisciplinary and work closely with users. A product team owns a single product (or a small family of related products) and has the autonomy to deliver the features required based on the overall goals defined in their product strategy, and accountability for the outcomes achieved.
Secondly, you have Domain teams. These teams are predominantly technical in nature with a small element of subject matter expertise relating to the processing rules specific to the domain in question. They are responsible for the technology and implementation of the domain system and its API and have the autonomy to make any internal changes necessary to make their domain system operate as efficiently and effectively as possible.
Domain teams are not user facing; their primary responsibility is to respond to any changes or additions required to support the needs of the product teams (but only if those changes can be supported solely by their domain and fit within the predefined scope of their domain).
Division of labour
The rules for interaction between these teams are also simple. If a product team needs a new piece of data to be stored or provided, they identify to which domain it belongs and ask that domain to implement a change to support it. The domain team makes the change and exposes it via the API. The need is defined by the product team. How it is implemented is the responsibility of the domain team.
For this model to work, product teams need to understand that any aggregation and formatting is up to them. They can’t “delegate” product work to domain teams, or dictate how a change is implemented. In return, domain teams must be highly responsive. They need to treat the product teams as valuable customers, and make any changes swiftly. As an example, when this model was adopted in a large bank, the challenge set for the first domain team was to deliver any change requested on a Monday into the live environment by Friday with zero outages.
Needless to say, as the previous SLA had been 3 months, this expectation was considered unrealistic in the extreme. In practice, the team found that not only were they able to easily meet this expectation, they also had spare time to tune and improve their domain system
An added benefit of this autonomy to continuously improve is that avoiding a system becoming legacy becomes part of what the team does every day, rather than something that gets ignored until it becomes a programme in its own right.
Common challenges
Apart from the “it can’t be that simple” challenge, which can only be met with the equally unhelpful “oh yes it can” response, here are some typical questions with accompanying answers:
Q1: What if my product needs to present the user with a wide range of data aggregated from a variety of different sources? Surely providing a middle-tier aggregator would reduce the work required in the product?
A1: If you really want to provide your user with such a complex screen then your product will have to do all that aggregation. But an aggregator won’t save you any effort – it just offloads the effort to someone else. Also, I have yet to see a user interface that benefitted from cramming everything onto one page.
Q2: What if a particular aggregation of data across multiple domains happens to be used by more than one product? Wouldn’t an aggregator prevent duplication of code?
A2: If the instances of aggregation occur in different products, with different owners, processes and/or users then the commonality is unlikely to survive for long. Localised business change will result in deviation of need across the products and the reuse will tend towards a big “if” statement serving two different purposes. Better to separate the functionality now and allow the products to evolve independently of each other.
Q3: If I want to add a piece of information to a domain that isn’t currently supported, I have to rely on another team to add that data, and that impacts other users of the domain. Wouldn’t owning a copy of the data allow me to make the change independently of others?
A3: It would, but now you own a copy of the original data, and that data will need to be synchronised. If all products took this same approach there would be many copies in circulation and multiple synchronisations taking place. The opportunities to mess up the data grow with each new product. In a large organisation, such an approach can lead to so many copies of customer data that dealing with it becomes an industry in itself. This approach makes each domain so simple that adding data is quick and safe. There are no ripple effects as the domain has no integrations with other domains.
Q4: Where does the business logic live? In the product layer or the domain layer?
A4: Both, dependent on context. Business logic relating to the overall process belongs in the product layer. Business logic relating to local processing of domain data belongs in the domain layer (but only if it can be performed on the data within the domain itself without access to information or process state external to the domain).
Q5: This isn’t suitable for data analytics purposes. Many such activities require aggregation across multiple domains and across large quantities of data. Separation like this simply can’t support such a need.
A5: Absolutely correct. This is a model for managing operational information and processes. A separate model is needed for data analytics, and an extract from this model to an analytics platform should be part of the broader solution architecture. The same is true in reverse, of course. A model suitable for analytics purposes is not a good model for operational purposes where data access is transactional, real time, and granular. Furthermore, no organisation of any decent size should be performing high volume analytics on the same data sources as operational systems.
The devil is in the detail
Surely there’s more to it than this? This seems so simple as to be naive. The solution is certainly simple. The complexity, as always, lies in making it happen.
Domain systems, for example, can be new implementations, but it is more likely that they will start off as facades for existing systems. Implementation will need to be gradual and adopt a drain down (or strangler) pattern to convert and separate those existing systems into genuine domains. In many cases, off the shelf products can be used to implement domains, but if this approach is taken they should be headless and granular. Many traditional vendor systems rely on complex integration to encourage the purchasing of entire suites.
The same is true for existing products. Many will have been built inclusive of the data storage, or sold as part of an integrated vendor suite. Taking ownership of these products is a process that needs to be tackled over time, driven by business and user needs
This simple model, and the supporting team approach, have been implemented at enterprise level, so it does work for complicated situations, and it does deliver real advantages in terms of separation of concerns and speed of delivery.
However, you have to stick to the two most important rules: products don’t store data and domains don’t interact with each other. These might seem like minor points, but they’re absolutely key to making it work. If you’d tried something like this without sticking to these constraints and it didn’t work, that’s one possible reason.
Clearly this is just an overview of an approach, and there’s a lot more to moving from an existing model to any new one, but this is just a blog post, and it’s more than long enough as it is. It’s not big, and it’s not clever, and that’s the point. Life is complicated enough as it is, without adding to it with complicated architectures.
A final thought. Everything is a compromise – there is no ideal solution. I’ve worked with and implemented many different architectures in the past, and this is the one I’ve found to be the least problematic in the long run.
I hope you don’t mind me sharing it with you.