3 posts tagged with "Software Architecture"

"It is the developer’s assumptions which get shipped to production" - Alberto Brandolini

In parts one and two of this series we explored why software architecture is needed and some of the common pitfalls and failure modes that afflict "architects". This article will go into some of the principles and practices that are part of my toolbox. Very little of this is new and I've tried to link to the original sources so be sure to check the references. If I have misrepresented anything, that's solely on me.

Before we start, let's dive into what "Software Architecture" is, as there are different definitions out there:

• Software architecture is the study of the components of a system and their relationships;

• Software architecture is the collection of important/irreversible design decisions;

Both of these definitions are useful, however I would venture a slightly different one: Software architecture is a method of inquiry, for creating a shared understanding of how systems operate in their organizational context, helping drive better (i.e. fit for purpose, sustainable in the particular context) technical decisions.

A consequence of the definition I posited is that Architecture is not "a thing" per se, it is a process - it is never "done". Artifacts like diagrams or specific techniques to reason about a problem (e.g. DDD) have their place, but ultimately these are instruments to help make sense of the socio-technical environment we are operating in. It is context-sensitive and it should be practiced across the organization. You don't need to be one of the anointed ones to practice architecture.

Enable distributed decision making

Building software in larger organizations is a team sport, therefore centralized decision making, is fragile and very easily becomes counterproductive. Therefore design and decision-making should be decentralized, with teams on the driver's seat, and ultimately responsible for crafting the decisions that directly affect them. And since decision-making has an almost fractal quality to it as happens at all levels, thoughtful, disciplined, hopefully not-too-bad decision-making is a skill that software engineers can and should learn.

In order to support teams, and let people develop their design chops, establishing an ensemble practice for example via an "Architecture Advisory Forum" ¹ where team members are free to come with proposals on the evolution and change of their systems can be an excellent tool. The goal is for the group to engage as peers and help surface tradeoffs, different implementation options, or viability crushing constraints that may affect the proposal and shape the ultimate decision².

This can serve as the connective tissue between people operating in different scopes and abstraction levels, and provides a forum for practices such as:

• "Bytesize Architecture Sessions"³ to surface the assumptions and mental models that people have about the systems under their purview;

• "Let's break it" sessions where the goal is to come up with stressors⁴ and see how the current architecture, that is the individual systems and their relationships, are able to cope. These "random walks" through your system often uncover interesting inconsistencies and other urban legends.

• Encouraging a writing culture. If you work long enough in this industry you will find yourself in the situation where your team inherited a piece of software that has some implementation decisions that are... well questionable. And to make things worse, the reason as to why they came to be has been lost to the sands of time! What do you do then? Do you live with this because you're afraid of unintended consequences of your action - a classic case of Chesterton's fence⁵?

  Capturing why a decision was taken at a particular point in time can be a life-saver in those circumstances, through ADRs⁶ for example. In addition to this, clear writing helps clarify and structure ideas while providing a good basis for communication and knowledge sharing. This is often labeled (dismissively so) as a "soft-skill", but good writing skills are crucial for growth especially in the age of geographically distributed teams and work-from-home.⁷

Advisory

Teams should have the ultimate word about the decisions affecting them. Unless working in the team directly, a "software architect" should behave as an advisor to the team. The reason for this is that the team is usually in the best position to come up with a solution that works in their context, balancing relevant attributes (performance, security, scalability, etc) with the inevitable trade offs in terms of feasibility, cognitive load and operational overhead. As a rule of thumb: If you're not on the on-call rotation, then your default position should be that of an advisor. This will demonstrate trust, and will empower teams to own their decisions and learn from their mistakes.

Air support

If an organization has dedicated architecture roles (can also apply to Staff+ roles) a key part of the job will be to use the organizational power that comes with the title to provide "air support" for teams. This means making their work visible and readable to others when and where it is appropriate to do so. The role of the architect in this context can be crucial in order to allow different stakeholders to "connect the dots" and contextualize why certain technical investments are coherent with the overall direction where the organization wants to go and/or provide more optionality later down the line^{8 9}.

Enabling constraints

We often think of constraints in a negative light, after all constraining the range of permissible options seems like a bad thing, right? It actually depends, this is one of the most powerful tools that senior technical folks have at their disposal. Setting some constraints on the range of permissible technical options can actually be empowering. The classic example is Jeff Bezos mandate that all services in Amazon should communicate via "service interfaces", AKA APIs, that are designed to be externalizable. This certainly constrains how teams integrate their services, for example integrations via sharing a database or memory is off-limits, however teams now have more freedom to choose the right technology for the job, and the organization's resources and know-how can be focused on supporting this style of integration.

Such constraints need to be applied judiciously, but they can enable teams to operate with less friction in a particular socio-technical ecosystem¹⁰. So some constraints on system integration patterns, languages and frameworks usage, or how and where to find system documentation can actually be quite helpful to teams (just remember there may be exceptions!). And if we look at the stability of a system as a function of connectivity and variability - stability = connectivity x variability - then reducing variability, for instance by introducing some constraints on how services integrate with one another, can have a positive impact by creating a more stable system overall¹¹.`

"Every software‐intensive system has an architecture. In some cases that architecture is intentional, while in others it is accidental. Most of the time it is both" - Grady Booch

As we build more ambitious software intensive systems and software permeates more and more of our daily lives, often producing unexpected and surprising results, software should be thoughtfully designed.

Outside of very specific teams and social contexts, a fully emergent, one could say quasi-anarchical approach to software design will fail. The larger question, especially as waterfall-esque thinking seems to be gaining some momentum¹², is how to successfully integrate software architecture with an iterative, empirical software development process. This entails empowering teams to make better decisions, and taking a balanced approach with just the right level of decision making process and constraints imposed on teams¹³. This article explored my current toolbox to do so and hopefully this will be useful for some of you out there.

---

Footnotes​

1. The idea of an "Architecture Advisory Forum" is explained in further detail here. ↩

2. I find it important, especially when initially introducing such a practice to come up with some proposed structure in mind - for example focus on discussing ADRs for new features. This should undergo some iterations until it works in your context. If you're a more senior engineer introducing this practice, don't be afraid to start with some ground rules and depending on the maturity of your organization make sure that the ways of working are set in a charter-like document so that everyone knows the rules-of-engagement. ↩

3. Bytesize Architecture Sessions ↩

4. Barry O'Reilly makes a very compelling case for the idea that random simulation can yield resilient architectures. Definitely check this introduction to the topic of [Residuality theory](https://www.youtube.com/ watch?v=0wcUG2EV-7E) ↩

5. Chesterton's Fence ↩

6. ADR: Architecture Decision Record. Love Unrequited: The Story of Architecture, Agile, and How Architecture Decision Records Brought Them Together ↩

7. When trying to reason about how to communicate technical decisions I find that Good Strategy, Bad Strategy: The difference and why it matters provides very good advice on writing documents that have some substance in them, instead of just playing buzzword bingo. ↩

8. If this sounds like playing politics, well yes it is. Politics gets a bad reputation - perhaps deservedly so - however one should keep in mind that it is through politics that societies become more than just a sum of individuals and their choices. This is how humans organize collective, coherent and cohesive action at scale. ↩

9. Gregor Hohpe's book: The Software Architect Elevator: Redefining the Architect's Role in the Digital Enterprise provides a good metaphor for this sort of work. A big part of the role of an architect in a large organization is to ride an elevator from the engine room where technical decisions all the way up to the penthouse where key management decisions are taken. ↩

10. Ruth Malan's free ebook is great and offers additional insight into this. Her courses are also really good, I highly recommend them. ↩

11. This is one of the topics that Residuality Theory expands upon. You can find more about Kauffman Networks and how this can be valuable in the context of sofware engineering, which is essentially what I am getting to in Residues: Time, Change, and Uncertainty in Software Architecture. ↩

12. One could argue that the idea of treating software development as a mechanistic productive process with well defined stages has some appeal to organizations. This is wrong because that is not the game we are playing - we're really playing a design and intrinsically exploratory game - but nevertheless Waterfall keeps raising it's head. Dave Farley and Kent Beck had an excellent conversation that touched on the subject. ↩

13. Eduardo Silva's article on Minimalistic Architecture is quite good: Less is More with Minimalist Architecture, Ruth Malan and Dana Bredemeyer ↩

"Two weeks of coding can save you an hour of planning" — Unknown

In part one of this series¹ I tried making the argument that some degree of software architecture is required, and explored how a naive reading of agile software development practices coupled with organizational incentives create a toxic mix where software design is not valued, leading to a lack of clarity and coherence, viability crushing technical risks being ignored, and creeping and crippling technical debt that slows down software delivery.

In this second part I will briefly explore some of the anti-patterns that give Software Architecture a bad name.

So, why does Software Architecture, and by extension architects, get a bad reputation?

The stereotypical software architect is the gray bearded bloke - let's call him the Ivory Tower Architect - that thinks very hard on problems and produces documents with pretty diagrams and varying degrees of adherence to reality. This is a stereotype, and as all stereotypes it is unfair, however it also captures some hard truths.

The Ivory Tower Architect figure has fallen to the siren song of "the big plan": the idea that a non-trivial system can be fully captured by extensive documentation² and it is essentially the sum of its parts. And this perspective, based on a platonic ideal of the system, serves as the mental model that guides his action, ignoring that a lot of interesting behaviors (including failure modes) are emergent, no matter how well the boxes and arrows are drawn.

If we scratch the surface and question how a particular view of a system is constructed, we will often find that people construct their view based on their preferred modeling techniques and a lot of gut feeling - I'm no different, I regularly use Domain Driven Design³ and Wardley Mapping⁴ and I'm actively looking into incorporating STAP/STAMP⁵ and Residuality Theory⁶ into my toolbox - however it's important to understand that the choice of methodologies reflects the range of modeling techniques that people know and are familiar with. This can easily lead to an overreliance on particular techniques/tools/viewpoints to make sense of the world, forgetting the fact that "all maps are wrong, some are useful", and therefore no matter what, important details from a certain point of view will necessarily be missed.

These inevitable epistemological⁷ limitations become glaringly obvious if our Ivory Tower Software Architect is not actively engaged in actually building software. By not having skin in the game, the software architect is in a very comfortable position where he is shielded from the negative consequences of his decisions. Human nature being what it is, it will be exceedingly easy to blame failure on a team's inability to build according to what has been outlined, and at this point our Ivory Tower Architect friend has become a bona-fide Intellectual-Yet-Idiot⁸. Fundamentally operating in an open loop, misses critical implementation details that may derail the whole architecture (as well as potential improvement opportunities), on top of which the Ivory Tower Architect's technical skills will inevitably atrophy.

At a more fundamental level, a style of top-down architecture-by-decree limits the opportunities of teams to build up their architectural muscle, limiting their autonomy - which can be detrimental to their motivation and engagement - but hey it may grant a modicum of job security and brings some nice status perks, since after all it is a fancy title.

As I mentioned in my previous post, coordination of joint action, framing important technical initiatives in the wider business context and generally bridging the gap between technical and non-technical stakeholders is absolutely crucial. A lot of this will intersect with the practice of software architecture and, for the cases where there are dedicated architecture roles, it's very easy to fall into the trap of becoming an Ivory Tower Architect, where one is perpetually creating nice documents and dealing with organizational fuckery⁹.

In the next post of this series I will dive into some of the practices that I find valuable in my own software architecture practice - and how those help build a more resilient and healthy engineering organization.

---

Footnotes​

1. I did a minor update to last week's article to mention the fact that agile software delivery practices offer precious little guidance about the practice of sofware design. In addition to this I also expanded a bit more about how technical risks (depending on the nature of the project/product) need to be considered. ↩

2. The map is not the territory, and the effort to fully capture any non trivial system reminds me of this short paragraph by Jorge Luis Borges: On the exactitude of science ↩

3. Domain Driven Design ↩

4. Wardley Mapping ↩

5. STPA Handbook ↩

6. An Introduction to Residuality Theory - Barry O'Reilly ↩

7. I think software design - thus architecture - of large scale systems is also an epistemological problem: How do we "know" a system? What kinds of facts are possible to know in the first place? Is there a risk of projecting unfalsifiable beliefs into our designs - thus having a mental model that is wrong but very hard to disprove? ↩

8. "The IYI pathologizes others for doing things he doesn’t understand without ever realizing it is his understanding that may be limited." - Nassim Taleb, link ↩

9. Architects, anti-patterns and organizational fuckery ↩

“Simplicity and elegance are unpopular because they require hard work and discipline to achieve” — Edsger Dijkstra

Having a role that requires me to sometimes wear the "software architect's" hat at a relatively large organization has given me ample material to digest and reflect on the practice of software architecture, i.e. the design of software systems comprising multiple components owned by different teams. I am hoping to write a small series of posts that articulate (hopefully in an intelligible manner) why some architecture is relevant in our day and age, why it should be a seen as core skill for teams (especially senior+ engineers) and reflect on my own view of the practice of software architecture. Since architecture is part of the broader process of building any non-trivial software system, let's start there.

So, without further ado, is software architecture needed?

As an industry we've been in crisis since, well... forever. The term first appears in the aftermath of the 1968 NATO Software Engineering¹ Conference².

This state of affairs didn't prevent software from "eating the world", and being at the heart of the most valuable companies out there. The history of software development is marked by the trend of working at increasingly higher abstraction levels, from close to the metal development in Assembly and C, to modern distributed systems leveraging elastic computing capacity and programmed in high level languages.

In parallel with this technical evolution, the methodologies for managing software intensive projects and products also evolved. Initially imported from other industries, typically process-heavy, in an attempt to improve predictability (of both time and costs) and control the software development process, waterfall³, treats software development as an industrial production problem, and therefore it's core idea is that software development should follow well defined phases in sequence: requirements "engineering", analysis, program design, coding, testing and operations/maintenance. While this can be a valid approach in certain domains like aerospace, or medical devices, the reality is that in most contexts, software development is a learning problem rather than a production problem. Having to wait until all requirements are specified beforehand, or testing only after the code is "done" is a fool's errand and has been detrimental to our industry.

As a reaction to this in the late 90's and early 2000's there was the rise of "agile" software development methodologies⁴, which put an emphasis on delivering working software, in small iterations and with a pragmatic focus on what works in a given context versus strict adherence to a process, thus allowing teams to choose what works for them in their particular environment - the tectonic shift here is that these principles treat software development as an exercise in learning. These practices contribute to an the empirical approach to software development that is invaluable in my view.

However, this clashed with the reality that businesses need predictability, control (or the illusion thereof) and some way of attesting that things are being done in a "good" way - to be fair some software engineers often have a hard time productively engaging with non-technical stakeholders, and building a relationship of trust, so there is plenty of blame to go around.

Fast forward to 2024, and we continue our inexorable march to build ever more ambitious systems. I would argue that software design is more needed than ever.

We find ourselves in an interesting situation where a lot of organizations practice a sort of sclerotic fake agile⁵, that goes through the motions of daily stand-ups, retros, sprints and story-points but ultimately never reflect on those practices and question their value - and at the end teams are asked to produce a yearly plan.

While no one advocates that teams shouldn't stop and think, there are a couple of common traps that result from the toxic combination of a naive understanding of agile software development methodologies, and the incentives at play in organizations. In particular, I've seen teams thinking they can "afford" to go from iteration to iteration, delivering whatever is on the next sprint without taking the time step away, reflecting, factoring new risks and course correcting. After all they are following the plan, and if design is emergent and there is always the option to refactor (spot the cognitive dissonance?), why should teams invest time planning and designing and projecting how current choices will play out? On the other hand due to the pressures of fast moving organizations, all sorts of interesting had-hoc justifications as to why significant refactorings may be postponed enter the conversation. In general this contributes to an environment where some degree of planning and design is viewed with suspicion. The process becomes very dependent on individual's sensibilities of what "agile truly means" and I've seen teams reject some degree of planning as "big design upfront"⁶, better close those tickets, ship and refactor later. At the end of the day, this set of pressures and the fact that agile deals primarily with the software delivery process offering little to no guidance for how to bake design into the process, creates fertile ground for strongly held opinions, and egos coming into the mix. This is counterproductive, depending on your context you should consider that:

• Not all choices are created equal, some choices carry significant path dependence⁷ and may significantly constraint the ability of the team to operate or evolve the system (thus are hard to reverse). For example choice of data store is a classic example: DynamoDB is very performant for queries that against a primary key, range key or index - the moment you need to do queries against arbitrary columns or store more than 400KB per row, then you may be out of luck⁸.

• Some technical risks if not addressed early on, may prove to be viability crushing later on - teams coding themselves into a corner is not unhread of. There is a balancing act at play between proactively addressing risks and the "You ain't gonna need it" principle. At the end of the day it boils down to context specific bets on when and how to address these risks.

• Refactoring as a technique is invaluable, it may not be sufficient at scale and especially when dealing with cross-team dependencies. For tightly coupled systems, refactoring may be exceedingly hard because changes from one team may require significant coordination.

• Embedding technical qualities like simplicity (modularity, loose coupling, clear "contracts" and sensible interfaces), testability, security, performance and developer experience in the team's day-to-day work goes a long way in helping ensure that the cost of adding new feature down-the-line is not massively higher than building those first few features. This however, requires educating the the team and fostering a trust-based, productive relationship with other non-technical stakeholders in order to create the conditions for these qualities to be built/exercised.

• As organizations grow it's easy for the system as a whole to lose coherence. Teams operating autonomously (as they should) can easily fall into the trap of delivering services that are not really consistent with the goals of the organization or don't integrate well with what other teams have built - creating common ground and shared understanding requires investment⁹. Complex deliveries may take significant time, requiring the contribution of stakeholders and teams with different skill sets (engineering, legal, marketing, UX, etc). When operating at increasingly greater time spans of discretion¹⁰ that outgrow what is controlled by a single team and take longer than a few sprints, a modicum of common ground and coordination is paramount to ensure that the joint action is coherent.

• Stories play an important role in how humans make sense of, and experience the world. It is important to contextualize how the iterative work the team is doing fits the bigger picture, and how certain technical investments and practices may pay off. A good narrative that captures the problems that need addressing, what are the hypotheses being explored and how the team is contributing to the broader organization objectives can be a powerful instrument to elicit good feedback, or simply to get everyone on the same page. Technical excellence is a must-have, however just relying on a big dose of "programming motherfucker"¹¹ energy will not suffice outside of very specific cultures. Large scale software development is a socio-technical endeavor and a team sport.

So we're back to the beginning - software engineering is still in crisis, and by now I hope to have convinced you that we do need some design - AKA architecture - in our software systems. It's a real thing and not complete bullshit when done right.

In the next post I am going to cover some of the common anti-patterns that usually arise when we think about Software Architecture (note the capitalization) and offer some of my own observations on the subject. Charity Major's article¹² - spicy as it may be - provides a lot of good food for thought, and there is a lot to agree with - but there is also plenty more to be said about this.

---

Footnotes​

1. On the term "Software Engineering" see this very good essay by Hillel Wayne ↩

2. The 1968/69 NATO Software Engineering Reports ↩

3. https://blog.jbrains.ca/assets/articles/royce1970.pdf ↩

4. Agile manifesto ↩

5. https://martinfowler.com/articles/agile-aus-2018.html ↩

6. I find that this varies a lot with the experience of the team. Usually more mature folks have a more nuanced and more pragmatic approach. But considering how much our industry has grown, the odds are that teams will tend to have younger folks, which may lack some of the maturity of industry veterans - and the stereotype of the lone, red-bull drinking, code-slinger (coupled with ageism) still resonates (FYI: if you subscribe to that idea, then sorry to rain on your parade, but this is a team sport). ↩

7. Path dependence on Wikipedia ↩

8. When in doubt Postgresql is a great choice. ↩

9. Common ground and coordination in joint activity ↩

10. Time span of discretion and scope of complexity ↩

11. Programming motherfucker ↩

12. Architects, anti-patterns and organizational fuckery ↩