What does unicode, audio processing and admittedly bad early 2000s Internet memes have to do with one another?

In the previous post in the deep dive into unicode series we explored how combining characters like diacritics work. One interesting property of unicode is that it is possible to combine multiple combining characters together.

The example above shows the letter C with several combining characters, and as we can see they stack up quite nicely. This is the basis for an Internet meme of the early 2000s called Zalgo text¹. We can take this to the next level with a "winamp style" analyzer bar, but fully in (zalgo) text for an extra metal look and feel. 🤘 🤘

The reality is that this was really an excuse to play around with the Web Audio API² and some modern React (I hadn't touched front-end development in a while), and there were a few learnings in the process.

Implementation and technicalities

From an implementation perspective the first challenge was to understand what the Web Audio API offers in terms of digital signal processing and how to use it. The documentation is excellent and gist of it is that audio operations happen in an Audio Context that represents an audio processing graph built from several AudioNodes linked together in such a way that the output of one node serves as the input for the next. Because I wanted to extract the frequency domain from the audio signal in order to render it on screen, I used an AnalyzerNode³, which doesn't modify the audio but returns data about the frequency domain using a trusty old FFT⁴.

The following code example puts all of these concepts together:

const context = new AudioContext();
const theAnalyser = context.createAnalyser();
const source = context.createMediaElementSource(audioNode.current);
// build the audio processing graph connecting the input source
// to the analyzer node, and the output of the analyzer to the
// output of the Audio Context.
source.connect(theAnalyser);
theAnalyser.connect(context.destination);

Another interesting learning was about the advantages of requestAnimationFrame⁵ (RAF) versus a plain old setInterval for rendering. Since in this case I wanted performant and smooth updates RAF was an interesting choice as its refresh rate tries to match the display's refresh rate and calls are paused when running in background tabs or hidden - meaning better performance and battery life.

Finally, why not put everything together in a nice NPM package? Since I don't usually work in the JS ecosystem it was a nice opportunity to get some hands-on experience with this. The npmjs⁶ documentation is very good and the setup was straightforward, especially if you've published packages in Maven Central, Artifactory or equivalent. Top marks there. You can find the package here: https://www.npmjs.com/package/@felix.bruno/zalgo-player and installation is of course super easy:

$ npm install @felix.bruno/zalgo-player

This being the Javascript/Typescript ecosystem not everything was smooth sailing and I discovered that Create React App⁷ still doesn't support Typescript 5, and Github seems a bit dead which is a bit of a bummer. After spending some time looking Vite⁸ seemed like a decent choice to set up a basic react library with properly configured Typescript support.

In this case, since I wanted to publish only a React component and not a full-blown web application I had to make some changes to what Vite offers out-of-the-box⁹, but I am quite happy with the end result. The npm module is less than 15KB uncompressed, and has no dependencies (since this is a React component, it can only be used in that context, and thus we don't need to ship React with the package).

The code is available in Github: https://github.com/felix19350/zalgo-player

Note: In a next iteration I will work a bit to enable the component to be responsive, so if you view this in a mobile phone this may not render very well.

Programming as Theory Building¹ is an almost 40 year-old paper that remains relevant to this day. In it, the author (Peter Naur, Turing Award winner and the "N" in BNF²) dives into the fundamental question of what is programming, and builds up from that to answer the question about what expectations can one have, if any, on the modification and adaptation of software systems.

This paper offers a "Theory building view of programming" in which the core of a programmer's work is to create a "theory of how certain affairs of the world will be handled by, or supported by, a computer program". Such a theory informs the basic principles, abstractions and assumptions that are baked into a system's code, and thus the extensibility of a program is enabled or constrained by the degree teams are able to maintain a coherent theory for the system.

Naur adopts a definition of theory that is predicated on a distinction between intelligent and intellectual work - a theory being required for the later. Intelligent work on one hand is defined by the ability to do a certain task well (according to some criteria), correcting errors and learning from examples. Intellectual work on the other hand, is defined as the knowledge required to perform a task intelligently, but also be able to explain things, answer questions and critically appraise a particular solution/practice.

A programmer's work is therefore to formulate a theory of how certain processes and outcomes can be supported by code executing a computer. Equipped with a theory the programmer is able to:

1. Have an understanding of the relevant parts of the "world" that are in scope for the system and how the program and its structure relates to the desired outcomes;

2. Why is a program structured in a specific way and what are the underlying design principles/metaphors;

3. How a program can best be extended by leveraging the existing abstractions and capabilities;

It's models all the way

An interesting nuance is that if software development requires a theory of how a program will achieve certain outcomes, that in turn requires a theory of the underlying problem - otherwise how to make an informed judgment about which facts or processes in the world are relevant to a particular technical solution?

This leads to an interesting observation that the quality of the mental model of the solution is directly correlated with the quality of the mental model one has of the underlying problem. And over time, the ability to gracefully extend a software system is directly correlated with the ability of new joiners forming a coherent mental model of the problem and the system, as well as understanding the mental models of the programmers that came before.

The life and death of software systems

"The program text and its documentation has proved insufficient as a carrier of some of the most important design ideas."

Having seen my fair share of different systems throughout my career, and even "inheriting" some in a less-than-stellar shape, makes me appreciate how systems may end up having an expiry date even if in their current shape they are still producing value. How many times have you looked at a code base with important but ultimately not very enlightening documentation (e.g. initial project setup and little else) and packages/modules named after generic technical constructs such as controllers or services? Without access to the people that originally created the system it is exceedingly hard to understand the rationale behind some decisions. Practices like ADRs³ may help, however there are limitations to the level of understanding that can be gained from documentation alone.

"On the basis of the Theory Building View the decay of a program text as a result of modifications made by programmers without a proper grasp of the underlying theory becomes understandable."

This has implications on some of the assumptions that go into the practice of software development. First and foremost, socializing the theories and metaphors that underpin systems should be seen as a worthwhile investment in a system's long term sustainability. The corollary of this idea is that a software system "dies" once the theory behind it is lost to its current operators. The system may be kept running, but further modifications will most likely result in a patchwork of one-off fixes. This in turn provides ample grounds to challenge the perception that software systems are extremely adaptable by definition. If the team owning a system no longer has a coherent theory for that system (and maybe even the underlying problem), any changes will probably be crudely bolted on top of the existing implementation and that system is well on its way to become a toxic asset, no matter how bright people are or how fast they type. In such circumstances it is worth considering if it is better to start from scratch (after all buildings get demolished all the time, why not software systems?). Such a decision should not be taken lightly, but in many cases it can be done iteratively⁴ and thus it is possible to avoid the most common pitfalls that constitute the "second system syndrome"⁵.

Relevance in today's world

None of this is new, after all the paper is almost 40 years old. Practices like XP have been around for quite a while and "Agile" became mainstream (and probably got body-snatched in the process), yet I think it is extremely relevant in the present context. The idea that the major cost driver in software systems is writing the code is sadly still very much alive. This is untrue and betrays an at best superficial understanding of what goes into building software systems, but nonetheless it has quite a bit of traction among managers and all sorts of expert beginners⁶. As we are speed-running through a generative AI bubble⁷, the premise behind a lot of "Gen AI enabled" tools or “AI developers” is that writing code will be dramatically cheaper and thus programmers can either be way more effective or even replaced altogether.

As this paper argues, that is a fundamentally incorrect perspective. The key activity in developing software is coming up with a theory of the problem, figure out a theory for the solution, socialize it, and keep it up-to-date as new stressors are discovered and the team and surrounding organization changes. For the time being that remains squarely a human activity.

In the previous entry of this series I went through a lightning tour of what is Unicode and provided some details into the various encodings that are part of the standard (UTF-8/16/32). This serves as the baseline knowledge for further exploration of how Unicode strings work, and some of the interesting problems that arise in this space.

Codespace organization

Before we proceed with more practical aspects of Unicode string matching, I would like just to make a brief tangent for completeness sake and briefly touch upon how code points are organized.

As we previously discussed, the unicode standard allows for more than a million code points¹ (1114112 to be precise). The next question is of course, how is this code point space (codespace in Unicode parlance) organized internally? And why does that organization matter?

The codespace is not just a linear collection of code points. Characters are grouped by their attributes, such as script or writing system, and the highest level group is the "plane", which corresponds to 64k code points.

Plane 0, or the Basic Multilingual Plane (BMP), encodes most characters in current use (as well as some historical characters) in the first 64k code points. A nice side-effect of this is that it is possible to effectively support all current languages with a 16 bit fixed character size - although forgetting that UTF-16² is a variable length encoding can land you in trouble!

Beyond the BMP there are several other planes: The Supplementary Multilingual Plane (SMP or Plane 1) encodes seldomly used characters that didn't fit into the BMP, historical scripts and pictographic symbols. Beyond Plane 1, we find the Ideographic Plan (Plane 2) and the Tertiary Ideographic Plan (Plane 3) that encode Chinese, Japanese and Korean character (CJK) that are used for less frequent CJK characters that don't fit in the BMP. And finally we have the Supplementary Special Purpose Plane (SSP, plane 14) used as a spillover for format control characters that don't fit in the BMP and finally two Private Use planes (Planes 15 and 16) that are allocated for private use and expand on the private use characters located in the BMP.

Internally, each plane is arranged into several blocks, so for instance in BMP the area from 0x0000 to Ox00FF (the first 256 code points) match the ISO Latin-1 and ASCII encoding for retro compatibility.

Diacritics and other "strange" markings

Okay, back to the regular scheduled content: an aspect to consider is how Unicode deals with diacritics (and other marks and symbols). For the Latin alphabet this would probably be trivial (as we've seen Unicode is compatible with the ISO 8859-1 / Latin-1 encoding), however this is far from an extensible mechanism, so Unicode introduces the concept of combining characters, which are essentially marks that are placed relative to a base character. The convention is that the combining characters are applied after the base character.

An interesting fact is that more than one combining character may be applied to a single base character, this can open the door to some very creative uses like building an audio spectrum analyzer bar using the fact that you can "stack" combining characters. The code needs some tweaking and I will update it later:

There are exceptions to this principle especially due to retro compatibility reasons, so this means that there are different equivalent sequences.

For instance the character: ç can be represented by the code point U+00E7 or the U+0063 (the c) followed by U+0327 (the cedilla).

Now this poses an interesting question, are vanilla string classes in popular programming languages aware of this when making string comparisons?

Let's start with a basic example to see how this actually works (the content below is rendered in a React component):

If you want to test it yourself, you can past the following code in your browser's debug console:

const singleCodePoint = String.fromCodePoint(0xE7);
const combiningCharacters = String.fromCodePoint(0x63, 0x327);

console.log("Single code input: " + singleCodePoint);
console.log("Combining characters: " + combiningCharacters);
console.log(singleCodePoint === combiningCharacters);

One could say this is a Javascript quirk, however that is not the case. If you have Python installed in your system (please use Python 3) you can test the following code:

singleCodePoint = chr(0xE7)
combiningCharacters = chr(0x63) + chr(0x327)
print("Single code input: " + singleCodePoint)
print("Combining characters: " + combiningCharacters)
print(singleCodePoint == combiningCharacters)

Clearly vanilla string comparison fails for strings that are visually and semantically equivalent³, which is not good and may break applications in weird and wonderful ways (e.g. think about the effects of this in data structures like sets or maps/dictionaries).

And if you think that this is an exclusive of those pesky interpreted languages, well, even the trusty old compareToIgnoreCase in Java fails this test:

void main() {
    String singleCodePoint = new String(new int[]{0xE7}, 0, 1);
    String combiningCharacters = new String(new int[]{0x63, 0x327}, 0, 1);

    System.out.println("Single code input: " + singleCodePoint);
    System.out.println("Combining characters: " + combiningCharacters);
    System.out.println(singleCodePoint.compareToIgnoreCase(combiningCharacters) == 0);
}

To run this you can simply paste the code above to a .java file, in this case Main.java and compile it:

$ javac --source 21 --enable-preview Main.java
$ java --enable-preview Main 

Unsurprisingly at this point the last line outputs false meaning that both strings are not equal. So what can be done about this?

Normalization

Clearly comparing Unicode strings is not as straightforward as one may think, especially when dealing with strings that can be considered to be equivalent (as in the examples above). Fortunately the Unicode standard defines algorithms to create normalized forms that eliminate unwanted distinctions.

In order to understand how Unicode normalization works it's important to understand the concepts of canonical equivalence and compatibility equivalence.

Canonical equivalence: Two strings are said to be canonical equivalents if their full canonical decompositions are identical. For example:

• Combining sequences: 0x00E7 is equivalent to 0x0063, 0x0327
• Ordering of combining marks: q+◌̇+◌̣ is equivalent to q+◌̣+◌̇
• Singleton equivalence: U+212B (Angstrom Sign) is equivalent to U+00C5 (Latin Capital Letter a with Ring Above). In the normalization process singletons will be replaced.
• Hangul & conjoining jamo

Note that language specific rules for matching and ordering may treat letters differently from the canonical equivalence (more on that in a later post).

Compatibility equivalence: Two character sequences are said to be compatibility equivalents if their full compatibility decompositions are identical. This is a weaker type of equivalence, so greater care should be taken to ensure the equivalence is appropriate. A compatibility decomposition is an algorithm that maps an input character both the canonical mapping and the compatibility mappings found in the Unicode Character Database. For example⁴:

• Font variants
• Linebreaking differences
• Positional forms
• Circled variants
• Width variants
• Rotated variants
• Superscripts/Subscripts
• Squared characters
• Fractions

Unicode offers four normalization forms (NF)⁵, that either try to break apart composite characters (decomposition) or convert to composite characters (composition):

The following example (rendered in a React component) shows the normalization forms in action⁶:

What does this mean in practice?

The normalization forms perform modifications to the text and may result in the loss of important semantic information, so they are best used like the typical uppercase and lowercase modifications, i.e. definitely very useful, but not always appropriate dependending on the context.

In the next post in this series we're going to apply normalization, plus a few other tricks for more realistic scenarios such as matching of names, so stay tuned!

"The ecology of the distributed high-tech workplace, home, or school is profoundly impacted by the relatively unstudied infrastructure that permeates all its functions" - Susan Leigh Star

Representing, processing, sending and receiving text (also known as strings in computer-speak) is one of the most common things computers do. Text representation and manipulation in a broad sense, has a quasi infrastructural¹ quality to it, we all use it in one form or another and it generally works really well - so well in fact that we often don't pay attention to how it all works behind the scenes.

As software developers it is key to have a good grasp of how computers represent text, especially as in certain application domains even supposedly "simple" operations like assessing if two strings are the same has a surprising depth to it. After all should the string "Bruno Felix" be considered to be the same as the string "Bruno Félix" (note the acute "e" on the last name)? The answer of course is: "it depends".

But let's start from the beginning, in this series I am going to explore the Unicode standard, going a bit beyond the bare minimum every developer needs to know, and into some aspects that are not often not widely talked about, in particular about how characters with diacritics are represented, and how this impacts common operations like string comparisons and ordering.

The briefest history of text representations ever

There is already a lot of good material out there about how computers represent text² which I highly recommend. For the sake of this article I'm going to speed run through the history of how computers represent text, in order to be able to dive in more detail at the internals of Unicode.

So in a very abbreviated manner: computers work with numbers, so the obvious thing to do to represent text is to assign a number to each letter. This is basically true to this day. Since the initial developments in digital computing were done in the USA and the UK, English became (and still is) the lingua franca of computing. It was straightforward to come up with a mapping between every letter in the English alphabet, digits, common punctuation marks plus some control characters, and a number. This mapping is called an encoding, and probably the oldest encoding you will find out there in the wild is ASCII, which is able to do exactly this for the English alphabet - and do so using only in 7 bits.

We can actually see this in action if we save the string Hello in ASCII in a file and dump its hexadecimal content.

$ hexdump -C hello-ascii.txt
00000000  48 65 6c 6c 6f                                    |Hello|
00000005

Of course this was far from perfect, especially if you're not an English speaker. What about all the other languages out there in the world? And to do that in a way that maintains retro compatibility with ASCII?

Since computers work in multiples of 2 thus 8 bit bytes, this means ASCII leaves one bit available, the ASCII 7 bit encoding is quite easy to extend by adding an additional bit (nice as it maintains retro compatibility), doubling the number of characters available and still making everything fit in a single byte. Amazing! This is actually what vendors did, and this eventually got standardized in encodings like ISO 8859-1. Of course 255 characters is not enough to fit every character for every writing system out there, so the way this was initially approached was to have several "code pages", that essentially map the 1 byte number to different alphabets (e.g. the Latin alphabet has one code page, the Greek alphabet had another).A consequence of this is that in order to make sense of a piece of text one needs additional meta-information about which code page to use: after all character 233 will change depending on the code page (and of course it still doesn't work for alphabets with thousands of characters)

If we have the string Félix written in ISO-8859-1 (Latin) and read the same string in ISO-8859-7 (Greek) we get Fιlix, despite the fact that the bytes are exactly the same!

$ hexdump -C felix-code-page-confusion.txt
00000000  46 e9 6c 69 78                                    |F.lix|
00000005

A brief intro to Unicode and character encodings

The example above is not only an interoperability nightmare, but it still doesn't cover all languages in use so it's not a sustainable solution to the issue of text representation. Unicode³ tries to address these issues by starting from a simple idea: each character is assigned its own number, that is é and ι will be assigned different numbers (code points in Unicode terminology). Code points are typically represented by U+ followed by the hexadecimal encoded value of the character, so for instance é is represented as U+00E9 and ι is represented as U+03B9. The code points are also selected in such a way that retro compatibility is maintained with ISO-8859-1, and ASCII.

Currently Unicode has defined code points for more than 149186 characters (AKA code points), and this covers not only languages that are in active use today, but also historical (e.g. Cuneiform) or fictional languages (e.g. Tengwar⁴). Although it is important to note that most characters in use are encoded by the first 65,536 code points. In total Unicode allows the definition of up to 1114112 code points characters, so it is quite future-proof.

An important thing to note is that code points don't specify anything at all about how they are converted to actual bytes in memory or on a disk - they are abstract ideas. This is one of the key things to keep in mind when thinking about Unicode, there is by design a clear difference between identifying a character, representing it in a way that a computer can process it and rendering a glyph on a screen.

The good thing about standards is that you get to choose - Unknown

So if code points are notional, abstract ideas, how can computers make use of them? This is where encodings come into the picture. Unicode Standard offers several different options to represent characters: a 32-bit form (UTF-32), a 16-bit form (UTF-16), and an 8-bit form (UTF-8). The key idea here is that a code point (which is an integer at the end of the day) is represented by one or mode code units. UTF-32/16/8 offer different base sizes for these code units.

When working with UTF-32 and UTF-16, the endianness,that is the order of the most significant or least significant byte needs to be considered.

UTF-32 provides fixed length code units, making it simple to process. This comes at the cost of increased memory or disk storage space for characters.

$ hexdump -C hello-utf32-little-endian.txt
00000000  48 00 00 00 65 00 00 00 6c 00 00 00 6c 00 00 00  6f 00 00 00                    |H...e...l...l...o...|
0000000a

$ hexdump -C hello-utf32-big-endian.txt
00000000  00 00 00 48 00 00 00 65 00 00 00 6c 00 00 00 6c  00 00 00 6f                    |...H...e...l...l...o|
0000000a

UTF-16 provides a balance between processing efficiency and storage requirements. This is because all the commonly used characters fit into a single 16-bit code unit, but it is important to keep in mind that this is still a variable length encoding (a code-point may span one or two code units). Fun fact: the JVM and the CLR use UTF-16 strings internally.

$ hexdump -C hello-utf16-little-endian.txt
00000000  48 00 65 00 6c 00 6c 00  6f 00                    |H.e.l.l.o.|
0000000a

$ hexdump -C hello-utf16-big-endian.txt
00000000  00 48 00 65 00 6c 00 6c  00 6f                    |.H.e.l.l.o|
0000000a

Finally, UTF-8 is a byte oriented variable length encoding (so be careful with the assumption that each character is a byte, that is not what the 8 in UTF-8 means!). Since it is byte oriented, and the code points have been carefully chosen, this encoding is retro compatible with ASCII (note the example below). On the other hand a code point may be anywhere from 1 to 4 8 bit code units long, so processing will be more complex.

$ hexdump -C hello-utf8.txt
00000000  48 65 6c 6c 6f                                    |Hello|
00000005

This essentially allows programmers to trade off simplicity in processing with resources (memory or storage space) and retro-compatibility requirements according to the specific needs of their application. The following picture (directly from Chapter 2⁵ of the Unicode Standard may further clarify things):

Hopefully this brief intro provides a good foundation as to why Unicode has become the de-facto way to represent text, and the key difference between code points and encodings. This serves as a stepping stone to further explore the Unicode Standard, and in the next post in this series I will dive a bit deeper into how code points are structured, the types of characters that exist and how they are combined and can be normalized.

"It is the developer’s assumptions which get shipped to production" - Alberto Brandolini

In parts one and two of this series we explored why software architecture is needed and some of the common pitfalls and failure modes that afflict "architects". This article will go into some of the principles and practices that are part of my toolbox. Very little of this is new and I've tried to link to the original sources so be sure to check the references. If I have misrepresented anything, that's solely on me.

Before we start, let's dive into what "Software Architecture" is, as there are different definitions out there:

• Software architecture is the study of the components of a system and their relationships;

• Software architecture is the collection of important/irreversible design decisions;

Both of these definitions are useful, however I would venture a slightly different one: Software architecture is a method of inquiry, for creating a shared understanding of how systems operate in their organizational context, helping drive better (i.e. fit for purpose, sustainable in the particular context) technical decisions.

A consequence of the definition I posited is that Architecture is not "a thing" per se, it is a process - it is never "done". Artifacts like diagrams or specific techniques to reason about a problem (e.g. DDD) have their place, but ultimately these are instruments to help make sense of the socio-technical environment we are operating in. It is context-sensitive and it should be practiced across the organization. You don't need to be one of the anointed ones to practice architecture.

Enable distributed decision making

Building software in larger organizations is a team sport, therefore centralized decision making, is fragile and very easily becomes counterproductive. Therefore design and decision-making should be decentralized, with teams on the driver's seat, and ultimately responsible for crafting the decisions that directly affect them. And since decision-making has an almost fractal quality to it as happens at all levels, thoughtful, disciplined, hopefully not-too-bad decision-making is a skill that software engineers can and should learn.

In order to support teams, and let people develop their design chops, establishing an ensemble practice for example via an "Architecture Advisory Forum" ¹ where team members are free to come with proposals on the evolution and change of their systems can be an excellent tool. The goal is for the group to engage as peers and help surface tradeoffs, different implementation options, or viability crushing constraints that may affect the proposal and shape the ultimate decision².

This can serve as the connective tissue between people operating in different scopes and abstraction levels, and provides a forum for practices such as:

• "Bytesize Architecture Sessions"³ to surface the assumptions and mental models that people have about the systems under their purview;

• "Let's break it" sessions where the goal is to come up with stressors⁴ and see how the current architecture, that is the individual systems and their relationships, are able to cope. These "random walks" through your system often uncover interesting inconsistencies and other urban legends.

• Encouraging a writing culture. If you work long enough in this industry you will find yourself in the situation where your team inherited a piece of software that has some implementation decisions that are... well questionable. And to make things worse, the reason as to why they came to be has been lost to the sands of time! What do you do then? Do you live with this because you're afraid of unintended consequences of your action - a classic case of Chesterton's fence⁵?

  Capturing why a decision was taken at a particular point in time can be a life-saver in those circumstances, through ADRs⁶ for example. In addition to this, clear writing helps clarify and structure ideas while providing a good basis for communication and knowledge sharing. This is often labeled (dismissively so) as a "soft-skill", but good writing skills are crucial for growth especially in the age of geographically distributed teams and work-from-home.⁷

Advisory

Teams should have the ultimate word about the decisions affecting them. Unless working in the team directly, a "software architect" should behave as an advisor to the team. The reason for this is that the team is usually in the best position to come up with a solution that works in their context, balancing relevant attributes (performance, security, scalability, etc) with the inevitable trade offs in terms of feasibility, cognitive load and operational overhead. As a rule of thumb: If you're not on the on-call rotation, then your default position should be that of an advisor. This will demonstrate trust, and will empower teams to own their decisions and learn from their mistakes.

Air support

If an organization has dedicated architecture roles (can also apply to Staff+ roles) a key part of the job will be to use the organizational power that comes with the title to provide "air support" for teams. This means making their work visible and readable to others when and where it is appropriate to do so. The role of the architect in this context can be crucial in order to allow different stakeholders to "connect the dots" and contextualize why certain technical investments are coherent with the overall direction where the organization wants to go and/or provide more optionality later down the line^{8 9}.

Enabling constraints

We often think of constraints in a negative light, after all constraining the range of permissible options seems like a bad thing, right? It actually depends, this is one of the most powerful tools that senior technical folks have at their disposal. Setting some constraints on the range of permissible technical options can actually be empowering. The classic example is Jeff Bezos mandate that all services in Amazon should communicate via "service interfaces", AKA APIs, that are designed to be externalizable. This certainly constrains how teams integrate their services, for example integrations via sharing a database or memory is off-limits, however teams now have more freedom to choose the right technology for the job, and the organization's resources and know-how can be focused on supporting this style of integration.

Such constraints need to be applied judiciously, but they can enable teams to operate with less friction in a particular socio-technical ecosystem¹⁰. So some constraints on system integration patterns, languages and frameworks usage, or how and where to find system documentation can actually be quite helpful to teams (just remember there may be exceptions!). And if we look at the stability of a system as a function of connectivity and variability - stability = connectivity x variability - then reducing variability, for instance by introducing some constraints on how services integrate with one another, can have a positive impact by creating a more stable system overall¹¹.`

"Every software‐intensive system has an architecture. In some cases that architecture is intentional, while in others it is accidental. Most of the time it is both" - Grady Booch

As we build more ambitious software intensive systems and software permeates more and more of our daily lives, often producing unexpected and surprising results, software should be thoughtfully designed.

Outside of very specific teams and social contexts, a fully emergent, one could say quasi-anarchical approach to software design will fail. The larger question, especially as waterfall-esque thinking seems to be gaining some momentum¹², is how to successfully integrate software architecture with an iterative, empirical software development process. This entails empowering teams to make better decisions, and taking a balanced approach with just the right level of decision making process and constraints imposed on teams¹³. This article explored my current toolbox to do so and hopefully this will be useful for some of you out there.

"Two weeks of coding can save you an hour of planning" — Unknown

In part one of this series¹ I tried making the argument that some degree of software architecture is required, and explored how a naive reading of agile software development practices coupled with organizational incentives create a toxic mix where software design is not valued, leading to a lack of clarity and coherence, viability crushing technical risks being ignored, and creeping and crippling technical debt that slows down software delivery.

In this second part I will briefly explore some of the anti-patterns that give Software Architecture a bad name.

So, why does Software Architecture, and by extension architects, get a bad reputation?

The stereotypical software architect is the gray bearded bloke - let's call him the Ivory Tower Architect - that thinks very hard on problems and produces documents with pretty diagrams and varying degrees of adherence to reality. This is a stereotype, and as all stereotypes it is unfair, however it also captures some hard truths.

The Ivory Tower Architect figure has fallen to the siren song of "the big plan": the idea that a non-trivial system can be fully captured by extensive documentation² and it is essentially the sum of its parts. And this perspective, based on a platonic ideal of the system, serves as the mental model that guides his action, ignoring that a lot of interesting behaviors (including failure modes) are emergent, no matter how well the boxes and arrows are drawn.

If we scratch the surface and question how a particular view of a system is constructed, we will often find that people construct their view based on their preferred modeling techniques and a lot of gut feeling - I'm no different, I regularly use Domain Driven Design³ and Wardley Mapping⁴ and I'm actively looking into incorporating STAP/STAMP⁵ and Residuality Theory⁶ into my toolbox - however it's important to understand that the choice of methodologies reflects the range of modeling techniques that people know and are familiar with. This can easily lead to an overreliance on particular techniques/tools/viewpoints to make sense of the world, forgetting the fact that "all maps are wrong, some are useful", and therefore no matter what, important details from a certain point of view will necessarily be missed.

These inevitable epistemological⁷ limitations become glaringly obvious if our Ivory Tower Software Architect is not actively engaged in actually building software. By not having skin in the game, the software architect is in a very comfortable position where he is shielded from the negative consequences of his decisions. Human nature being what it is, it will be exceedingly easy to blame failure on a team's inability to build according to what has been outlined, and at this point our Ivory Tower Architect friend has become a bona-fide Intellectual-Yet-Idiot⁸. Fundamentally operating in an open loop, misses critical implementation details that may derail the whole architecture (as well as potential improvement opportunities), on top of which the Ivory Tower Architect's technical skills will inevitably atrophy.

At a more fundamental level, a style of top-down architecture-by-decree limits the opportunities of teams to build up their architectural muscle, limiting their autonomy - which can be detrimental to their motivation and engagement - but hey it may grant a modicum of job security and brings some nice status perks, since after all it is a fancy title.

As I mentioned in my previous post, coordination of joint action, framing important technical initiatives in the wider business context and generally bridging the gap between technical and non-technical stakeholders is absolutely crucial. A lot of this will intersect with the practice of software architecture and, for the cases where there are dedicated architecture roles, it's very easy to fall into the trap of becoming an Ivory Tower Architect, where one is perpetually creating nice documents and dealing with organizational fuckery⁹.

In the next post of this series I will dive into some of the practices that I find valuable in my own software architecture practice - and how those help build a more resilient and healthy engineering organization.

“Simplicity and elegance are unpopular because they require hard work and discipline to achieve” — Edsger Dijkstra

Having a role that requires me to sometimes wear the "software architect's" hat at a relatively large organization has given me ample material to digest and reflect on the practice of software architecture, i.e. the design of software systems comprising multiple components owned by different teams. I am hoping to write a small series of posts that articulate (hopefully in an intelligible manner) why some architecture is relevant in our day and age, why it should be a seen as core skill for teams (especially senior+ engineers) and reflect on my own view of the practice of software architecture. Since architecture is part of the broader process of building any non-trivial software system, let's start there.

So, without further ado, is software architecture needed?

As an industry we've been in crisis since, well... forever. The term first appears in the aftermath of the 1968 NATO Software Engineering¹ Conference².

This state of affairs didn't prevent software from "eating the world", and being at the heart of the most valuable companies out there. The history of software development is marked by the trend of working at increasingly higher abstraction levels, from close to the metal development in Assembly and C, to modern distributed systems leveraging elastic computing capacity and programmed in high level languages.

In parallel with this technical evolution, the methodologies for managing software intensive projects and products also evolved. Initially imported from other industries, typically process-heavy, in an attempt to improve predictability (of both time and costs) and control the software development process, waterfall³, treats software development as an industrial production problem, and therefore it's core idea is that software development should follow well defined phases in sequence: requirements "engineering", analysis, program design, coding, testing and operations/maintenance. While this can be a valid approach in certain domains like aerospace, or medical devices, the reality is that in most contexts, software development is a learning problem rather than a production problem. Having to wait until all requirements are specified beforehand, or testing only after the code is "done" is a fool's errand and has been detrimental to our industry.

As a reaction to this in the late 90's and early 2000's there was the rise of "agile" software development methodologies⁴, which put an emphasis on delivering working software, in small iterations and with a pragmatic focus on what works in a given context versus strict adherence to a process, thus allowing teams to choose what works for them in their particular environment - the tectonic shift here is that these principles treat software development as an exercise in learning. These practices contribute to an the empirical approach to software development that is invaluable in my view.

However, this clashed with the reality that businesses need predictability, control (or the illusion thereof) and some way of attesting that things are being done in a "good" way - to be fair some software engineers often have a hard time productively engaging with non-technical stakeholders, and building a relationship of trust, so there is plenty of blame to go around.

Fast forward to 2024, and we continue our inexorable march to build ever more ambitious systems. I would argue that software design is more needed than ever.

We find ourselves in an interesting situation where a lot of organizations practice a sort of sclerotic fake agile⁵, that goes through the motions of daily stand-ups, retros, sprints and story-points but ultimately never reflect on those practices and question their value - and at the end teams are asked to produce a yearly plan.

While no one advocates that teams shouldn't stop and think, there are a couple of common traps that result from the toxic combination of a naive understanding of agile software development methodologies, and the incentives at play in organizations. In particular, I've seen teams thinking they can "afford" to go from iteration to iteration, delivering whatever is on the next sprint without taking the time step away, reflecting, factoring new risks and course correcting. After all they are following the plan, and if design is emergent and there is always the option to refactor (spot the cognitive dissonance?), why should teams invest time planning and designing and projecting how current choices will play out? On the other hand due to the pressures of fast moving organizations, all sorts of interesting had-hoc justifications as to why significant refactorings may be postponed enter the conversation. In general this contributes to an environment where some degree of planning and design is viewed with suspicion. The process becomes very dependent on individual's sensibilities of what "agile truly means" and I've seen teams reject some degree of planning as "big design upfront"⁶, better close those tickets, ship and refactor later. At the end of the day, this set of pressures and the fact that agile deals primarily with the software delivery process offering little to no guidance for how to bake design into the process, creates fertile ground for strongly held opinions, and egos coming into the mix. This is counterproductive, depending on your context you should consider that:

• Not all choices are created equal, some choices carry significant path dependence⁷ and may significantly constraint the ability of the team to operate or evolve the system (thus are hard to reverse). For example choice of data store is a classic example: DynamoDB is very performant for queries that against a primary key, range key or index - the moment you need to do queries against arbitrary columns or store more than 400KB per row, then you may be out of luck⁸.

• Some technical risks if not addressed early on, may prove to be viability crushing later on - teams coding themselves into a corner is not unhread of. There is a balancing act at play between proactively addressing risks and the "You ain't gonna need it" principle. At the end of the day it boils down to context specific bets on when and how to address these risks.

• Refactoring as a technique is invaluable, it may not be sufficient at scale and especially when dealing with cross-team dependencies. For tightly coupled systems, refactoring may be exceedingly hard because changes from one team may require significant coordination.

• Embedding technical qualities like simplicity (modularity, loose coupling, clear "contracts" and sensible interfaces), testability, security, performance and developer experience in the team's day-to-day work goes a long way in helping ensure that the cost of adding new feature down-the-line is not massively higher than building those first few features. This however, requires educating the the team and fostering a trust-based, productive relationship with other non-technical stakeholders in order to create the conditions for these qualities to be built/exercised.

• As organizations grow it's easy for the system as a whole to lose coherence. Teams operating autonomously (as they should) can easily fall into the trap of delivering services that are not really consistent with the goals of the organization or don't integrate well with what other teams have built - creating common ground and shared understanding requires investment⁹. Complex deliveries may take significant time, requiring the contribution of stakeholders and teams with different skill sets (engineering, legal, marketing, UX, etc). When operating at increasingly greater time spans of discretion¹⁰ that outgrow what is controlled by a single team and take longer than a few sprints, a modicum of common ground and coordination is paramount to ensure that the joint action is coherent.

• Stories play an important role in how humans make sense of, and experience the world. It is important to contextualize how the iterative work the team is doing fits the bigger picture, and how certain technical investments and practices may pay off. A good narrative that captures the problems that need addressing, what are the hypotheses being explored and how the team is contributing to the broader organization objectives can be a powerful instrument to elicit good feedback, or simply to get everyone on the same page. Technical excellence is a must-have, however just relying on a big dose of "programming motherfucker"¹¹ energy will not suffice outside of very specific cultures. Large scale software development is a socio-technical endeavor and a team sport.

So we're back to the beginning - software engineering is still in crisis, and by now I hope to have convinced you that we do need some design - AKA architecture - in our software systems. It's a real thing and not complete bullshit when done right.

In the next post I am going to cover some of the common anti-patterns that usually arise when we think about Software Architecture (note the capitalization) and offer some of my own observations on the subject. Charity Major's article¹² - spicy as it may be - provides a lot of good food for thought, and there is a lot to agree with - but there is also plenty more to be said about this.

An important part of the job of a software engineer is to ensure that our systems operate as expected. Establishing a feedback loop between the features a team develops and how it behaves in production is one of the traits of high performing teams¹, and the idea that "you build it, you operate it" goes a long way to alleviate a lot of the toxic interaction patterns between traditional development and operations teams.

At the time of this writing, the excesses of a zero interest rate policy are being purged (often excessively so) from organizations. Throughout this industry-wide adjustment process, it is common to find teams of developers that are missing the critical know-how, mental models and hard earned experiences of senior devops practitioners. This situation is made worse by the fact that by-and-large practices around operations, monitoring and observability are taught on the job. So teams are left to sink or swim - they lack a basic theory on how to approach this problem.

In this post I will briefly introduce some of my ideas on this topic, hopefully serving as a primer for you to think and dive deeper into this subject.

Operations are hard! Software systems, no matter how well structured, operate in a messy organizational context. Decisions that made sense two months ago can be viability crushing because a competitor launched a new product, or new regulation was passed, or a war started (see the concept of hyperliminal systems²). Systems also operate at capacity³ - new practices/technologies/methodologies open new perceived opportunities to create value and systems and teams are expected to incorporate them (we're seeing this now with Generative AI). On top of this, the hard work of keeping systems up and running is often invisible. As with much of the infrastructure that makes modern life possible, we only become aware of its existence and its complexities, sometimes acutely so, when something breaks.

So, why do we need operations in the first place? It's hard, often unappreciated work, so what's the deal? I will argue that if you care about the "quality" of what you deliver - as any software engineer should - you definitely need to care about the operational aspects of the systems under your purview. This leads us to an interesting question: what is "quality"? If you ask your teams, who are usually under pressure to deliver the next shiny feature, this question you may get some blank stares, puzzled looks and mumbled semi-articulated responses. The pragmatic definition I would like to offer in this context is that "quality" is the result of a system operating according to its specification (this includes system outputs, behavior and resource usage). For example, if you care about serving web pages, quality may be defined by a combination of response time (99% of requests are served under X milliseconds) and errors (99.99% of requests return an HTTP status 200 to the client).

As with a lot of other software engineering related activities, it is important to keep in mind that you will probably benefit from an iterative approach to understanding what does "quality" entail in your specific context.This is an exercise in progressively getting a better understanding of your system (until you are fundamentally surpirsed and need to ditch some of your assumptions). Formulate an hypothesis on what makes sense to measure, implement it, and reflect on whether or not your expectations were met. This disciplined approach of creating feedback loops to make sense of your system and course-correct, is really at heart of a healthy software engineering practice.

Equipped with this definition and methodology, we can start probing in more concrete directions in order to paint a more complete picture of the quality aspects of a system. As per the definition of quality introduced above there are two facets to monitor:

• System outputs and behavior: is the system achieving its goal, i.e. satisfying its clients, with an acceptable performance?

  • What indicators reflect the customer/stakeholder's experience of my system? Often you may have to resort to proxy indicators, and that's okay. Equipped with a careful selection of indicators, you can craft meaningful Service Level Objectives. For a deep dive into this topic I recommend this book⁴;
  • An often overlooked aspect is the quality of the data your system produces. Returning an HTTP Status 200 100% of the time but producing bad/corrupted/inconsistent data 10% of the time is not great. Enforcing strict schemas is a great starting point - these act as contracts between the system and its clients - and is all you need in many cases. However that may be insufficient, often responses are complex, with optional fields, and subtle dependencies between values that are hard to capture in with tools like JSON schema or interface definition languages like AVRO/Protobuf. There are interesting practices that can be adopted from the data engineering and data science worlds, like data quality scores⁵.

• Resource usage: is the goal being achieved within the economic constraints of the organization?

  • This is typically where teams start their monitoring and observability journey, as these metrics are usually readily available. There are several good methodologies that inform how you can monitor resources (e.g. CPU, RAM) and how to use them when faced with an operational issue (e.g. the USE or the RED methods⁶);

This list is obviously incomplete, and it should be viewed more as a starting point to a conversation, rather than an authoritative recipe on what to measure - remember: Context is king, so your mileage will vary.

The most important takeaway is the lens with which teams can chip away at this problem. It boils down to a pragmatic usage of the scientific method⁷: formulate an hypothesis regarding what indicators/facets of your system you need to monitor to achieve a certain objective, implement it, observe the results and reflect. Rinse and repeat.

What's the most convenient way to set up a technical blog in 2024? The fact is that in this day and age there is an almost overwhelming plethora of choices from Wordpress, to Github pages it seems that there is something for everyone.

Thinking a bit more about my goals and what I value these were the initial requirements I set for this project:

• Cost efficient - if possible I don't want to pay for hosting or a server for a blog that probably will not get a lot of traffic
• Low maintenance overhead - I really don't want to fiddle with server management and updates
• Content is version controlled - There is power in simplicity. Content should be stored in plain text (preferably Markdown), authored in my favorite tools and stored under version control.
• Allows for custom domains

With this in mind, solutions like Wordpress really didn't fit the bill, and thus I started looking at what's out there in the Jamstack world.

I evaluated three different infrastructure providers (by no means complete, but one does what one can):

Github pages:

Pros:

• Seamless integration with Github

Cons:

• Forces you to use Jekill (personal preference... But I am not a Ruby man)

AWS S3 + Route 53

Pros:

• Rock solid
• I'm super familiar with the AWS ecosystem
• Flexibility - you can choose pretty much any site generator out there

Cons:

• SSL support requires usage of AWS Cloudfront
• Requires setting up a github action to deploy
• I would probably end up writing some Terraform code to manage the AWS resources - so more complexity

Cloudflare pages

Pros:

• Flexibility - you can choose pretty much any site generator out there
• Seamless integration with Github

Cons:

• N/A

Conclusion:

At the end of the day Cloudflare pages hit the sweet spot for my needs, and so far the experience has been stellar. The setup is straightforward and it amounts to allowing Cloudflare to access the github repo, and adding the build command and the target directory for the build (I'm using Docusaurus).

If you have a domain registered with Cloudflare associating it with the website is also quite straightforward - a much better experience that what you have in the AWS ecosystem.

For a long time I've been thinking about using the techlead.net domain to create my very own blog. While not terribly exciting, the reality is that this serves two purposes:

• It's a good way for me to be more disciplined in my own writing and learning process
• Become a place where I maintain a curated list of articles and resources I find relevant.

With that out of the way, I guess this is the customary Hello world post.