Documenting architecture of a mid-size chaos

How I documented the architecture of a mid-size platform using a Claude skill, went slightly mad and crazy and learned the skill is more valuable than the product.

Writing software architecture docs is horrible. Especially in a mid-size company that doesn’t have architect role or department. Even thinking about the architecture of the company’s information system seems so “redundant” and “holding us back”. Yet repeatedly, mid-size companies end up with duplicated systems, duplicated data, and chaos.

The Size of the Chaos

The platform started 15 years ago as one monolith, then it organically — without much forethought — started splitting into more modules (splitting UI was probably the first), services, repositories, etc. A few years later we agreed that the way to go is domain services, because:

• It is the right level of decomposition for our codebase and company size.
• It follows the status quo (the monolith could be classified as one domain with one eye closed) which was already working for us.

That decision was smart (even in retrospect) and still holds today. In engineering, the platform architecture understanding is fairly ok. The understanding of individual domains? Not so much. The understanding of interconnections and shared resources lives in tribal knowledge of ~3 people. Outside of engineering, the understanding is nil.

To frame the problem, we have over 1k GitHub repositories, out of those about 40 repositories are relevant to the platform itself — these are the “running services” not ephemeral components, extensions, experiments, and other garbage. This is about 1.8M lines of code (~70MB) in 6 programming languages. The ~40 repositories contain approximately 30 domain services and one monolith (that still contains some rarely used UI). Some of the repositories are mono-repositories (yes, we do have multiple mono-repositories) and some contain just libraries with reusable code. While just knowing all this represents a massive reduction of the problem, it still leaves plenty of code and dependencies to document. I’d consider this a mid-sized system that doesn’t fit into one person’s brain.

I have one great advantage though — I know the whole system very well. Out of the ~30 services I actively designed or developed ~15. About the rest I know what they do and how they interact with the outside world. Naturally, I’m far from knowing everything, but it helps tremendously, and I pity all you architects trying to document things you’ve never seen before.

For years I tried to generate and maintain the architecture docs, and I’m not afraid to say that I failed miserably. Twice I managed to produce some comprehensive docs, though the results weren’t that great.

Old architecture diagram, intentionally unreadable.

The above diagram was the best I got. It served some mild use during onboarding, but otherwise it was too clumsy to be useful. The main problem was there was never time to maintain it. Also, there is no rigid change process in the company. Without following everything everywhere, it was impossible to maintain it. The documentation was getting less and less useful every year.

A New Excuse to Try Again

A few weeks ago, the C-suite came with the idea that they will finally get rid of all the engineers so they can just write all the code themselves with AI agents. But neither of them understands the current stack too well. So I finally got the opportunity to spend some dedicated time to document the platform architecture.

The given goal was to replace all human engineers with AI. That suits me just fine, I have plenty of books to read, my garden is overgrown and maybe my todo list will finally start shrinking. But before I become unemployed I diverted from that goal a bit. The reason is I don’t believe that having architecture documentation is that groundbreaking thing standing between having human engineering and having just AI coding agents (more on that in another post). So I approached the problem with a slightly different — but more immediately practical — goal in mind. That is impact analysis, which is of two flavors:

• Development impact —e.g. If I want to change the Service 1 API contract, which services need to be updated? Which teams do I need to talk to?
• Operational impact — e.g. If the Service 1 API is down, what breaks? What doesn’t break? Who can fix it?

Both kinds of questions should be answerable from the architecture schema, not from someone’s memory.

Such questions can be answered to some extent by observability tools (e.g., we have Datadog). But there are limits — things that are not obvious from runtime go undetected. Recently, I was decommissioning an API and there were just two endpoints being called. Only after thorough code investigation did I discover a monthly cronjob that had slipped, then an on-demand job that hadn’t run for three months. Also, it is problematic to analyze cloud and other external resources. Observability tools will tell you that you’re calling S3, but they won’t tell you what bucket it is or what you’re doing with it. Generally, it’s difficult to assess the purposes of relationships — Datadog will tell you “S1 uses S2”. Code analysis will tell you “S1 stores user sessions in S2”. One might not notice this, because we know what S1 uses S2 for, but once you try to formally document it, the lack of information becomes apparent.

C4 Model

For the task, I chose C4 Model with Structurizr mostly because:

• It is git friendly — models are stored as plain text, changes are diffable and reviewable.
• It supports model validation — the DSL compiler finds dangling relations; inconsistencies surface immediately rather than going undetected in freeform diagrams.
• It produces a single model; relationships are defined once; rendered in standard L1/L2/L3 views (see below); custom views can be created without touching the model.
• It’s AI friendly — machine readable files, easy to generate and process by Claude Code and similar tools.
• It has !include concept — the model may be split into per-service files; adding or modifying a service doesn’t change unrelated ones.
• Enforces a consistent four-level hierarchy — offering a best practices how the model should be organized, rather than just being freeform diagrams.

I don’t really know of any alternative tool or standard having all these properties. If you do, be sure to let me know.

A quick C4 primer

So what is C4 anyway? C4 is a set of hierarchical abstractions for describing software architecture. Each level zooms into the previous one. That’s probably still very abstract and there are still more ways to interpret what to put at each hierarchy level. The important part is the zoom-in. Let’s consider a simple opinionated example close to what I want to achieve:

L1 (System Context) view of entire modeled IT system

On the top level I have the entire Software System I want to model. In this case one platform with two roles working with it. Next, I can zoom in on the platform to see what’s inside:

L2 (Container view) of the platform

The platform is composed of three Containers — services (S1-S3), two of which use one (shared) database server. Next I can zoom into service S3 to see what it is composed of:

L3 (Component view) of the S3 Service

On the L3 view I can see that the service is composed of multiple Components — API and worker, uses the database (together with S1) and additionally uses AWS S3 and AWS SQS services.

Simple, right? So what’s the fuss? The point is all of the above diagrams are stored in a simple text file:

workspace "Sample" "A simple C4 sample with two roles and three services." {
    model {
        # ---- People ----
        user  = person "User"  "End user."
        admin = person "Admin" "Administrator."

        # ---- The single software system ----
        platform = softwareSystem "Platform" "The platform composed of three services." {

            s1 = container "S1" "Service 1" {
                s1Api = component "S1 API" "S1 API" "HTTP"
            }

            s2 = container "S2" "Service 2" {
                s2Api    = component "S2 API" "S2 API"    "HTTP"
                s2Worker = component "Worker" "S2 worker" "Background process"
            }

            s3 = container "S3" "Service 3" {
                s3Api    = component "S3 API"  "S3 API"         "HTTP"
                s3Worker = component "Worker"  "S3 worker"      "Background process"
                s3S3     = component "AWS S3"  "Object storage" "AWS S3"
                s3Sqs    = component "AWS SQS" "Message queue"  "AWS SQS"
            }

            db = container "Database Server" "Hosts S1 and S3 databases" "RDBMS" "Database"

            # Component -> Database (container) relationships
            s1Api    -> db "Reads from and writes to"
            s3Api    -> db "Reads from and writes to"
            s3Worker -> db "Reads from and writes to"

            # Component -> Component relationships inside S3
            s3Api    -> s3S3  "Reads from and writes to"
            s3Worker -> s3S3  "Reads from and writes to"
            s3Api    -> s3Sqs "Publishes to"
            s3Worker -> s3Sqs "Consumes from"
        }

        # ---- Container-level relationships ----
        user  -> s1 "Uses"
        user  -> s2 "Uses"
        admin -> s2 "Uses"
        s1    -> s3 "Uses"
        s2    -> s3 "Uses"

        # ---- Component-level relationships (cross-container) ----
        s1Api -> s3Api "Calls"
        s2Api -> s3Api "Calls"
    }

    views {
        systemContext platform "Context" {
            include *
            autolayout lr
        }

        container platform "Containers" {
            include *
            autolayout lr
        }

        component s3 "S3-Components" {
            include *
            autolayout lr
        }
    }
}

If the architecture changes and Users start using the service S3 directly, all I need to do is add a single line user -> s3 “Uses”. The relationship will automatically show in all the views where it’s supposed to show. At the same time, if I would add guest -> s3 “Uses” , the model won’t compile, because role guest isn’t defined. Then with few lines I can create a new view, that will exclude the cloud resources, etc.

Just Build it!

All theory, my friend, is gray, but the golden tree of life springs ever green.

Mephistopheles meant this ironically — to mock science — which I don’t want to do, but he had a point there. I’d like to emphasize that while C4 answers many questions on how the model should be constructed, it doesn’t answer all questions. C4 has the following definitions:

• Software System — The highest level of abstraction — something that delivers value to a person;
• Container — An independently deployable unit within a system (process, service, database, etc.);
• Component — A grouping of related functionality within a container (e.g. a Deployment, a CronJob).

These are well explained on the site and I encourage everyone to read them in full at least twice. Yet when you start deciding on what to put where, new questions arise.

First, I started with the middle layer — containers. That might look weird to start with, but since defining the top abstraction (what the Software System is) is often difficult with most unknowns — I started with the most known thing in the middle.

Quirks & Wrinkles

The containers seemed to be “obvious”. The system has a public list of services, that lists 23 services on API index and at the same time lists 18 services APIs. Neither of the two lists (or another two lists that I found) is of course complete. Well, whatever; I’ll sort it out — be it 18, 23, or 35 services. Since the platform is composed of domain-services rather than microservices, it still seems clear. One domain, one service, one problem. For example Notifications are implemented by notification-api, located in notification-service repository. Single deployable unit, clearly a container.

Actually, no.

Except, the service itself is composed from multiple units. It contains an API and a couple of asynchronous workers. When I look at the Kubernetes templates, it’s 5 unique pods of constantly running code. Plus some things running on demand — typically database migration jobs. And I’m of course ignoring replicas as they’re merely dumb copies. They have no influence on the architecture schema IF the basic assumption is high availability.

Ok, ok, back to earth. The GitHub repository has a number of CI/CD pipelines, but only one of them creates production docker images. These are pushed to production image registry, which in turn triggers update of the Kubernetes deployments. That means that for all practical purposes the whole repository is one deployable unit and I treat it that way — a single C4 container. To corroborate that I asked some questions:

• Would someone expect to find a different image on the API and the worker? → No, it would be a surprise.
• Would it have any practical use case? → Unlikely, because they share the same code.
• How difficult is that the API would use a different image than a worker? → Quite Difficult.
• Is this something we want to start doing? → No, there is no benefit.
• In case of incident, how do we revert it? → We revert it as if it is one.

After this, I reformulated the C4 definition “Container — An independently deployable unit” for myself to “Container — A thing that’s treated as one deployable unit”.

Peculiarities and Irregularities

Then there is a service for running jobs, that is composed of 2–3 APIs (depending on who you ask), and about 15 workers. All of this is one mono-repository except for one worker which is so big that it resides in a separate repository (with 4 other workers). And then there are two more repositories with something else. Altogether it’s ~8 docker images being built and ~15 running Kubernetes resources. There are 3 CI/CD pipelines building the 8 docker images and triggering the update of the Kubernetes cluster. One repository is a mono-repository, where the CI/CD pipeline triggers deployment of two services at once (with two images), but only in some cases. Another part is in a separate repository, but at the same time it doesn’t have a CD pipeline on its own because it’s only deployed using another (3rd) repository. That third repository has CI/CD pipeline that builds one docker image, but triggers deploy of two docker images. Then one of the APIs is completely private because it’s not visible to any other service all, but at the same time it is deployed individually (sometimes).

This is madness.

Ok, ok, that’s enough. It’s a mess, but let those without sin throw the first model. The thing exists, is heavily used and I decided to model it as one container, because for (not all, but some) practical reasons it is one. If you ever touch any part of the beast, the chance of needing to touch all of them is high. Even if you’re working on the mono-repository you’ll often find yourself opening a PR to two other repositories, because they’re freaking closely coupled. I asked a couple of questions again:

• Would someone expect to find a different image on the API and the worker? → Yes, that’s normal, but they have the same build number.
• Would it have any practical use case? → Yes, it does happen when deploying stuff gradually.
• How difficult is that the API would use a different image than a worker? → Not a problem.
• Is this something we want to start doing? → We’re already doing it, maybe the right question is if we want to stop?
• In case of incident, how do we revert it? → We revert it per parts.

Not very persuasive, eh? In other words, I decided to treat multiple APIs, multiple Kubernetes resources and multiple docker images as one container because it’s sometimes being used like that and it’s always being thought about like that. Ok, now I’m literally executing on Conway Law, go on, shoot me. But in the end it seems appropriate. I’m here to document the existing situation, not my wishes about how reality should be.

Abominations and Mutants

Am I done yet? Of course not. There is the monolith service, which for many purposes we treat as 2–3 services (because we usually forget about the third one). Although they are being implemented and served by one API (one docker image, one Kubernetes deployment), they are so different that we often treat them separate things. Often, but not always. In case of an incident they’re still reverted as one, they’re being developed by one team, they use the same cloud resources, they share the same codebase and database. So I decided this to be one container too.

What am I trying to say?

Whatever “rule” I decided there was always some exception to it. Bear in mind that I don’t have 300 services, just 30. Still, there was always something that didn’t feel right or was clearly violating reality, when I applied rules mechanically. You could argue that we’re messy idiots unable to agree on a system, but I’d say that’s not the case. The whole application is 15 years old, it went through many “best practice” iterations simply because the world around us has changed. If the best practice changes, you start doing it, but you don’t stop for a month or so to bring every piece of codebase of this size to that newest, bestest, shiniest practice.

So a lot of gut feeling came into classifying what is a container, but eventually I think that falling back to the Conway law was the safe net. Though I didn’t do it intentionally — I only thought about it when writing this post.

From Containers up to Systems

All this was happening via a lengthy Claude conversation. During the conversation I discovered some containers I haven’t thought of uncommissioned services (but already partially running), and some decommissioned services (but not yet completely turned off). But most importantly I identified 2–3 satellite systems. It is something that looked like container initially (e.g. because it is listed as a service internally) but it turned out to be so disconnected and special that it didn’t fit into the main system at all.

All services are deployed in each of our 20+ platform stacks. A stack in our terminology is an isolated deployment in a given cloud provider and region (e.g., Azure North Europe or AWS US East). But then there is one service that is listed normally among services, but it is not deployed into those stacks at all and sits in a cloud of its own with only a scheduled based synchronization between the service and one of the stacks. While it’s not another product (because we only sell one product), it’s still so crazily different and disconnected that it doesn’t fit into the product and is treated as a separate satellite system.

Another thing that occurred to me when analyzing just the list of the services was that I had to deal with operational exceptions. The main platform is running in the stacks. Although there are 20+ of them, no two are identical. They differ in cloud provider being used (AWS/Azure/GCP). They have different forms of billing, some have some customer-specific extensions. Initially, I tried to capture this in the C4 and realized that it would be a mess. So I decided to treat the whole system in the model as an idealized stack — a stack that contains basically everything we have, meaning that the information gained from it can later be cut down (“on stack A, service B isn’t deployed, so finding C doesn’t apply”).

Side note: a colleague objected to this, saying — “just put everything in the C4, the AI will deal with it”. Well no, it won’t. Using Claude (mostly Sonnet 4.6) in building this was extremely helpful in this case as:

• it is aware of C4 and best practices,
• it can scan through the code at light speed,
• it can provide really valuable tips.

But throwing 30 services at it and saying “solve it” doesn’t work. The raw conversation just about what containers should we have and how to treat them was ~13MB and 11k+ lines (compacted 8 times). This does not include analyzing individual services to components (though it includes creation of skill to do so). Also, the whole point of C4 that it’s an architecture model and not every bogus-readme-todo-reference.

From Containers Down to Components

After the first round of work I ended up with three systems, and ~30 containers in the main platform. Now on to adding components inside the containers. One C4 definition of a component is “a grouping of related functionality encapsulated behind a well-defined interface”. Which suits me, so I picked the individual Kubernetes workloads to be the components. That way I have a notification container that implements the Notification domain service is composed of a notification-api and notification-queue-worker components. It seems natural to our cause, but it’s probably one abstraction level higher than the examples given in the C4 docs.

When analyzing individual services, another question popped up. What to do with cloud resources? As noted above, the main platform runs in all three main clouds (but never at once, that’s the idealized stack), each cloud provider having their own specialties.

All pods are running in managed EKS/AKS/GKE cluster. The C4 docs suggest putting the cloud resources as containers (because you manage and “deploy” them as other services). Technically, yes, we also have the cloud resources defined through Terraform in a dedicated repository.

But.

Get Ondřej Popelka’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

But practically, the ownership of the cloud resources is under the service. I.e. if a person working on Notification service decides they need an S3 bucket, they’ll add it directly into the infrastructure repository under the apps\notification-service folder. The bucket is deployed by the team managing that Notification service at the time, they don’t go and ask someone else to create the bucket for them. If the service is decommissioned, the bucket goes with it. That is also the responsibility of the application developers. So for all practical purposes, the S3 bucket is part of (i.e. a component of) that service (i.e container). Hello again Melvin Conway!

Enter the Skill

Like I said before, I did the whole analysis using Claude. During the analysis of the components, I also created a Claude skill to “automate” the analysis. “Automate” is in quotes, because it’s far from “just run this on a service and we’re done”. But it helps tremendously. Though the countryside of the services is vast and hilly and bumpy (remember 15 years and 6 languages!), there are still some regularities I can take advantage of.

Skills.

For example, the infrastructure sits in a separate repository where it’s organized by each stack type and service and it’s all in Terraform (with just a handful of exceptions and still using three different methods of organizing: everything in one .tf file, .tf file per cloud resource in cloud provider directory, .tf file per business requirement — e.g., queues.tf). But still! It’s not a blind “find out what cloud resources this service uses” but a direct “here are the Terraform files that define all cloud resources this service uses, they are organized either this way or this way or this way”. Providing this kind of information improves the Claude model output by a mile.

Another example is the definition of Kubernetes resources which is in another repository. It’s composed of Helm charts or Kubernetes templates organized by application. Again there are a lot of exceptions, but the basic system is there. Similarly, in PHP services, there is a kind of universal directory of other services, which makes finding interservice dependencies easy, because you just need to search for getNotificationServiceUrl(). Again there are irregularities, where someone forgot about it and just used str_replace(‘connection’, ‘notification’, self::url()); to obtain another service address.

I started the analysis with the services I’m most familiar with, so I instantly knew when Claude was pulling my leg (either missing a component or dependency, or hallucinating one). In correcting it, I updated the skill and retried. Since the skill contains a lot of non-public information, I created an anonymized version that you can browse on GitHub. It’s very representative of the actual content though. You can see this inside the general skill file:

### Sources of truth — read these only

- `{INFRASTRUCTURE_DIR}/terraform/modules/app-{SERVICE_NAME}/{aws,azure,gcp}/*.tf`
- `{INFRASTRUCTURE_DIR}/terraform/{cloud}/stage-prod/app_{SERVICE_NAME}.tf`
- `{K8S_STACKS_DIR}/apps/{SERVICE_NAME}/values.yaml` and `templates/`
- `{C4_DIR}/service-knowledge.md`

And then this in the Go specific skill file:

### 3d. Single Terraform module variant — check before assuming cloud split

Unlike PHP services, Go services often have a **single Terraform module** with
no `aws/`, `azure/`, `gcp/` subdirectories. This is the case when the service
only uses cross-cloud resources (e.g. shared Postgres injected from a shared
module) and has no cloud-specific resources.

If the module is flat: read `main.tf` directly. The module likely only creates
a K8s namespace and injects DB credentials — no service-owned cloud resources.
If subdirectories exist, apply the full Step 1 procedure from the common skill.

See? I didn’t lie to you when I said there are three methods of organizing the Terraform files. The key to the successful skill was finding out how to find the required information. What is the best source of truth?

Improve your skills

For example, for a PHP service, the Symfony services.yaml is taken as a source of truth for environment variables the service requires, together with Kubernetes values.yaml, which provides the environment values to the service. I learned to completely skip the .env files; though they exist, but they are messy.

The order of taking each piece of information is also important — it starts with the environment values the service definitely needs. Then it looks at what environment values are provided and tries to resolve the inconsistencies (extra environment values might be used somewhere other than Symfony service definition, or they might be used dynamically, etc.).

It’s important to construct the skill so it knows how to find the authoritative information, rather than look for some authoritative list. Because in reality, the landscape is so messy, that no such list exists.

As another example — the skill has an instruction like this:

| `Platform\Encryptor\*` | application-level KMS / Key Vault key | NOT encryption-at-rest. Resolve the actual `kms-key-*…

Some services use cloud managed keys (AWS KMS/Azure KeyVault, Google KMS) for encrypting stuff. This is done using one of the two libraries (PHP or Go). The library then might assume some environment variables about the keys. So the authoritative information is the library present in the code, not the environment values present. If you think the other way would be better, it’s not. Looking just at the Terraform files, Claude would assume that a service has cloud managed keys because it can find a link between the database server and the keys. But these are different keys! They serve the encryption-at-rest purpose. Or Claude may assume that the service cloud managed keys are used for encryption-at-rest too, which is not the case either! There’s lot of places where it can go wrong.

To keep the skill general enough I extracted the service specific knowledge to service-knowledge.md, this describes the exceptions. The motivation to keep the skill general isn’t to have one skill for everything everywhere (because the skill is just a bunch of exceptions anyway), but to make the skill futureproof. The tricky part is distinguishing what goes where. The general rules and general exceptions (where’s the facepalm emoji when you need it?) are part of the skill. The per-service exceptions are part of the service-knowledge.md file

The skill contains information on how the infrastructure is organized and how the services are generally organized (most PHP services follow the same pattern). It contains information about shared libraries (e.g., the Platform\Encryptor), because their interfaces are usually stable (both in the sense of being time-proven and difficult to change). On the other hand there is some information which is purely service-dependent mess.

For example, most PHP services use a central library to determine other service URLs (“### 3a. services.yaml — ServiceClient method calls” in the PHP skill file). Some Python services take one service and replace part of the DNS (“### 3c. URL derivation patterns (CRITICAL — easy to miss)’ in the Python service skill file). And then there’s one service which has a LOG_RECORD_URL environment variable that’s supposed to point to a different service (and no, it’s not a logging service). There’s no reasonable way how to gain this information other than digging through the readme or the code. If you miss it, you’re dead. Instead of trying to force the general skill not to miss the one-line note in one readme file of one specific service (which is unreliable and would pollute the skill), I’d rather explicitly state this is in the knowledge file. So there is this entry in the service-knowledge.md for the incriminated service:

## dreaded-service

### appSecrets ENV mappings
| ENV key | Target container ID | Notes |
| — -| — -| — -|
| `LOG_RECORD_URL` | `stream` | Stream service ingest endpoint for recording feed |

Rework

When I was done with scanning and reviewing about 10 services I thought that the skill is good enough, so I handed it to my colleagues, and it worked! They were able to describe the architecture of two more services through a simple prompt:

In path_to\c4\ there is a skill to generate c4 docs for the Connection service path_to_service\connection

Yes, building the skill paid off. Though it was everything but quick and easy. The skill isn’t just description of a task — it’s a guide for navigating our jungle of services.

When I looked at the partial result, I decided to change a couple of things in the model. One was removing the Kubernetes cluster dependency “runs on”. The whole system runs on Kubernetes. Adding it to every component seemed to be just a noise in the C4 model. I kept only the other types of Kubernetes dependencies. For example, we have a service that “runs on” Kubernetes and simultaneously executes user-defined workloads on that same Kubernetes cluster. The first relationship is hidden (it’s automatic), the second relationship is kept.

Next, I grouped the database instances into one container and not cloud vendor — e.g., an application uses either Azure/AWS/GCP hosted MySQL, but it does not really care about the cloud hosting, it just needs DB connection string. This is different from other cloud resources where queues, storage or keys are still recorded as individual resources. The drawing illustrates the change:

Grouping of cloud resources.

Why? Again, it’s being treated differently. While the cloud resources are normally owned by the service (and the people developing it), the DB server instances are owned by the SRE team (but the databases on the servers are again owned by the respective services). Put differently: while the service cares about individual implementations of e.g., cloud queues (at least to the point that it needs to have three different libraries in dependencies), it doesn’t care about the hosting of the MySQL server, because it only cares about the DB connection string.

Then I made another change — record specific instances for each service. I initially made it just classes (e.g., Service 1 depends on SQS). I changed it to record the actual cloud resource instances, the drawing below illustrates the change:

Ungrouping of cloud resources.

The grouping to cloud resource class proved impractical because it hides dependencies. Yes, we have multiple services using the same resource. Does it go against the ownership principle stated above? Yes, sure! Did I mention there are exceptions? More than that it is also impractical from the point of view of the impact analysis (what happens if we change this queue message format?) and it is also incorrect from the point of view of C4. My initial motivation why start with the grouping was that there would be too many resources in the model and that we should leave this detail for the 4th level of C4. That was a mistake.

Don’t forget about the rules.

Then I did some minor changes; for example, I excluded all Datadog dependencies, because they again are default. I’m getting on thin ice here. One could argue that maybe not having a service properly tied to observability tool is something that should stand out in the architecture diagram. Luckily, we have it setup so that it appears there automatically, so it’s a known default in my case. I would consider this fine tuning.

After all these changes, another round of re-scans and reviews came, which I had to repeat few times again (to properly identify the individual cloud resources). Thank God for the skill. But still, you have to keep in mind that it’s an LLM and it can make very plausible but incorrect conclusions. The great benefit of using AI is that it won’t argue about its infallibility. If only all people were like this! So I made reviews and corrections:

I confirmed the application-side queue boundary, but the actual Terraform resource names (and whether session/connection-events truly use distinct queues per cloud) live in stacks/.../app_service.tf, which I didn't read.

And corrections:

So your intuition was right. Only main and commands are true Pattern A (direct publish/consume). Everything else — audit-log, events-elastic— is fan-out via SNS/Event Grid/Pub/Sub, with Connection's own worker queue as one subscriber alongside any cross-service subscribers.

And Corrections:

Good instinct to verify. Let me check the stacks Elasticsearch chart and see if there are actually multiple deployments: You were right. There are two separate Elasticsearch instances in one chart

Etc.; altogether I rescanned each service 5–6 times on average and made about five conversation turns per service, which is pretty remarkably low, imho.

Gold-plating

Colors! As the model developed, the colors became a mess. Luckily it doesn’t mean complete rework. But it requires to stop, drop, and think. After all these iterations I ended with a pretty messy system of colors:

• C4 hierarchy — Persons, System, Container, Component.
• Languages — GitHub-ish brand colors (PHP, Go, Python, TS, JS, Other).
• Lifecycle — Legacy and Uncommissioned.
• Cloud / external — vendor brand colors per service (SQS, SNS, S3, …).

One caveat is that the force-diagram rendering in Structurizr supports only circles for shapes and then you can have two colors — one for the fill and one for the border. Therefore, you can encode only two dimensions of information. The rendered diagram also allows different shapes. Since I found that we’ll probably use more the force-diagram, I decided to go for two and a half:

• colors to the resource types (Queue, DB, Keys, etc),
• border coloring for cloud vendors Azure/AWS/GCP,
• legacy / decommissioned coloring because that applies only to containers and shouldn’t clash with the types.

Results & Learning & Takeaways

The real resulting view of containers (anonymized, we don’t have a FlugKräuter service).

The model itself is not that valuable, so fiddling with the colors is gold-plating. The valuable part is the skill and the service-knowledge file, which basically contains my brain dump of all the exceptions I could think of or find. Finally I’m replaceable and not a walking company library.

The result: the model is surprisingly small — ~200KB including generated service descriptions (~140KB without them). It comfortably fits into a model context window, so I can ask the AI questions like “What will happen if the notification service goes down?”. And I get a very correct answer quickly (without scanning 20+ repositories) like this (abbreviated & anonymized):

Impact if the Notification service goes down:

Direct callers

Only two components have an explicit dependency on notification, both inside the queue container, from the worker-type-2. No other container or component in the model talks to notification.

Functionality lost

Subscription-based platform event notifications — most notably the “failed job” / job lifecycle emails and webhooks that users configure per project.

Any UI surface that reads from the notification service for subscription management.

Functionality NOT lost (this is the important part)

Jobs still run. The daemon’s dependency on notification is fire-and-forget for lifecycle events. Job scheduling, flow transitions, and pod spawning, have no notification dependency.

Transactional email is unaffected. Invitations, token sharing, project notifications go via connection-api → sendgrid directly, bypassing the notification service entirely.

Besides this, I want to include it in a review bot that will quickly be able to determine if there are some dangling ends to a pull request.

This whole thing took slightly more than 6 days to construct, with a lot of rework. The skill itself easily runs 10–30 minutes on a single service and then it depends on how many turns you need to get the result correct. I still consider that fast because I happen to have extensive knowledge of the system so I could fix the errors very early in the process.

With the model, the issue is of course updates. I haven’t automated this yet, but the idea is to run the skill with the instruction to make a diff between two git commit hashes. Again that works very well, so when I ask:

In pathTo\c4\ there is C4 model, it captures connection service (pathTo\connection\) on commit fede1e2efe, what would need to change in the model to capture the state at commit da7a6a8a8a?

I get:

Between da7a6a8a8a (Apr 18) and fede1e2efe (Apr 28) only one L3-model-relevant change happened in the connection source: PR #1234 / AJDA-4321 (“remove sandboxes-api-php-client dependency”), merged on April 24. It deleted:

keboola/sandboxes-api-php-client from composer.json (verified — gone at fede1e2efe) usage from ChangeSnowflakeHostname (Snowflake hostname change CLI) and ResetSandboxesPasswordForProject (BYODB password reset CLI)…

That is exactly the source of the edge currently modelled in l3/connection.dsl:
connection-manage-api -> sandboxes “Updates sandbox credentials during BYODB migrations and Snowflake hostname changes (CLI commands)”

I dare say that’s fucking cool — ten days of changes (53 non-trivial merge commits) compressed to a single updated line. Such a thing can run once a week and assign code owners of the services for approval of the model changes if any are detected.

Some learnings:

• Have a clear idea of why you’re modelling the architecture. If you just want to look at diagrams or optimize request speed by avoiding unnecessary connections — observability tools may be a better choice.
• Try to find any regularities, customs & conventions. Even if they don’t always hold, they are still very helpful.
• Having a dedicated description of known exceptions is very helpful too, it is much better than trying to pretend there are none.
• For sure, use AI (and use the best you have available; this kind of task needs it).
• For God’s sake, don’t trust it (and do your best to fact check everything).
• Starting with the most clear thing is the way to go, the others will bubble up. If it fails, rework it. It’s better than starting with ideas and then trying to coerce reality into them.
• Order is important, the model is not completed until fully scanned (the scanning is only one way “what this thing uses”). If you choose wisely, you can imagine how the final result will look and rework midway.

Some takeaways that really stand out:

• What I would consider the most important and difficult is distinguishing what is important and what not, spend your energy on that.
• Creating the skill and using it is the way to go. The first C4 version is probably not the right one. The inevitable rework is a piece of cake if you are prepared. The skill is more valuable than the model itself.
• Resist the urge to capture everything — the beauty of a C4 model or any architecture model is that it’s relatively small in size and can easily be reasoned about.
• Look at how things are treated and understood (not how they’re committed/deployed/etc.). Document reality, not wishes.