A Couple Million Lines of Haskell: Production Engineering at Mercury

A Couple Million Lines of Haskell: Production Engineering at Mercury

This was surprisingly relevant and pragmatic essay.

A system operates reliably because it can absorb variation: it degrades gracefully, its operators can understand and adjust it, and the architecture makes the right thing easy and the wrong thing difficult

Very interesting variation of Quality by Design thinking

This is why I increasingly think of the type system as an operational aid more than a correctness proof. […] In a fast-growing company (any company really), people leave, people transfer teams, people go on vacation or parental leave, people join, and the churn means that things people knew walk out the door with them unless you have written them down somewhere. Ideally, you have written them down in a form that the compiler can read, because the compiler is much more disciplined than the average wiki page.

Never getting tired of seeing something great achieved by aiming at completely different target. Ability to be checked by an independent system, preferably automated, seems to be really strong power for many designs.

The type system has not proven anything especially deep about your events. It has done something more practical: it has made the correct operational procedure the path of least resistance.

Great example of pattern I saw relatively often in the libraries where I needed to remember correct order of actions. I guess it can also help agents a lot but this is not a given.

Durable Execution

Temporal is indeed very nice system and it’s likely most challenges people have with Temporal model stems from the inherent complexity of the operations rather than accidental complexity of Temporal itself.

The fix is conceptually simple: model your domain errors as domain types. A payment that fails because of insufficient funds should be an InsufficientFunds, not a 402. A duplicate request should be a DuplicateRequest, not a 409. These are things your business logic can match on, retry against, log meaningfully, and handle differently depending on context.

In practice it’s quite hard to nail what is domain and what is transport from the first attempt.

Encoding invariants into types is powerful. It is also expensive. Not at runtime, but in cognitive overhead, in the rigidity it introduces, and in the difficulty of changing things later when the requirements shift. And the requirements will shift. […] . If the constraint is “we currently happen to do things this way,” or “I read this article about dependent types and I simply must apply that to my authorization logic,” you have likely just made your codebase harder to change for no operational benefit.

Where this boundary lies seems to also be very hard to catch reliably. The simplest logic and decision-making probably puts a “cost” on amount of types and then one needs to justify if their particular domain has enough budget to afford many specific types.

The solution I reach for most often is records of functions. Instead of exposing a module full of concrete functions, you expose a record whose fields are the functions.

I think this pattern is quite common in Typescript when developers implement DI without container. Accept functions as parameters and pass containers with functions if necessary.

Designing for Introspection. […] You can inject faults for testing. You can swap the implementation for a mock.

Those are also requirements for systems written to support discrete event simulations. Introspection ability as well as simulation-readiness might be critical for agentic development while simultaneously unblocked by the same agentic systems.

The hiring pool problem is real if you need a hundred Haskell experts tomorrow. It is much less real if you are willing to hire good generalists and invest in teaching them.

Great that it works for their team.

We have found that the investment pays off on the order of months, not years. This is particularly true in financial services, where the cost of a data integrity bug is measured not in user complaints but in regulatory findings and in other people’s money.

It’s surprising speed to positive ROI.