Test strategy and integration testing

Chapters 16 and 17 cover the tools. This one covers the judgement: what to test, at which level, with which kind of double, and how to tell a test that catches bugs from a test that merely runs code. It is the chapter that maps onto the interview question “how do you decide what to test”, which is asked far more often than “what does mock.fn do”.

The code is language-agnostic in intent and written in Python for compactness — 21 tests plus a mutation-testing harness, all executed on CPython 3.11.15. Everything transfers directly to node:test.

Table of contents

• 1. What a test is for
• 2. The five test doubles
• 3. Mocks versus fakes, and the brittleness tax
• 4. The shape of a test suite
• 5. What to test at which level
• 6. Testability is a design property
• 7. Functional core, imperative shell
• 8. What makes a test good
• 9. Does your suite actually catch bugs?
• 10. Choosing test cases
• 11. Test data: fixtures, factories, builders
• 12. Testing the hard things
• 13. Flaky tests
• 14. Contract testing
• 15. Property-based testing
• 16. Adding tests to code that has none
• 17. TDD, honestly
• 18. Tests in CI
• 19. Interview questions

Measured follow-up. The pyramid/trophy/honeycomb discussion below is a design argument. It now has a measurement attached: a unit-only and an integration-only suite over the same module, at equal effort, both reaching 100% statement coverage and both scoring 11/15 on mutation — while killing different mutants. See 20 §4.

1. What a test is for

Tests buy four different things, and confusing them is the root of most bad testing arguments:

1. Regression protection. The code works now; a test makes tomorrow’s change safe. This is the main economic justification and the reason coverage of changed code matters more than absolute coverage.
2. Design pressure. Code that is hard to test is usually badly coupled. Writing the test first turns that into feedback you get in minutes rather than months.
3. Documentation. A test named does_not_decrement_stock_when_payment_fails states a requirement that no comment will keep current.
4. Debuggability. A failing unit test localizes a bug to a function. A failing end-to-end test tells you the system is broken.

The costs are equally real: tests are code you maintain, they slow the feedback loop, and a test coupled to the implementation makes refactoring harder rather than safer. A test suite is an asset only if it fails when the behaviour breaks and stays green when the implementation changes. Every decision in this chapter comes back to that sentence.

2. The five test doubles

graph TD
    TD["Test Double"] --> Dummy["Dummy<br/>(never called)"]
    TD --> Stub["Stub<br/>(canned answers)"]
    TD --> Spy["Spy<br/>(records, assert after)"]
    TD --> Mock["Mock<br/>(expectations up front)"]
    TD --> Fake["Fake<br/>(real, simpler impl)"]
    Dummy --> NV["No verification"]
    Stub --> NV
    Spy --> IV["Interaction verification"]
    Mock --> IV
    Fake --> SV["State verification"]
    SV -.->|"contract test"| Real["Real dependency"]

Gerard Meszaros’s taxonomy. The words get used interchangeably in conversation, and knowing the actual distinctions is a reliable senior signal — mostly because it forces you to say what you are verifying.

All five, doing the same job, so the difference is visible:

class SignupService:
    def __init__(self, store, notifier, now):
        self.store, self.notifier, self.now = store, notifier, now

    def register(self, email):
        if "@" not in email: raise ValueError("invalid email")
        if self.store.get(email) is not None: raise ValueError("already registered")
        user = {"email": email, "created_at": self.now()}
        self.store.put(email, user)
        self.notifier.send(email, "welcome")
        return user

# 1. DUMMY — must never be called; says so loudly if it is.
class DummyNotifier:
    def send(self, to, body): raise AssertionError("a dummy must never be called")

# 2. STUB — canned answers, accepts and forgets.
class StubStore:
    def __init__(self, existing=None): self._existing = existing or {}
    def get(self, k): return self._existing.get(k)
    def put(self, k, v): pass

# 3. SPY — records; you assert afterwards.
class SpyNotifier:
    def __init__(self): self.sent = []
    def send(self, to, body): self.sent.append((to, body)); return "msg_1"

# 4. MOCK — expectations first, verified at the end.
class MockNotifier:
    def __init__(self, expected): self._expected, self._actual = expected, []
    def send(self, to, body): self._actual.append((to, body)); return "msg_1"
    def verify(self): assert self._actual == self._expected

# 5. FAKE — a working implementation that enforces the real constraints.
class FakeStore:
    def __init__(self): self._d = {}
    def get(self, k): return self._d.get(k)
    def put(self, k, v):
        if k in self._d: raise KeyError(f"duplicate key {k!r}")   # the REAL invariant
        self._d[k] = dict(v)

Note that a dummy that raises is better than a dummy that no-ops: if the code path changes and the collaborator is reached, the test tells you instead of silently passing.

The demonstration that matters

The fake is the only double that catches this bug:

def test_the_fake_is_the_only_double_that_catches_this(self):
    stub_svc = SignupService(StubStore(), SpyNotifier(), now=lambda: 0)
    stub_svc.register("a@b.c")
    stub_svc.register("a@b.c")           # NO ERROR — the stub never stored anything

    fake_svc = SignupService(FakeStore(), SpyNotifier(), now=lambda: 0)
    fake_svc.register("a@b.c")
    with self.assertRaises(ValueError):
        fake_svc.register("a@b.c")       # the fake remembers

A stub whose put forgets makes the duplicate-registration check untestable, and worse, makes it look tested. That is the strongest practical argument for building a fake of anything stateful.

3. Mocks versus fakes, and the brittleness tax

The real distinction is not the vocabulary; it is state verification versus interaction verification.

def test_brittle_interaction_test(self):
    store = create_autospec(FakeStore, instance=True)
    store.get.return_value = None
    svc.register("a@b.c")
    # Asserts the IMPLEMENTATION: one get, then one put, in that order.
    store.get.assert_called_once_with("a@b.c")
    store.put.assert_called_once()
    self.assertEqual([c[0] for c in store.method_calls], ["get", "put"])

def test_robust_state_test(self):
    fake = FakeStore()
    svc.register("a@b.c")
    # Asserts the OUTCOME. Any refactor that preserves the outcome keeps this green.
    self.assertEqual(fake.get("a@b.c")["email"], "a@b.c")

Both pass. Now add a cache lookup before the get, or reorder for efficiency, or batch the two calls into one upsert. The first test breaks with no bug introduced; the second stays green. That is the brittleness tax, and it is why an over-mocked suite eventually gets deleted instead of maintained.

When interaction verification is right

Mocks earn their keep when the interaction is the requirement and there is no state to inspect:

• “Did not do the dangerous thing.” payments.charge.assert_not_called() after a failure is a first-class assertion and there is no state that proves it.
• Ordering requirements. “Validates before touching the database” is asserted by repo.find.assert_not_called(), and nothing else can express it.
• Fire-and-forget side effects. An email sent, a metric emitted, an audit line written — the return value is nothing, so the call is the behaviour.
• Third-party boundaries you must not cross in a test. A payment gateway, an SMS provider.

The heuristic: assert on state when there is state; assert on interactions when the interaction is the observable behaviour. And prefer a fake over a mock for anything you own and that has state.

The obligation a fake creates

A fake is a second implementation, so it can drift. The answer is a contract test suite both must pass:

def _contract(self, store):
    self.assertIsNone(store.get("missing"))
    store.put("x", {"v": 1})
    self.assertEqual(store.get("x")["v"], 1)
    with self.assertRaises(KeyError):
        store.put("x", {"v": 2})

def test_fake_satisfies_the_contract(self):  self._contract(FakeStore())
def test_real_satisfies_the_same_contract(self): self._contract(RealStore())

That is the complete answer to “how do you know your in-memory double behaves like the real database”. Implementation details in unittest §16.

4. The shape of a test suite

The pyramid (many unit, some integration, few end-to-end) is the classic model, and its logic still holds: cost and runtime rise with the level, and localization falls.

Two well-known corrections worth knowing by name:

• The testing trophy (Kent C. Dodds) fattens the integration layer, on the argument that for application code — especially UI — most bugs live in the wiring, not in the units. Reasonable for frontends and thin CRUD services.
• The honeycomb (Spotify) applies the same argument to microservices: heavy integration testing at the service boundary, few unit tests, few end-to-end.

My read, and a defensible interview position: the right shape follows from where your complexity lives. A library with real algorithms (a parser, a scheduler, a pricing engine) is pyramid-shaped because the logic is in the units. A service that mostly moves JSON between an HTTP handler and a database is trophy-shaped because there is barely any logic to unit test and all the risk is in the wiring, the SQL and the serialization. Arguing for one universal ratio is the weaker answer; naming the trade-off is the stronger one.

The ice cream cone is the anti-pattern: mostly manual and end-to-end testing, few unit tests. It is slow, flaky, and gives no localization. Recognize it by “our test suite takes 90 minutes and we retry failures”.

5. What to test at which level

A decision procedure that works:

The two failure modes to name:

Testing implementation details. Private methods, exact call sequences, internal data structures. The test breaks on every refactor and you learn to ignore or delete tests. Test through the public interface; if a private method is complex enough to need direct tests, it wants to be its own unit (section 6).

Mocking what you should integrate with. A repository mock that returns {"stock": 4} encodes your belief about the schema. When the column is renamed, every unit test still passes and production breaks. That is what the SQLite constraint tests in node:test §15 and unittest §14 exist to demonstrate: a mocked repository accepts a duplicate primary key, a negative price, and an order against a nonexistent SKU. The real engine refuses all three.

6. Testability is a design property

Untestable code is a design smell with a name for each cause. Here is code with all four, and each fix:

def process_order_untestable(sku, qty):
    order_id = f"ord_{random.randint(1000, 9999)}"     # hard-coded randomness
    if os.environ.get("STAGE") == "prod":              # hard-coded configuration
        pass
    return {"id": order_id, "sku": sku, "qty": qty, "at": time.time()}   # hard-coded clock

Testing it requires patching three module paths:

with patch("mod.random.randint", return_value=1234), \
     patch("mod.time.time", return_value=99), \
     patch.dict(os.environ, {"STAGE": "test"}):
    out = process_order_untestable("ABC", 2)

That works — and it couples the test to the module layout, the import style, and the standard library. Rename the module and the test breaks with no production change. Compare:

def process_order(sku, qty, *, now=time.time, gen_id=None, stage=None):
    gen_id = gen_id or (lambda: f"ord_{random.randint(1000, 9999)}")
    stage = stage if stage is not None else os.environ.get("STAGE", "dev")
    return {"id": gen_id(), "sku": sku, "qty": qty, "at": now(), "stage": stage}

# The test:
out = process_order("ABC", 2, now=lambda: 99, gen_id=lambda: "ord_1234", stage="test")

No patching, no string paths, and the production default is unchanged. Every patch("a.b.c") in a suite is a small piece of coupling between your tests and your module layout. Some of it is unavoidable (third-party code, legacy code); the amount that is avoidable is a design measurement.

The four seams, in the order I would try them:

1. Parameterize the dependency with a default. Zero cost to callers, full control in tests.

2. Wrap the dependency behind an interface you own — a Clock class rather than time.time scattered through the code:

   class FrozenClock:
       def __init__(self, t): self._t = t
       def now(self): return self._t
       def advance(self, d): self._t += d
   
   clock = FrozenClock(1000); s = Session(clock)
   assert not s.expired()
   clock.advance(31)
   assert s.expired()

3. Sprout a method — extract the new or interesting logic out of a big untested function and test the extract, leaving the scary function alone:

   class LegacyReport:
       def render(self, rows):                       # still untested, still scary
           lines = [self._format_row(r) for r in rows]
           return "\n".join(["REPORT"] + lines + [f"TOTAL {self._total(rows)}"])
       def _format_row(self, r):                     # new, fully testable in isolation
           return f"{r['sku']:<6}{r['qty']:>4}{r['price'] * r['qty']:>8}"

4. Extract the pure core — section 7.

7. Functional core, imperative shell

The highest-leverage testing decision in most codebases: push the decisions into pure functions and keep the I/O in a thin shell.

def decide(stock: int, requested: int, price: int) -> dict:
    """Pure: no clock, no I/O, no randomness. Trivially and exhaustively testable."""
    if requested <= 0:    return {"ok": False, "reason": "bad_qty"}
    if stock < requested: return {"ok": False, "reason": "out_of_stock"}
    return {"ok": True, "total": price * requested}


def place_order(repo, sku, qty):                    # the imperative shell
    item = repo.find(sku)
    decision = decide(item["stock"], qty, item["price"])
    if not decision["ok"]:
        raise ValueError(decision["reason"])
    repo.decrement(sku, qty)
    return decision["total"]

The core gets a table-driven test with no doubles at all:

cases = [(10, 0, 5,  {"ok": False, "reason": "bad_qty"}),
         (10, -1, 5, {"ok": False, "reason": "bad_qty"}),
         (1, 5, 5,   {"ok": False, "reason": "out_of_stock"}),
         (5, 5, 5,   {"ok": True, "total": 25}),
         (10, 3, 7,  {"ok": True, "total": 21})]
for stock, req, price, expected in cases:
    with self.subTest(stock=stock, req=req):
        self.assertEqual(decide(stock, req, price), expected)

The shell needs one thin test per path, and those are the only tests that need a double:

repo = Mock(**{"find.return_value": {"stock": 5, "price": 10}})
assert place_order(repo, "ABC", 2) == 20
repo.decrement.assert_called_once_with("ABC", 2)

repo2 = Mock(**{"find.return_value": {"stock": 0, "price": 10}})
with self.assertRaisesRegex(ValueError, "out_of_stock"):
    place_order(repo2, "ABC", 1)
repo2.decrement.assert_not_called()

What this buys: the branch matrix — the part with real combinatorial complexity — is tested with plain values and no setup, and the mocked tests shrink to two. Compare with the version where decide is inlined into place_order: every branch needs a configured repository mock, and the tests are five times longer for the same coverage.

Same idea, different names: hexagonal architecture, ports and adapters, “dependency rejection”. The testing consequence is the one that makes it worth doing.

8. What makes a test good

F.I.R.S.T. — the properties, and what each one buys:

Structure: arrange, act, assert

def test_does_not_decrement_stock_when_payment_fails(self):
    # arrange
    self.charge.side_effect = RuntimeError("card declined")
    # act
    with self.assertRaisesRegex(RuntimeError, "card declined"):
        self.svc.place("ABC", 1, "tok")
    # assert
    self.repo.decrement.assert_not_called()

Three rules that survive contact with reality:

Name the behaviour, not the method. test_place tells you nothing when it fails at 2am. test_does_not_decrement_stock_when_payment_fails tells you the requirement, and the failure message is then almost redundant. The should_X_when_Y or X_when_Y shape forces you to name a condition.

One reason to fail per test. Not literally one assertion — asserting five fields of one returned object is one reason. But a test that exercises two behaviours will fail for two reasons and you lose the localization that made unit tests worth writing.

No logic in tests. No if, no loops over branches, no computing the expected value with the same algorithm the code uses. A loop over a table of cases is fine (that is subTest); a loop that computes what the answer should be is a second implementation with its own bugs.

The single best question to ask about a test

“If I broke the code, would this test fail?” If you cannot answer yes immediately, the test is decoration. Which leads directly to the next section.

9. Does your suite actually catch bugs?

Coverage cannot answer that question. Mutation testing can: change the code slightly, and see whether the suite notices. Surviving mutants are places where you have coverage and no verification.

I ran it by hand on two suites over the same two functions, both with 100% line coverage:

def clamp(x, lo, hi):
    if x < lo: return lo
    if x > hi: return hi
    return x

def discount(total, qty):
    if qty >= 10: return round(total * 0.9)
    if qty >= 5:  return round(total * 0.95)
    return total

class WeakSuite(unittest.TestCase):
    """100% line coverage. Verifies almost nothing."""
    def test_clamp_runs(self):
        clamp(5, 0, 10); clamp(-1, 0, 10); clamp(99, 0, 10)
        self.assertTrue(True)

class StrongSuite(unittest.TestCase):
    """Same coverage, real assertions, including boundaries."""
    def test_discount_boundaries(self):
        for total, qty, want in [(100, 1, 100), (100, 4, 100), (100, 5, 95),
                                 (100, 9, 95), (100, 10, 90), (100, 20, 90)]:
            with self.subTest(qty=qty): self.assertEqual(discount(total, qty), want)

Five mutants, run against both:

mutant                           WeakSuite   StrongSuite
--------------------------------------------------------
clamp: < becomes <=               survived      survived
clamp: swap lo/hi returns         survived        KILLED
discount: 10 becomes 11           survived        KILLED
discount: 0.9 becomes 0.8         survived        KILLED
discount: >= becomes >            survived        KILLED
--------------------------------------------------------
mutation score                         0/5           4/5

Both suites have 100% line coverage of clamp() and discount().

0/5 versus 4/5, at identical coverage. That table is the most useful thing in this chapter: it makes concrete why “we have 90% coverage” is not an answer to “is your code tested”.

The surviving mutant in the strong suite is worth its own note. x < lo becoming x <= lo cannot be killed: when x == lo, the original returns x and the mutant returns lo, which are the same value. That is an equivalent mutant — a mutation that provably does not change behaviour — and it is the known limitation of mutation testing. Mutation scores are never 100%, and chasing the last few percent means writing tests for distinctions that do not exist.

Real tools: mutmut or cosmic-ray for Python, StrykerJS for JavaScript and TypeScript. They are slow (the suite runs once per mutant), so run them on the module you are about to refactor rather than the whole codebase. The concept is worth more than the tooling in an interview.

10. Choosing test cases

Coverage tells you what you ran. These techniques tell you what to run.

Equivalence partitioning. Group inputs that should behave the same and test one per group. For qty: negative, zero, one, typical, at the stock limit, above the limit, non-integer. Seven cases, not seven hundred.

Boundary value analysis. Bugs cluster at boundaries, so test at and either side of each one. The mutation table above is the proof: the >= 10 versus > 10 mutant is killed only by testing qty=9 and qty=10 — and qty=5 and qty=4 for the other threshold. Off-by-one is the most common bug class in the industry and boundary testing is its direct antidote.

Error paths and the unhappy path. Most suites test the happy path and one error. Enumerate: invalid input, missing resource, dependency failure, timeout, partial failure, concurrent modification, permission denied. In the order service, the interesting tests are the ones asserting the card was not charged and the stock was not decremented.

Decision tables for combinatorial logic. Three booleans is eight rows; write the table, then test the rows. If it is 64 rows, that is a signal the logic wants decomposing.

Zero, one, many. The classic collection triple. Add: null/None, empty string, whitespace-only, duplicates, already-sorted, reverse-sorted, all-equal, one element, maximum size. That list is the same one from Sorting §13, where the 12 sorts are checked against exactly those 8 input shapes.

The bug you just fixed. Every bug is evidence that a test case was missing. Write it at the lowest level that reproduces it, and note that this is the only test case selection technique with a perfect hit rate.

11. Test data: fixtures, factories, builders

The problem: tests need objects, most fields are irrelevant to any given test, and the irrelevant fields hide the relevant ones.

# The noise problem: which field is this test about?
order = Order(id=1, sku="ABC", qty=2, price=250, currency="USD", customer_id=99,
              created_at=..., status="draft", discount=0, shipping=None, notes="")

Three answers, in increasing order of power:

Shared fixtures (setUp, a module-level constant). Cheap, and they create coupling: a test that needs a different value either mutates the shared object (breaking isolation) or the fixture grows to serve everyone.

Factories with defaults — the pattern that solves 90% of it:

def make_order(**overrides):
    return {"id": 1, "sku": "ABC", "qty": 2, "price": 250, "currency": "USD",
            "status": "draft", **overrides}

def test_rejects_zero_quantity(self):
    with self.assertRaises(ValueError):
        validate(make_order(qty=0))          # the ONE relevant field is the only one visible

Every test states exactly what is different about its world and nothing else. This is the same makeService(overrides) shape used in node:test §15, and it is the single most useful test-data pattern.

Builders when construction has ordering or invariants:

order = (OrderBuilder().with_sku("ABC").with_qty(2).paid().shipped_to("NL").build())

More ceremony, and worth it when the object has states rather than just fields.

Two rules regardless of approach. Make the relevant data visible and the irrelevant data invisible — that is the whole objective. And never share mutable fixtures across tests; use a factory (a function) rather than a constant (an object), which is why the flakiness example in section 13 uses self._fresh() returning a new list rather than a shared one.

12. Testing the hard things

The unifying principle: make the nondeterminism a parameter. Every row above is the same move.

Concurrency, specifically

The honest answer is that concurrency is the one area where testing gives weak guarantees, and saying so is better than pretending otherwise. What actually helps:

• Test the pieces deterministically. A lock-free queue’s invariants can be tested single-threaded.
• Assert invariants, not outcomes. After N concurrent increments the counter must equal N; you do not care about the order.
• Force the interleaving. Inject a barrier, a semaphore, or a hook that lets the test control when each thread proceeds — turning a race into a scripted scenario.
• Run it many times, in CI, with --test-randomize-style shuffling. Weak, but it catches the frequent races.
• Reach for the real tools for the hard cases: ThreadSanitizer, loom in Rust, Java’s jcstress. Python’s GIL hides some races and creates others (x += 1 is three bytecodes — see Python core §6).

13. Flaky tests

A test that fails 1% of the time is worse than no test: it trains the team to re-run CI instead of reading failures, and once that habit exists a real failure gets re-run too.

The five causes, each with its deterministic fix — all verified:

# 1. REAL TIME — the top cause.
#    FIX: inject the clock.
clock = FrozenClock(0); s = Session(clock); clock.advance(31)
self.assertTrue(s.expired())                     # instant and deterministic

# 2. UNSEEDED RANDOMNESS.
#    FIX: seed it, so a failure reproduces.
rng = random.Random(42)

# 3. ITERATION ORDER of sets (and pre-3.7 dicts, and filesystem listings).
#    FIX: sort, or use assertCountEqual.
self.assertEqual(sorted({"b", "a", "c"}), ["a", "b", "c"])
self.assertCountEqual({"b", "a", "c"}, ["c", "a", "b"])

# 4. SHARED MUTABLE STATE between tests -> order dependence.
#    FIX: a factory, not a shared object.
def _fresh(self): return []

# 5. FLOAT COMPARISON.
#    FIX: assertAlmostEqual with places= or delta=.
self.assertNotEqual(0.1 + 0.2, 0.3)
self.assertAlmostEqual(0.1 + 0.2, 0.3)

Beyond those five: hard-coded ports (fix: listen(0) / port 0 — see both tool chapters), unawaited promises and un-awaited coroutines (fix: no-floating-promises, and note that await_count == 0 with call_count == 1 is how AsyncMock catches it), test pollution through module state and caches (fix: scoped mocks — t.mock and addCleanup), reliance on external services, and timezone or locale assumptions (fix: pin TZ and the locale in CI).

The policy question

Quarantine, do not retry. An automatic retry makes the signal disappear while the underlying race stays in production code, where nobody retries. The workflow that holds:

1. Flaky test found -> open a bug, tag the test, exclude it from the blocking suite.
2. Fix it within a bounded window, or delete it. A quarantine with no expiry is a graveyard.
3. Track the flake rate as a metric. If it is rising, the suite is losing value and that is worth a conversation.

The one legitimate retry is at the end-to-end level against genuinely unreliable infrastructure, and even there it should be logged and counted, not silent.

14. Contract testing

Two services agree on a message format. Both have green test suites. Production breaks, because each one tested against its own belief about the other.

Consumer-driven contract testing fixes it: the consumer writes down what it needs, and the provider’s CI verifies it can deliver that.

Consumer test              -> produces a contract (a pact file)
                              { request: GET /orders/1, response: { id: number, total: number } }
Provider verification test -> replays the contract against the real provider

The moving parts: the consumer’s test runs against a local stub configured from the contract (so it is fast and offline), the contract is published to a broker, and the provider’s pipeline replays every consumer’s contract against the real service. A provider change that breaks any consumer fails the provider’s build — before deployment.

What it buys over integration tests: you never need both services running together, and each side keeps a fast independent pipeline. What it costs: a broker to operate, and discipline about who owns the contract. Pact is the reference implementation.

The lightweight version, when the full machinery is too much: share a schema and test both sides against it. An OpenAPI document, a JSON Schema, or protobuf definitions, with the consumer validating responses and the provider validating its own output against the same file. Less rigorous, a fraction of the setup, and it catches the most common class of break — a renamed or retyped field.

The same idea appears one level down as the fake-versus-real contract suite in section 3. Same principle: when two implementations must agree, write the agreement once and run it against both.

15. Property-based testing

Instead of examples, state a property and let the tool search for a counterexample.

def test_fizzbuzz_invariants(self):
    rng = random.Random(20260820)          # SEEDED: a failure reproduces
    for _ in range(500):
        n = rng.randint(1, 10_000)
        with self.subTest(n=n):
            out = fizzbuzz(n)
            if n % 15 == 0:  self.assertEqual(out, "FizzBuzz")
            elif n % 3 == 0: self.assertEqual(out, "Fizz")
            elif n % 5 == 0: self.assertEqual(out, "Buzz")
            else:            self.assertEqual(out, str(n))

The properties worth looking for, because they are where the bugs are:

The seeded-loop version above gets you most of the value with no dependency. Real tools — hypothesis (Python) and fast-check (JS/TS) — add three things worth paying for: shrinking (it reports the minimal failing input, which is the difference between a 400-element counterexample and a 2-element one), smart generators for realistic data, and a failure database that replays past counterexamples first.

Where property testing is the right tool: parsers and serializers, data structures with invariants, anything with an algebraic law, and any function where you can write a slow-but-obviously-correct reference implementation. Where it is not: business rules with no invariant beyond “the spec says so”, and anything whose expected output is a judgement call.

16. Adding tests to code that has none

Michael Feathers’s definition is the useful one: legacy code is code without tests, regardless of age. The bind is that to test it you must change it, and to change it safely you want tests.

The way through:

1. Write a characterization test first. Not a correct test — a test that records what the code currently does, bugs included. Run the function, print the output, paste it into an assertion. Now you have a safety net for the refactor, and any behaviour change shows up as a diff.
2. Find a seam. A place where behaviour can be changed without editing that code: parameterize, wrap, or subclass-and-override. Section 6 has the four in order of preference.
3. Sprout, do not rewrite. New logic goes in a new, tested method or class that the untested code calls. The scary function stays scary but stops growing.
4. Then refactor behind the tests, in small steps, running the suite each time.

What not to do: a big-bang rewrite with tests written after (you will encode the new bugs), or a demand for 80% coverage on a legacy module before any feature work (nobody will do it, and coverage of code nobody is changing has almost no value).

The pragmatic coverage policy for legacy code: require tests on changed lines, not on the file. New and modified code is tested; untouched code is left alone until it needs touching. That is a ratchet that actually turns, and it is the policy most CI coverage tools support directly.

17. TDD, honestly

Red, green, refactor: write a failing test, make it pass simply, then improve the design with the test as a net.

What it genuinely gives you. A specification you cannot misremember; proof the test can fail (which kills the whole class of tests that pass no matter what — see the WeakSuite above); design pressure toward small, injectable units; and a debugging loop measured in seconds. For anything with a clear input-output contract — a parser, a calculation, a state machine, a bug fix — it is faster than debugging, not slower.

Where it works badly. Exploratory work where you do not yet know the interface; UI layout; performance work (the test is a benchmark, and benchmarks are not pass/fail); and integration-heavy code where the “unit” is a wire protocol. In those cases the honest sequence is spike first, then delete the spike and rebuild with tests — or keep the spike and add characterization tests.

The strongest version of the argument, and the one I would give in an interview: the valuable part is not the ceremony, it is writing the test before you are attached to the implementation. A test written after the code tends to assert what the code does; a test written first asserts what the code should do. You can get that benefit without strict red-green-refactor by writing the test signature and assertions first and filling in the implementation afterwards.

And the bug-fix case has no counterargument: reproduce the bug as a failing test before fixing it. Otherwise you do not know you fixed it, and you have no protection against it returning.

18. Tests in CI

The two policies worth arguing for:

Fail fast on the fast suite, and never on flakes. If unit tests fail, stop the pipeline — nothing downstream matters. If an e2e test is flaky, it should not be able to block a merge; fix it or quarantine it.

Coverage as a ratchet, not a target. “Coverage on changed lines must be >= 80% and total coverage must not decrease” is enforceable and produces real behaviour change. “We must reach 90%” produces assert True (score: 0/5 in section 9).

19. Interview questions

Q: How do you decide what to test?

A: By where the complexity and the risk are. Pure logic with branches gets exhaustive unit tests; boundaries (SQL, serialization, HTTP status codes, constraints) get integration tests because a mock there only encodes my belief about the boundary; user journeys get a handful of end-to-end tests; getters and one-line delegations get nothing. And every bug gets a regression test at the lowest level that reproduces it.

Q: Difference between a mock, a stub, a spy and a fake?

A: A stub gives canned answers and verifies nothing. A spy records so you can assert afterwards. A mock has expectations declared up front and verifies them. A fake is a real working simpler implementation. The useful distinction underneath is state verification versus interaction verification — prefer state, use interaction when the interaction is the observable behaviour.

Q: When is mocking the wrong choice?

A: When you own the thing and it has state — build a fake, because a stub that forgets makes duplicate-detection and other stateful rules untestable while looking tested. And when you are asserting call sequences: that couples the test to the implementation, so a refactor breaks it with no bug introduced.

Q: How do you know your fake behaves like the real thing?

A: A contract test suite both must pass. One set of behavioural tests, one TestCase per implementation, inherited from a mixin that does not itself subclass TestCase.

Q: Is 100% coverage a good target?

A: No. I ran two suites with identical 100% line coverage against five mutants: one killed 0, the other killed 4. Coverage measures execution, not verification. Use it as a floor and a ratchet on changed lines; use mutation testing if you actually want to know whether the assertions bite.

Q: What is mutation testing?

A: Introduce small changes to the code and check whether the suite fails. Surviving mutants mark covered-but-unverified code. Note that some mutants are equivalent — x < lo versus x <= lo is unkillable when the two return the same value — so scores are never 100%, and chasing the last few percent means testing distinctions that do not exist.

Q: Pyramid or trophy?

A: It depends on where the complexity is. A library with real algorithms is pyramid-shaped because the logic is in the units. A service that moves JSON between HTTP and SQL is trophy-shaped because there is barely any logic to unit test and all the risk is in the wiring. The universal answer is the weak one; naming the trade-off is the strong one. What is always wrong is the ice cream cone.

Q: How do you test code that depends on the current time?

A: Inject the clock — a now() parameter or a Clock object — and advance it in the test. Freezing time globally works and has a much larger blast radius. And injecting is strictly stronger for things like backoff, because you can assert the durations ([100, 200]) rather than just that it waited.

Q: How do you test retry with exponential backoff?

A: Inject the sleep function as a spy and assert on the sequence of durations it received, plus the number of attempts. Runs in microseconds and asserts the actual requirement.

Q: How do you avoid port conflicts?

A: Bind to port 0 and read back the assigned port. Never hard-code. That single change removes most “passes alone, fails in parallel” failures.

Q: What do you do with a flaky test?

A: Treat it as a bug in the test, quarantine it (do not auto-retry — retrying hides the signal while the race stays in production), and fix or delete it within a bounded window. The usual causes are real time, unseeded randomness, iteration order, shared mutable state and float comparison — each has a deterministic fix.

Q: How do you make untestable legacy code testable?

A: Characterization test first to pin current behaviour, then find a seam — parameterize the dependency, wrap it behind an interface you own, or sprout the new logic into a testable method — then refactor behind the net. Never a big-bang rewrite with tests written afterwards.

Q: What is a characterization test?

A: A test that records what the code currently does, bugs included, so a refactor’s behaviour changes show up as diffs. It is not asserting correctness; it is asserting sameness.

Q: Do you practise TDD?

A: For anything with a clear input-output contract, and always for bug fixes — reproduce as a failing test before fixing, or you do not know you fixed it. Less strictly for exploratory work, where I spike first and then rebuild with tests. The valuable part is writing the test before I am attached to an implementation; the ceremony is optional.

Q: How do you choose test cases beyond the happy path?

A: Equivalence partitioning, then boundary values (off-by-one is the most common bug class, and the mutation table shows >= 10 versus > 10 is killed only by testing 9 and 10), then the error matrix — invalid input, missing resource, dependency failure, timeout, partial failure, permission denied — then zero/one/many for collections.

Q: What is contract testing and when do you need it?

A: The consumer records what it needs from a provider; the provider’s CI verifies it can deliver that. You need it when independently deployed services must agree and you do not want to run them together in a test. The cheap version is a shared schema (OpenAPI, JSON Schema, protobuf) validated on both sides.

Q: What is property-based testing good for?

A: Round trips (decode(encode(x)) == x), invariants, idempotence, and oracle comparisons against a slow reference implementation. Parsers, serializers and data structures benefit most. The stdlib version is a seeded loop plus an invariant; the real tools add shrinking, which is the feature that actually matters because it turns a 400-element counterexample into a 2-element one.

Q: How would you structure tests for a new service?

A: Functional core, imperative shell: push decisions into pure functions and table-test them exhaustively with no doubles; keep the I/O in a thin shell with one test per path using constructor-injected doubles; add integration tests at the boundaries — real SQLite for constraints and transactions, a real local HTTP server on port 0 for status codes and serialization; one or two end-to-end tests for the critical journey. Factory functions for test data, scoped mocks so nothing leaks between tests.

Q: How do you test concurrent code?

A: Honestly, weakly — and I would say so. Test the pieces deterministically, assert invariants rather than outcomes, force specific interleavings with injected barriers or hooks where the race matters, and run repeatedly with randomized ordering. For anything genuinely concurrency-critical, the answer is a model checker or a race detector, not more unit tests.

Q: Should tests ever assert on log output?

A: Yes, when the log line is part of the contract — an audit trail, or an error path with no return value. Both frameworks support it (assertLogs in Python, a spy logger in Node). What you should not do is assert on log formatting that nobody depends on.

Q: Your suite takes 40 minutes. What do you do?

A: Measure first — both runners can report the slowest tests. The usual findings are real sleeps (inject the clock), a database or container per test rather than per suite (share the schema, roll back per test), unnecessary end-to-end coverage of logic that could be unit tested, and no parallelism. Then split into a fast suite on every push and a slow suite on merge.

Next: Testing cheat sheet and drills, or back to node:test / unittest.

Verify it yourself

test-strategy/test_doubles.py

"""The five test doubles (Meszaros), each doing the same job, so the trade-offs are visible."""
import unittest
from unittest.mock import Mock, create_autospec


# ---------- the system under test ----------
class Notifier:
    def send(self, to: str, body: str) -> str: ...          # returns a message id


class SignupService:
    def __init__(self, store, notifier, now):
        self.store, self.notifier, self.now = store, notifier, now

    def register(self, email: str) -> dict:
        if "@" not in email:
            raise ValueError("invalid email")
        if self.store.get(email) is not None:
            raise ValueError("already registered")
        user = {"email": email, "created_at": self.now()}
        self.store.put(email, user)
        self.notifier.send(email, "welcome")
        return user


# ---------- 1. DUMMY: passed to satisfy a signature, never used ----------
class DummyNotifier:
    def send(self, to, body):
        raise AssertionError("a dummy must never be called")


# ---------- 2. STUB: canned answers, no verification ----------
class StubStore:
    def __init__(self, existing=None):
        self._existing = existing or {}
    def get(self, k): return self._existing.get(k)
    def put(self, k, v): pass                # accepts and forgets


# ---------- 3. SPY: records what happened, you assert afterwards ----------
class SpyNotifier:
    def __init__(self): self.sent = []
    def send(self, to, body):
        self.sent.append((to, body))
        return "msg_1"


# ---------- 4. MOCK: expectations declared UP FRONT, verified on demand ----------
class MockNotifier:
    def __init__(self, expected):
        self._expected, self._actual = expected, []
    def send(self, to, body):
        self._actual.append((to, body)); return "msg_1"
    def verify(self):
        assert self._actual == self._expected, f"expected {self._expected}, got {self._actual}"


# ---------- 5. FAKE: a real, working, simpler implementation ----------
class FakeStore:
    """A working in-memory store. Enforces the same invariants as the real one."""
    def __init__(self): self._d = {}
    def get(self, k): return self._d.get(k)
    def put(self, k, v):
        if k in self._d: raise KeyError(f"duplicate key {k!r}")   # the REAL constraint
        self._d[k] = dict(v)


class TestTheFiveDoubles(unittest.TestCase):
    def test_dummy_when_the_collaborator_is_irrelevant(self):
        svc = SignupService(StubStore(), DummyNotifier(), now=lambda: 0)
        with self.assertRaises(ValueError):
            svc.register("not-an-email")          # fails before the notifier is reached

    def test_stub_when_you_only_need_an_answer(self):
        svc = SignupService(StubStore({"a@b.c": {"email": "a@b.c"}}), DummyNotifier(), now=lambda: 0)
        with self.assertRaisesRegex(ValueError, "already registered"):
            svc.register("a@b.c")

    def test_spy_when_you_assert_afterwards(self):
        spy = SpyNotifier()
        SignupService(StubStore(), spy, now=lambda: 7).register("a@b.c")
        self.assertEqual(spy.sent, [("a@b.c", "welcome")])

    def test_mock_when_expectations_come_first(self):
        mock = MockNotifier(expected=[("a@b.c", "welcome")])
        SignupService(StubStore(), mock, now=lambda: 7).register("a@b.c")
        mock.verify()

    def test_fake_when_you_want_real_behaviour(self):
        fake = FakeStore()
        svc = SignupService(fake, SpyNotifier(), now=lambda: 7)
        svc.register("a@b.c")
        self.assertEqual(fake.get("a@b.c"), {"email": "a@b.c", "created_at": 7})
        # The fake enforces the real constraint, so this test needs no extra setup:
        with self.assertRaisesRegex(ValueError, "already registered"):
            svc.register("a@b.c")

    def test_the_fake_is_the_only_double_that_catches_this(self):
        """A stub `put` that forgets means the duplicate check silently passes."""
        stub_svc = SignupService(StubStore(), SpyNotifier(), now=lambda: 0)
        stub_svc.register("a@b.c")
        stub_svc.register("a@b.c")               # no error: the stub never stored anything

        fake_svc = SignupService(FakeStore(), SpyNotifier(), now=lambda: 0)
        fake_svc.register("a@b.c")
        with self.assertRaises(ValueError):
            fake_svc.register("a@b.c")           # the fake remembers


class TestOverMockingIsBrittle(unittest.TestCase):
    """A test coupled to HOW beats a test coupled to WHAT — until you refactor."""

    def test_brittle_interaction_test(self):
        store = create_autospec(FakeStore, instance=True)
        store.get.return_value = None
        svc = SignupService(store, SpyNotifier(), now=lambda: 0)
        svc.register("a@b.c")
        # This asserts the IMPLEMENTATION: one get, then one put, in that order.
        # Add a cache lookup, or reorder for efficiency, and it breaks with no bug introduced.
        store.get.assert_called_once_with("a@b.c")
        store.put.assert_called_once()
        self.assertEqual([c[0] for c in store.method_calls], ["get", "put"])

    def test_robust_state_test(self):
        fake = FakeStore()
        svc = SignupService(fake, SpyNotifier(), now=lambda: 0)
        svc.register("a@b.c")
        # This asserts the OUTCOME. Any refactor that keeps the outcome keeps the test green.
        self.assertEqual(fake.get("a@b.c")["email"], "a@b.c")


class TestContractBetweenFakeAndReal(unittest.TestCase):
    """The obligation a fake creates: prove it behaves like the real thing."""

    class RealStore:
        """Pretend this is backed by SQL."""
        def __init__(self): self._rows = []
        def get(self, k): return next((dict(r) for r in self._rows if r["k"] == k), None)
        def put(self, k, v):
            if any(r["k"] == k for r in self._rows): raise KeyError(f"duplicate key {k!r}")
            self._rows.append({"k": k, **v})

    def _contract(self, store):
        self.assertIsNone(store.get("missing"))
        store.put("x", {"v": 1})
        self.assertEqual(store.get("x")["v"], 1)
        with self.assertRaises(KeyError):
            store.put("x", {"v": 2})

    def test_fake_satisfies_the_contract(self):
        self._contract(FakeStore())

    def test_real_satisfies_the_same_contract(self):
        self._contract(self.RealStore())


if __name__ == "__main__":
    unittest.main()

test-strategy/test_seams.py

"""Making untestable code testable, without a rewrite: the four seam techniques."""
import unittest
from unittest.mock import patch, Mock

# ============ BEFORE: nothing here is testable ============
import time, random, os


def process_order_untestable(sku, qty):
    """Hard-coded time, hard-coded randomness, hard-coded env, hard-coded I/O.
    Every one of those is a reason this cannot be tested without patching."""
    order_id = f"ord_{random.randint(1000, 9999)}"
    if os.environ.get("STAGE") == "prod":
        pass                                     # imagine a real charge here
    return {"id": order_id, "sku": sku, "qty": qty, "at": time.time()}


# ============ AFTER 1: parameterize the seams ============
def process_order(sku, qty, *, now=time.time, gen_id=None, stage=None):
    gen_id = gen_id or (lambda: f"ord_{random.randint(1000, 9999)}")
    stage = stage if stage is not None else os.environ.get("STAGE", "dev")
    return {"id": gen_id(), "sku": sku, "qty": qty, "at": now(), "stage": stage}


# ============ AFTER 2: sprout a method, and test the sprout ============
class LegacyReport:
    def render(self, rows):                      # big, untested, scary
        lines = [self._format_row(r) for r in rows]      # <- the sprouted method
        return "\n".join(["REPORT"] + lines + [f"TOTAL {self._total(rows)}"])

    def _format_row(self, r):                    # new logic, fully testable in isolation
        return f"{r['sku']:<6}{r['qty']:>4}{r['price'] * r['qty']:>8}"

    def _total(self, rows):
        return sum(r["price"] * r["qty"] for r in rows)


# ============ AFTER 3: wrap the dependency behind an interface you own ============
class Clock:
    def now(self): return time.time()

class FrozenClock:
    def __init__(self, t): self._t = t
    def now(self): return self._t
    def advance(self, d): self._t += d


class Session:
    TTL = 30
    def __init__(self, clock=None):
        self.clock = clock or Clock()
        self.started = self.clock.now()
    def expired(self):
        return self.clock.now() - self.started > self.TTL


# ============ AFTER 4: extract the pure core, keep the shell thin ============
def decide(stock: int, requested: int, price: int) -> dict:
    """Pure: no clock, no I/O, no randomness. Trivially testable, exhaustively."""
    if requested <= 0:            return {"ok": False, "reason": "bad_qty"}
    if stock < requested:         return {"ok": False, "reason": "out_of_stock"}
    return {"ok": True, "total": price * requested}


def place_order(repo, sku, qty):                 # the imperative shell
    item = repo.find(sku)
    decision = decide(item["stock"], qty, item["price"])
    if not decision["ok"]:
        raise ValueError(decision["reason"])
    repo.decrement(sku, qty)
    return decision["total"]


class TestSeams(unittest.TestCase):
    def test_untestable_version_needs_patching_everything(self):
        with patch("test_seams.random.randint", return_value=1234), \
             patch("test_seams.time.time", return_value=99), \
             patch.dict(os.environ, {"STAGE": "test"}):
            out = process_order_untestable("ABC", 2)
        self.assertEqual(out, {"id": "ord_1234", "sku": "ABC", "qty": 2, "at": 99})
        # It works — and it is coupled to module paths, import style and the stdlib.

    def test_parameterized_version_needs_no_patching(self):
        out = process_order("ABC", 2, now=lambda: 99, gen_id=lambda: "ord_1234", stage="test")
        self.assertEqual(out, {"id": "ord_1234", "sku": "ABC", "qty": 2, "at": 99, "stage": "test"})

    def test_sprouted_method_in_isolation(self):
        r = LegacyReport()
        self.assertEqual(r._format_row({"sku": "ABC", "qty": 2, "price": 250}),
                         "ABC      2     500")
        self.assertEqual(r._total([{"qty": 2, "price": 250}, {"qty": 1, "price": 100}]), 600)

    def test_wrapped_clock_makes_expiry_testable(self):
        clock = FrozenClock(1000)
        s = Session(clock)
        self.assertFalse(s.expired())
        clock.advance(31)
        self.assertTrue(s.expired())

    def test_pure_core_is_exhaustively_testable(self):
        cases = [
            (10, 0, 5,  {"ok": False, "reason": "bad_qty"}),
            (10, -1, 5, {"ok": False, "reason": "bad_qty"}),
            (1, 5, 5,   {"ok": False, "reason": "out_of_stock"}),
            (5, 5, 5,   {"ok": True, "total": 25}),
            (10, 3, 7,  {"ok": True, "total": 21}),
        ]
        for stock, req, price, expected in cases:
            with self.subTest(stock=stock, req=req):
                self.assertEqual(decide(stock, req, price), expected)

    def test_shell_needs_only_one_thin_test_per_path(self):
        repo = Mock(**{"find.return_value": {"stock": 5, "price": 10}})
        self.assertEqual(place_order(repo, "ABC", 2), 20)
        repo.decrement.assert_called_once_with("ABC", 2)

        repo2 = Mock(**{"find.return_value": {"stock": 0, "price": 10}})
        with self.assertRaisesRegex(ValueError, "out_of_stock"):
            place_order(repo2, "ABC", 1)
        repo2.decrement.assert_not_called()


class TestFlakinessCauses(unittest.TestCase):
    """Each of the five common flakiness causes, with its deterministic fix."""

    def test_1_real_time_is_the_top_cause(self):
        # FLAKY: assertions about elapsed real time
        # FIXED: inject the clock (see FrozenClock above)
        clock = FrozenClock(0)
        s = Session(clock); clock.advance(31)
        self.assertTrue(s.expired())              # deterministic, and instant

    def test_2_unseeded_randomness(self):
        import random as r
        rng = r.Random(42)                        # seed it: reproducible failures
        first = [rng.randint(0, 100) for _ in range(5)]
        rng2 = r.Random(42)
        self.assertEqual(first, [rng2.randint(0, 100) for _ in range(5)])

    def test_3_iteration_order_of_a_set(self):
        # FLAKY: assertEqual(list(some_set), [...]) — set order is not guaranteed
        s = {"b", "a", "c"}
        self.assertEqual(sorted(s), ["a", "b", "c"])       # sort, or use assertCountEqual
        self.assertCountEqual(s, ["c", "a", "b"])

    def test_4_shared_mutable_state_between_tests(self):
        # FLAKY: a module-level list/dict/cache mutated by tests -> order dependence
        # FIXED: build fresh state per test (setUp), or reset it in addCleanup
        self.assertEqual(self._fresh(), [])
        self._fresh().append(1)
        self.assertEqual(self._fresh(), [])       # a factory, not a shared object

    def _fresh(self): return []

    def test_5_float_comparison(self):
        self.assertNotEqual(0.1 + 0.2, 0.3)                 # the trap
        self.assertAlmostEqual(0.1 + 0.2, 0.3)              # the fix
        self.assertAlmostEqual(1_000_000.1, 1_000_000.2, delta=0.5)


if __name__ == "__main__":
    unittest.main()

test-strategy/mutation.py

"""A hand-rolled mutation test: does the suite actually catch a broken implementation?

The idea: introduce a small change ("mutant") to the code under test. A good suite fails.
A suite that still passes has 100% coverage of that line and zero verification of it.
"""
import unittest, io, contextlib


# ---- the code under test ----
def clamp(x, lo, hi):
    if x < lo: return lo
    if x > hi: return hi
    return x


def discount(total, qty):
    if qty >= 10: return round(total * 0.9)
    if qty >= 5:  return round(total * 0.95)
    return total


# ---- two suites over the same code ----
class WeakSuite(unittest.TestCase):
    """100% line coverage. Verifies almost nothing."""
    def test_clamp_runs(self):
        clamp(5, 0, 10); clamp(-1, 0, 10); clamp(99, 0, 10)
        self.assertTrue(True)
    def test_discount_runs(self):
        discount(100, 1); discount(100, 5); discount(100, 10)
        self.assertTrue(True)


class StrongSuite(unittest.TestCase):
    """Same coverage, real assertions, including the boundaries."""
    def test_clamp(self):
        for x, lo, hi, want in [(5, 0, 10, 5), (-1, 0, 10, 0), (99, 0, 10, 10),
                                (0, 0, 10, 0), (10, 0, 10, 10)]:
            with self.subTest(x=x): self.assertEqual(clamp(x, lo, hi), want)
    def test_discount_boundaries(self):
        for total, qty, want in [(100, 1, 100), (100, 4, 100), (100, 5, 95),
                                 (100, 9, 95), (100, 10, 90), (100, 20, 90)]:
            with self.subTest(qty=qty): self.assertEqual(discount(total, qty), want)


MUTANTS = {
    "clamp: < becomes <=":        lambda: _patch_clamp(lambda x, lo, hi: lo if x <= lo else (hi if x > hi else x)),
    "clamp: swap lo/hi returns":  lambda: _patch_clamp(lambda x, lo, hi: hi if x < lo else (lo if x > hi else x)),
    "discount: 10 becomes 11":    lambda: _patch_discount(lambda t, q: round(t*0.9) if q >= 11 else (round(t*0.95) if q >= 5 else t)),
    "discount: 0.9 becomes 0.8":  lambda: _patch_discount(lambda t, q: round(t*0.8) if q >= 10 else (round(t*0.95) if q >= 5 else t)),
    "discount: >= becomes >":     lambda: _patch_discount(lambda t, q: round(t*0.9) if q > 10 else (round(t*0.95) if q > 5 else t)),
}

_orig_clamp, _orig_discount = clamp, discount
def _patch_clamp(f):
    global clamp; clamp = f
def _patch_discount(f):
    global discount; discount = f
def _restore():
    global clamp, discount; clamp, discount = _orig_clamp, _orig_discount


def run(suite_cls):
    suite = unittest.TestLoader().loadTestsFromTestCase(suite_cls)
    buf = io.StringIO()
    result = unittest.TextTestRunner(stream=buf, verbosity=0).run(suite)
    return result.wasSuccessful()


print(f"{'mutant':<30}{'WeakSuite':>12}{'StrongSuite':>14}")
print("-" * 56)
weak_killed = strong_killed = 0
for name, apply_mutant in MUTANTS.items():
    apply_mutant()
    w, s = run(WeakSuite), run(StrongSuite)
    _restore()
    weak_killed += not w
    strong_killed += not s
    print(f"{name:<30}{'KILLED' if not w else 'survived':>12}{'KILLED' if not s else 'survived':>14}")
print("-" * 56)
n = len(MUTANTS)
print(f"{'mutation score':<30}{f'{weak_killed}/{n}':>12}{f'{strong_killed}/{n}':>14}")
print("\nBoth suites have 100% line coverage of clamp() and discount().")