Software engineering

The big picture and the vocabulary

Making a sandwich is easy. Running a restaurant that serves a thousand sandwiches a day is hard. In the same way, writing a small program is easy, but building software that many people use for many years is hard. Software engineering is the discipline (a set of careful methods) that makes the hard case manageable.

Every software project, big or small, needs four basic activities. Think of baking a cake for a birthday: you decide what cake they want, you plan the recipe, you bake it and taste it, and later you fix or change it for the next party.

A [[stakeholder|anyone who has an interest in the system: users, customers, managers, testers, neighbours who are affected]] is any person who cares about the result. A requirement is a thing the system must do or a limit it must respect. A process is the ordered set of activities used to build software.

Process models: waterfall, incremental, agile

A process model is a recipe for how to arrange the four activities. There is no single best recipe. A good team picks the one that fits the project, like choosing between a bike, a car or a plane depending on the trip.

Waterfall: one step after the other

In the waterfall model the phases run in a sequence, like water falling down steps. Each phase must be finished and signed off before the next begins. It is called a plan-driven process.

Requirements
     |
     v
  Design
     |
     v
Implementation
     |
     v
 Testing
     |
     v
Maintenance

Waterfall is a good fit when the requirements are known and stable, and when mistakes are very costly. For example, the control software of a medical device built together with its hardware, under safety rules, needs careful planning and many documents. Its weakness is that changing your mind late is expensive.

The V model

The V model is a plan-driven process drawn like a V. Each test plan is prepared from a phase on the left side, so that the matching test on the right side can check it. For example, the acceptance test plan comes from the requirements specification: the customer's needs are checked against the final system.

Requirements ----------------> Acceptance test
   Design  ------------------> System test
      Code --------------> Unit test

Incremental development

In incremental development you build the system in small pieces called increments, and give each piece to the customer to try. It is like building a LEGO castle one tower at a time and showing each tower to your friend. The cost of accommodating changes in the customer's requirements is lower, and the customer gets useful software early.

Reuse-oriented engineering

Why bake the bread yourself when you can buy it? In reuse-oriented software engineering the system is built mainly from existing components, such as a commercial off-the-shelf (COTS) package. The stage that has no direct equivalent in other processes is modifying the requirements to match the components that are available. A risk is losing control over how the system evolves, because new versions of the package are decided by the vendor.

The spiral model and RUP

Boehm's spiral model goes around in loops. What makes it special is that it makes the explicit recognition and reduction of risk part of each loop. If a team is unsure whether users will accept a new voice interface, the spiral spends that loop on risk assessment and reduction, for instance by building a small test.

The Rational Unified Process (RUP) has four phases, in order: inception, elaboration, construction and transition. At the end of elaboration the team should understand the problem domain, have an architecture and a plan, and have identified the main risks. Sommerville describes RUP through three perspectives: dynamic (the phases over time), static (the process activities, such as business modelling and requirements workflows) and practice (good practices).

Prototypes and coping with change

A prototype is an early rough version of the system used to try ideas. A quick throwaway prototype should not become the final product, because it was built for speed: it ignores non-functional needs and is poorly structured and hard to maintain. Sommerville separates two ways of reducing the cost of rework: change avoidance (for example, a prototype that shows the customer the system early, before the real work) and change tolerance (for example, incremental delivery, so a change affects only a small piece). Some customers, such as a government agency whose procurement rules require a complete specification before a contract, cannot use incremental delivery easily, since it develops the specification together with the software.

Requirements engineering

Before building a house you must ask the family what they want: how many rooms, how much money, near which school? Requirements engineering is the work of finding out, writing down and checking what a system must do and the limits it must respect.

User and system requirements

User requirements are high-level statements, written for customers, about what the system should do. System requirements describe the same things in more detail, for the developers. Think of a menu (what you can order) and the kitchen recipe (exactly how it is cooked).

Functional and non-functional

Non-functional requirements are often more critical than individual functional ones, because failing one of them can make the whole system unusable. A fast, beautiful plane that is unsafe is useless. Sommerville classifies them as product requirements (how the product behaves: speed, reliability), organisational requirements (company rules, such as a required development process) and external requirements (factors outside the system and its development, such as a law or a regulator's rule).

Make goals testable

A client who says 'the system should be easy to use' gives a vague goal. A good rewrite makes it measurable: 'After four hours of training, clerks shall make on average no more than two errors per hour of use.' A requirement you cannot check is a requirement you cannot be sure you met.

Finding requirements: elicitation

• Interviews with stakeholders.
• [[Ethnography|an analyst spends time in the workplace and watches how people really work]]: great for real practice, but weak at finding requirements for innovative new features that change how people work.
• Scenarios and stories: a description of how a user does a task. A scenario for cash withdrawal at an ATM includes the normal flow and also what can go wrong, such as a wrong PIN or insufficient funds, and how it is handled.

When two stakeholders want opposite things (the head of nursing wants every medication change confirmed twice, the doctors want speed), the engineer does requirements prioritisation and negotiation to settle the conflict.

Writing and checking requirements

A requirements document should describe what the system should do, rather than how it should do it. Use language consistently: shall marks a mandatory requirement, should a desirable one. Natural language is easy to read but can be imprecise: 'remind the patient before each appointment' does not say how (SMS or email) or when, so state the channel and timing precisely, ideally in a structured form.

Validation checks that the requirements are what the customer really wants. Useful checks: a realism check (can it be built with the available technology and budget? a demand for real-time 3D rendering on a tiny device fails it) and test-case generation (if you cannot write a test for a requirement, it is probably vague or ambiguous).

Requirements change. A good change process has three steps in order: problem analysis and change specification, then change analysis and costing, then change implementation. Traceability records, for each requirement, who asked for it and which other requirements and design parts depend on it, so you can assess the impact of a proposed change before deciding.

System modelling with UML

Before building a toy city, you draw a map. A model is a simplified picture of a system that shows what matters and hides the rest. The Unified Modeling Language (UML) is a standard set of diagrams for drawing software systems, so everyone reads the pictures the same way.

Sommerville describes four perspectives for modelling: external (the context and environment), interaction (how things talk to each other), structural (the organisation of the system and its data) and behavioural (what the system does as it runs).

Context model

A context model is drawn in the early stages to define the boundaries of the system: which other systems are in its environment. It decides what is inside the box and what is outside.

Use case diagram

A use case diagram shows the actors (people or other systems) and the interactions they have with the system, but not their order. It answers 'who does what with the system?'

Sequence diagram

A sequence diagram shows the order of messages between objects, with time flowing downwards. In this one the ATM asks the Bank to verify the PIN before asking the customer for the amount.

Customer       ATM          Bank
   | insert card |             |
   |------------>|  verify PIN |
   |             |------------>|
   |             |<------------|
   |  ask amount |   PIN ok    |
   |<------------|             |

Class diagram

A class diagram shows the classes (kinds of things) and the relationships between them. The numbers on a line are the multiplicity. Reading Department 1 --- 1..* Employee means that each department has at least one employee, and each employee belongs to exactly one department.

State and activity diagrams

A state diagram shows the states of a thing and the events that move it from one to another. A vending machine in state Ready, when the stock is zero and someone presses the button, stays in Ready and shows 'sold out': the diagram tells you what happens for each event in each state.

An activity diagram shows the flow of work. A thick bar with one input and several outputs is a fork: the paths run in parallel. A thick bar with several inputs and one output is a join: it waits until all the incoming paths have finished. So 'Ship order' after a join starts only after both 'Charge card' and 'Pack items' are done.

Model-driven engineering

Model-driven architecture (MDA) uses three kinds of model: the computation independent model (the domain), the platform independent model (PIM), which describes the system without reference to a platform, and the platform specific model (PSM). The promise is to generate code for several platforms from the models. In practice it is hard to adopt because abstractions that are good for discussing a design do not always suit implementation, and the tools and the effort of keeping models and code in step can cost more than they save.

Architectural design overview

Before building a house, the architect decides how many floors there are, where the walls and pipes go. In software, architectural design is concerned with identifying the main structural components of a system and the relationships between them. It is the link between requirements and detailed design.

Views: the 4+1 model

One drawing cannot show everything about a house. The 4+1 view model uses several views: the logical view (the key abstractions), the process view (how the system runs, which parts are concurrent), the development view (how the software is broken down into pieces for the developers) and the physical view (hardware and where things are deployed). The '+1' is the use cases, which tie the views together.

Architectural patterns

+------------------+
|  User interface  |
+------------------+
|  Business logic  |
+------------------+
|  Data storage    |
+------------------+

A pipe-and-filter design works well for a nightly batch job that reads payments, matches them and writes reports, because it is built for streams of data. It fits interactive systems badly, since those react to unpredictable events from the user.

Architecture and non-functional requirements

Choices in the architecture decide the qualities of the system. If the most important requirement is maintainability, use fine-grain, self-contained components that can be changed easily, with data producers separate from consumers. If it is performance, use fewer, larger components, to reduce communication. Sommerville says both together are hard because performance favours a few large components, while maintainability favours many small ones.

Design, implementation and clean code

Writing code is like writing a letter that other people must read for years. Even you, a month later, will be a 'stranger' to your own code. This section is about designing with proven ideas and writing code that is easy to read.

Design patterns

A design pattern is a description of a recurring problem and the essence of its solution, abstract enough to be reused in different settings. The Observer pattern is an example: when a spreadsheet cell changes, the table, the bar chart and the pie chart all update, because they are observers that are told about the change by the subject.

Reuse and open source

Reuse happens at several levels. When a team does not copy any code but applies the knowledge of a successful design, that is the abstraction level. Open source licences differ. The GNU General Public Licence (GPL) is reciprocal (copyleft): if you distribute software that includes GPL code, you must release your software under the GPL too. Licences such as BSD and MIT are permissive: a company can include a BSD library in a closed-source product and modify it, keeping its own code private, if it keeps the notices.

Sommerville lists four fundamental configuration management activities during implementation: version management, system building, change management and problem tracking (also called issue tracking), among them.

Clean code: names

In Clean Code, a name should reveal its intent. A declaration like int d; // elapsed time in days should be renamed to elapsedTimeInDays, and the comment dropped. The name now says what the comment said.

// Bad: what is d? why 86400000?
let d = (end - start) / 86400000;

// Good: the name tells the story
const msPerDay = 86_400_000;
const elapsedTimeInDays = (end - start) / msPerDay;

Clean code: functions

• A function should do one thing, at one level of abstraction.
• A boolean parameter is a smell: it shows the function does two things. Split exportReport(true) into exportReportAsPdf and exportReportAsCsv.
• Do not return null for a list; return an empty list, so callers can loop without a special check (one forgotten null check can crash the program).
• Follow the Law of Demeter: a method should talk only to its immediate collaborators, not reach deep through objects (a.getB().getC().doIt()).

Clean code: comments, formatting, classes, tests

Martin's view of comments: a comment often makes up for code that fails to express itself, so prefer clear code, and keep comments for what the code cannot say (the reason for a decision, a warning). Formatting follows the newspaper metaphor: put the high-level concepts at the top and the details lower down, so the reader can stop reading when they have enough.

The Single Responsibility Principle says a class should have one reason to change. A class that changes when the payroll team changes pay rules, when the database team changes the schema and when finance changes reports has several reasons to change; split it. Clean tests follow the F.I.R.S.T. idea: fast, independent, repeatable, self-validating and timely. A test that fails when run alone but passes in order is not independent.

Finally, boundaries: if code calls a third-party HTTP library from 40 files, a new major version forces 40 edits. Wrap the library behind your own small interface, and write learning tests that check how the library behaves, so an upgrade changes one place.

Testing, evolution and maintenance

A bridge is tested before cars drive on it. Software too. Testing means running a program to look for defects (bugs) and to show that it does what it should. The famous remark of Dijkstra, which Sommerville repeats, says it all: testing can show the presence of errors, never their absence. If 2,000 tests pass, you only know that those 2,000 situations work. Other inputs and paths may still hide a bug, so we also use reviews and other checks, and we pick test cases with care instead of just piling up many.

Stages of testing

The first three stages are called development testing: they are done by the team that builds the system, and they hunt for bugs (defect testing). Release testing is different. A team that did not build the system tests it as a black box (without looking at the code), using the requirements, to show that it is good enough to be released. That is validation testing. Only the last stage, acceptance testing, is done by the customer.

Choosing test cases

You cannot test every possible input. Equivalence partitioning divides the inputs into groups (partitions) that the program should treat in the same way, and you test one value from each group. Bugs gather at the edges of the groups, where a < is easily mixed up with a <=. So also test the boundary values: the value on each edge and the values just outside it.

  rule: a ticket is free for ages 0 to 12 only

  partition:   below 0  |    0 .. 12    |  above 12
  boundaries:         -1 | 0         12 | 13
  tests:             -1      0   6   12    13

function isAdultWorker(age: number): boolean {
	return age >= 18 && age <= 65;
}
// boundary tests: 17 -> false, 18 -> true, 65 -> true, 66 -> false

How much of the code did the tests visit?

Coverage tells how much of the code the tests run. There are several criteria, and they are not equal. Statement coverage asks that every line runs at least once. Branch coverage asks that every decision (an if) is seen going both ways, true and false. Path coverage asks for every possible route through the function, and only it runs every combination of decisions.

function price(base: number, student: boolean, voucher: boolean): number {
	let total = base;
	if (student) total -= 5;
	if (voucher) total -= 3;
	return total;
}

With n independent decisions in a row there are 2^n paths: 10 decisions give 1,024 paths and 20 give 1,048,576. That is why full path coverage is rarely practical. And 100% statement coverage can still miss a bug: the case (true, false) above never ran in the one statement test.

Test-driven development and regression

In test-driven development (TDD) the test comes before the code. You work in very short rounds, one small piece of functionality at a time:

1. Write an automated test for the next small piece.
2. Run it and watch it fail. This proves the test can really detect something, and that the code does not exist yet.
3. Write just enough code to make it pass.
4. Run all the tests again, then clean the code (refactor) while they stay green.

A regression is a bug in something that used to work. Regression testing runs the old tests again after every change, because a change can break code that looks unrelated. Done by hand it is slow, so people skip it. With automated tests the whole suite can run after each change, which is why TDD and continuous integration love automation.

Evolution and maintenance

Software does not end when it is delivered: it lives on and changes. This is software evolution, also called maintenance. Sommerville names three kinds. Notice that fixing bugs is the smallest share. Most maintenance is evolution, not bug fixing.

Lehman and Belady studied how big systems change over the years and wrote down laws. The two best known explain why we spend time on cleaning:

• Continuing change: a system used in the real world must keep changing, or it becomes less and less useful.
• Increasing complexity: as it changes, its structure gets worse and more tangled, unless someone works to keep it simple.
• Continuing growth: it has to gain features to keep users happy.
• Conservation of familiarity: the amount of change in each release stays about the same over time, because people must keep understanding it.
• Organisational stability: the speed of development stays about constant, however many people you add.

Why changing after delivery costs more

The same change costs more after delivery than during development. Sommerville gives reasons:

• Team stability: the developers leave, and the maintainers must rebuild their understanding. The code does not record why it was designed that way.
• Contractual responsibility: when one company builds and another maintains, the builder is paid nothing for making the system easy to change. Skipping that work saves the builder money, and the next company pays the bill.
• Staff skills: maintenance is often given to less experienced people.
• Age and structure: old programs, changed many times, are harder to read.

Old systems: keep, fix or replace?

A legacy system is an old system that is still in use and hard to change. To decide what to do, Sommerville looks at two things: its business value (how much people need it) and its quality (how costly it is to change and run).

Refactoring and re-engineering

Both improve the structure of the code without changing what it does, but at different moments.

Project management and estimation

Planning a school trip means choosing a date, a bus, a budget and who goes. Project management does that for software: it makes sure the software is delivered on time, within budget, and meets the customer's needs.

Risk management

A risk is something that may go wrong. If the only developer who knows the payment integration announces that she is leaving, that is a project risk, because it threatens the schedule and resources. The risk management process, in order, is: identification, analysis, planning, monitoring. In analysis you estimate two things for each risk: its probability and the seriousness of its effects. A contingency plan is what you do if the risk happens (for instance, if the database may not cope with peak load, you prepare a plan to add capacity or caching, without changing the design now).

People and teams

Developers are motivated differently: task-oriented people enjoy the work itself, self-oriented people are driven by personal success, recognition and money, and interaction-oriented people like the company of colleagues. A cohesive team can fall into groupthink: it keeps approving its own decisions without serious criticism. Communication has a cost: with n people, the number of two-person channels is n(n-1)/2.

Planning and scheduling

A deliverable is a work product handed to the customer. A milestone is a checkpoint where progress can be assessed (it may not be delivered to anyone). Activities have durations and dependencies. The critical path is the longest chain of dependent activities; it decides the shortest time the project can finish.

In this plan, A takes 3 days. B and C start after A and run in parallel. D starts only when both B and C finish, i.e. after day 3 + 7 = 10. D ends on day 12. The critical path is A, C, D = 3 + 7 + 2 = 12 days. B has slack, it can be late by 2 days with no effect.

Estimating

In agile teams, velocity is the story points completed per sprint. If a team did 30, 34 and 26 points, the average is 30, so 150 remaining points take about 5 sprints. An algorithmic cost model has the form Effort = A × Size^B × M. If B is above 1 (for example 1.1), effort grows faster than size: doubling the size multiplies the effort by 2^1.1, about 2.14, not 2. M is the product of the cost-driver multipliers (as in COCOMO II). Drivers rated away from nominal are multiplied together. For example, a nominal 100 person-months estimate with multipliers 1.1, 1.05 and 0.98 gives about 113 person-months.

When a customer wants a fixed budget and a fixed delivery date with an agile process, fix time and cost, deliver the highest-priority features first, and negotiate how much of the scope is delivered.

Brooks: The Mythical Man-Month and No Silver Bullet

Fred Brooks led the building of IBM's OS/360 and wrote about it in The Mythical Man-Month (1975). His lessons are still true today.

The mythical man-month and Brooks's law

A man-month is the work of one person in one month. The myth is to think people and months can be traded: that 12 person-months can be 12 people for 1 month. This works only when the task splits among workers who need no communication with each other, like picking apples. When tasks depend on one another, more people means more talking. Brooks's law: adding people to a late software project makes it later. New people must be trained by those already doing the work, the work must be split again, and every extra person adds communication paths.

The law is a deliberate simplification. Adding people is least likely to hurt when they join early, the task splits into independent parts and they already know the domain.

Brooks's schedule rule

Notice that coding is only one sixth; most time goes to planning and testing.

The surgical team

Brooks described, from a proposal by Harlan Mills, a surgical team: one chief programmer designs and codes, and the others support that person in specialised roles. The reason is that programmer productivity varies about tenfold, and few minds give a more coherent design.

Conceptual integrity and the architect

Conceptual integrity is the most important consideration in system design: the system should reflect one coherent set of design ideas, even if that means leaving out some good features. Brooks separates the architect from the implementers. The architect decides the complete specification of what the user sees: the commands, their parameters and their behaviour. How it is built inside belongs to the implementers.

The second-system effect and 'plan to throw one away'

The second-system effect: a designer's second system tends to be over-designed, loaded with the ideas held back from the first. In 1975 Brooks advised: 'plan to throw one away; you will anyway.' Later he judged this advice too simple, because it assumed the waterfall model; he favoured incremental growth of a system.

No Silver Bullet (1986)

A silver bullet kills monsters; people wished for one tool to kill the software crisis. Brooks's central claim: no single technique or management method will, within a decade, give even a tenfold improvement in productivity, reliability or simplicity. He separates essential difficulties, inherent in the conceptual structure of software (complexity, conformity, changeability, invisibility), from accidental difficulties, which come from our tools (such as managing registers and memory addresses by hand in assembly).

• Complexity: software has too many states and parts.
• Conformity: it must follow many arbitrary rules of the outside world, such as tax laws, union agreements and bank formats.
• Changeability: it is constantly asked to change.
• Invisibility: it has no natural shape to draw.

Math example: suppose 60% of the effort is accidental and 40% essential. Even if you remove all the accidental part, 40% remains, so the time falls to 0.4 and the speed-up is 1 / 0.4 = 2.5 times, not 10.

Quality and configuration management

A chocolate factory checks every batch and keeps a record of every recipe version. Software teams do the same with quality management (checking that the software is good) and configuration management (keeping track of every version and change).

Standards, reviews and inspections

Product standards apply to the product (for example, a naming convention). Process standards define how work is done, such as a procedure that every change request must follow before it is approved.

A code inspection is a group reading of code to find defects. Its real advantage over testing is that it can find many defects in one session, because one defect does not hide the others (in a test, one crash can hide later problems). In a formal inspection the moderator plans the meeting and keeps the discussion focused on finding defects, not fixing them.

Metrics

A metric is something we measure. A control metric measures the process (for example, the average time to repair a reported defect), while a predictor metric measures the product (for example, cyclomatic complexity). Be careful: complexity is an internal attribute; it says something about an external attribute like maintainability only if the relation has been validated. Defect density is defects per thousand lines of code (KLOC): 45 defects in 15,000 lines is 45 / 15 = 3 defects per KLOC.

Configuration management

• Version control: records every change to the files. A baseline is a definition of a specific system version that names the exact version of every component.
• Distributed version control (such as Git): every developer has a full copy of the repository, including its history. In a centralised system there is one server copy.
• Change management: a change request is analysed for cost, impact and strategic value, and a change control board decides whether to approve it.
• System building: a build tool such as make rebuilds a target only when one of its direct inputs is newer, and the rebuilt targets cascade. If util.c changes, util.o is rebuilt, and then the final app that depends on it.
• Release management: to reproduce a bug in an old release (such as 2.3 shipped two years ago), you need the exact versions of the compiler, libraries, build scripts and configuration used.

main.c --> main.o --\
                      +--> app
util.c --> util.o --/

Continuous integration means merging often into the shared mainline. Before you push, pull the latest mainline, build the system with your change and run the tests locally, so the shared build stays healthy.

Dependability and safety

When you press the button of an elevator you trust it. Dependability is how much users can trust a system. For some systems, like a train controller or a medical pump, a failure can hurt people.

The properties

Availability and reliability are different. A web service that crashes once an hour but restarts automatically in under a second, without users noticing, has high availability but low reliability.

Availability as downtime: with 99.9% availability over a year of 8,760 hours, the downtime is 0.1% of 8,760, about 8.8 hours.

Safety terms and correctness

A hazard is a condition that could lead to an accident, for example the possibility that an insulin pump delivers an overdose. An accident is an unplanned event that causes harm. Correct software is not always safe: if the software never deviated from its specification, but the specification itself may be incomplete or wrong about hazards, then correct behaviour can still be unsafe.

Chain of events: a fault (a flaw in the code) can produce an error (a wrong internal state), which can produce a failure (the user sees wrong behaviour).

Redundancy and diversity

Redundancy means extra copies, like having a spare tyre. Diversity means the copies are different, so they do not fail for the same reason: for example, two speed sensors from different manufacturers, built on different principles. In N-version programming, three independent teams implement the same specification and a voter takes the majority output. The weakness: teams tend to make similar mistakes, especially where the shared specification is wrong or hard.

Security and safety can conflict

A hospital system that locks a user out for 30 minutes after three failed logins is secure, but during an emergency a doctor who mistypes the password cannot reach patient data. A security measure can reduce availability and, through it, safety.

Agile methods and lean startup

Imagine drawing a picture with a friend, showing it after every few lines and changing it as they say what they like. That is the spirit of agile methods: deliver small working pieces, listen and adapt.

The Agile Manifesto (2001)

1. Individuals and interactions over processes and tools
2. Working software over comprehensive documentation
3. Customer collaboration over contract negotiation
4. Responding to change over following a plan

'Over' does not mean 'never'. The right side still has value; it is only ranked below the left side. A team that says 'we are agile, so we write no documentation at all' misreads the manifesto.

Extreme Programming (XP)

XP practices include pair programming, test-first development, continuous integration, small releases and collective ownership: any pair of developers may change any part of the code, and nobody owns a module. XP also warns about the sustainable pace: long overtime (like sixty-hour weeks) lowers code quality and productivity, so the net gain is small or negative.

Scrum

Agile is hardest to apply, without adaptation, to a safety-critical system built by several distributed teams, where a regulator must approve a detailed specification and documentation.

The Lean Startup

Eric Ries applies the same learning spirit to start-ups, which sell under extreme uncertainty. The loop is build, measure, learn. A minimum viable product (MVP) is the version of a product that lets a team complete one loop with the least effort.

   Ideas
   /    \
Learn    Build
   \    /
  Measure

For a start-up, progress is validated learning about customers. Shipping all features on time with no evidence that customers want them is not progress. Vanity metrics such as total sign-ups can grow while nothing improves. Use cohort analysis, which follows groups of customers who joined in the same period. If 2% of each weekly cohort pays, in every cohort, then paying customers grew because more people signed up, not because the product improved.

Innovation accounting has three steps, in order: establish a baseline with an MVP, tune the engine towards the ideal, then decide whether to pivot or persevere. A pivot is a structured course correction. A zoom-in pivot makes a single feature the whole product (customers use only the time-tracking part of a big project suite, so build only that). Ries names two leap-of-faith assumptions: the value hypothesis (does the product deliver value?) and the growth hypothesis (how will new customers discover the product?).

Simplicity and technical debt

A messy room gets harder to clean the longer you wait. Code is the same. Max Kanat-Alexander's book Code Simplicity says that every piece of software has one purpose, to help people, and that design is the art of keeping things simple so the software can go on helping.

Technical debt

Ward Cunningham's metaphor compares a quick, messy shortcut to a loan. You get something now (speed), and you pay later. The metaphor has three parts.

Debt is not always wrong: taking it to hit a deadline can be a good trade, if you plan to repay it. But the interest is charged per change, so it depends on how often the code is touched and for how long it will live. Debt in code that is changed every week is the debt to pay first. Debt in code that will rarely change, or will be thrown away soon, costs little.

The numbers come from a small example, counted against the plan that takes the shortcut. The shortcut saves 4 days now, every change on top of it costs 1 more day, and replacing it later costs 6 days. After 10 changes, keeping it costs 10 days of interest, and doing it properly costs the 4 days that the shortcut saved. Replacing at once costs 6, and waiting for 3 changes costs 3 + 6 = 9. But if the module will get only 3 changes and then be retired, keeping the shortcut costs just 3 and wins. At 4 changes, keeping and doing it properly tie (4 and 4).

Martin Fowler adds that debt can be chosen or not, and wise or foolish:

The cost of a mess

Robert C. Martin opens Clean Code with this story. A team moves fast at first by making a mess. Every change then takes longer, and the productivity of the team goes toward zero. Everyone promises to clean up later. But later brings new deadlines, so later equals never (LeBlanc's law). The only way to go fast is to keep the code clean all the time.

The Boy Scout Rule says: leave the campsite cleaner than you found it. For code, check it in a little cleaner than you found it: a better name, a function split in two, a duplicate removed. It is the cheapest way to repay debt, because you already have the code open and understand it. It asks for small steps, not a rewrite on every visit.

Management sometimes asks for a grand redesign in the sky: the best developers build a new system from scratch while the others keep extending the old one. It is a race. The new system must do everything the old one does, and the old one keeps moving. It can take years, and a team with the same habits will make the new code messy too.

  old system:  v1 -> v2 -> v3 -> v4 -> v5 ...  (customers want changes)
  new system:  start ...... still catching up ......

  the finish line keeps moving away

Laws of Code Simplicity

• Purpose: the purpose of software is to help people. A design is good when it lets the software be as helpful as possible, keep being helpful, and be easy for programmers to create and maintain.
• Law of simplicity: the ease of maintenance of any piece of software is proportional to the simplicity of its individual pieces. Nobody keeps a whole big system in mind, so what counts is whether the part you are changing can be understood by itself.
• Law of defect probability: the chance of adding a bug is proportional to the size of the change.

The book also gives the equation of software design. The desirability of a change (D) grows with its value and shrinks with its effort, and both are split in time:

        Vn + Vf        Vn = value now       Vf = future value
  D = -----------
        Ei + Em        Ei = effort to implement
                       Em = effort to maintain

Implementation is paid once, but maintenance is paid for as long as the software lives, and future value keeps adding up too. So for software that lives for years, reducing the effort of maintenance matters more than reducing the effort of implementation. A simple design does exactly that. The equation is a way to reason about trade-offs, not a calculator that gives exact scores.

Mistakes with change

The book names three ways to go wrong, and two of them come from trying to guess the future:

• Writing code that is not needed (a feature or a plugin system nobody asked for).
• Not making the code easy to change.
• Being too generic (building for many cases when only one exists).

The cure is to build only what you know is needed, keep it simple so it is easy to change, and generalise when a second real case appears.

// Over-designed: nobody asked for other formats yet
class ReportExporter { /* plugins, a config language, 3 layers... */ }

// Simple: solves today's need
const toCsv = (rows: string[][]): string => rows.map((r) => r.join(",")).join("\n");

Small changes are safer

Imagine two ways to fix one bug: change 10 lines, or rewrite a whole file of 600 lines on the way. By the law of defect probability the rewrite is far more likely to add a new bug, because a change 60 times larger has 60 times more places to go wrong. New code has no old bugs, but it has its own. So make changes small and step by step, so that each one is easy to check. A rewrite may still be justified, but never because it is the safe choice.

Putting it all together

All the sections above are parts of one story: you learn what people need (requirements), draw and design it (UML, architecture), build it in a clean way, check it (tests, inspections), manage the people, time and risk, and keep it alive (evolution), while keeping it simple and safe.

Common mistakes

• Thinking agile means no documentation or no plan.
• Thinking more people always speeds a project.
• Confusing availability with reliability.
• Writing requirements that cannot be tested.
• Refactoring without tests.
• Reading a metric (like cyclomatic complexity) as if it were the quality itself.