4.13.1.5 - Evaluating a solution against its requirements
Learn how to judge a computer system using its agreed requirements, reliable evidence, limitations and realistic improvements.
Success criteria and evidence
Evaluation judges how well the completed solution meets the problem's requirements; testing supplies some of the evidence for that judgement.
Evaluation
A reasoned judgement of a solution's suitability and success, based on agreed criteria and evidence.
Agree success criteria before judging the result. A library reservation system might need to prevent two active reservations for the same copy, return a search within two seconds on a stated dataset, and allow a librarian to cancel a reservation. Each requirement needs a suitable check. A successful login screenshot does not show whether duplicate reservations are prevented.
Use a traceable chain: requirement → relevant test or observation → actual result → judgement. State whether a requirement is met, partly met, not met or not yet demonstrated. “Not yet demonstrated” matters: the absence of a test is not proof of failure, but it is also not evidence of success. Record the conditions under which an observation was made so that a result can be interpreted.
| Criterion | Evidence | Justified judgement |
|---|---|---|
| Prevent a second active reservation for the same copy | Two users attempt the same copy; only the first reservation is stored | Met for the tested case; simultaneous requests still need a suitable check |
| Search within two seconds with 20,000 records | Ten searches on the target computer take 0.8–1.4 seconds | Met in those conditions, not proven for a larger dataset |
| Cancel a reservation | Cancellation removes it from the user's active list but it remains unavailable to others | Not met: the stored availability state is inconsistent |
Evaluate all agreed requirements, including awkward failures. Do not replace the original criteria with easier ones after seeing the results.
Quality, efficiency and limitations
A solution can produce correct outputs in a few tests and still be unsuitable for its intended use.
Consider functional correctness, robustness when faced with invalid input or failure, usability for intended users, efficiency in time and memory, reliability, and maintainability. Security and compatibility are relevant where the problem requires them. Choose criteria that matter to this system, rather than treating these words as a universal checklist with equal weight.
Judge each quality using appropriate evidence. A librarian's observed success in cancelling a reservation supports usability more than the developer's opinion that the screen looks good. Tests involving invalid IDs and a lost storage connection can reveal robustness weaknesses. Representative timed searches on the target hardware can reveal performance limits. Clear responsibilities and documented interfaces support a reasoned maintainability judgement, although they do not prove that all future changes will be easy.
State the limitation precisely and its consequence. “Slow” is vague; “search exceeds the agreed two-second limit at 100,000 records, so the intended larger branch cannot use the current version during busy periods” connects evidence to suitability. Distinguish a known defect from an out-of-scope feature. If automatic reminder emails were explicitly excluded, their absence is not a failure of the agreed requirements. It can still be a possible future extension.
Trade-offs need context. A design that uses additional memory to answer repeated searches more quickly may be sensible on the target machine; it may be unsuitable on a device with little memory. Explain which requirement is affected and how seriously. Do not equate shorter source code with a faster or better system.
Evidence-based improvements
A useful improvement addresses a demonstrated limitation and explains how the changed system would be checked.
For the failed cancellation, first identify the cause: the display changed but the stored copy status did not. Propose updating both related states consistently, explain why that resolves the observed failure, and plan a retest showing that another user can reserve the released copy. Then test nearby behaviour, such as the cancellation of an already cancelled record, to check that the correction has not caused a new fault. A proposal is still a proposal until it is implemented and tested.
Prioritise improvements according to impact, feasibility, risk and the users' needs. Correct conflicting reservations before adding decorative themes. An optimisation may be worthwhile only if realistic use violates an agreed response-time limit. State resources or constraints that affect the proposal; do not promise a complete replacement simply because the existing system has a minor defect.
Independent user feedback can identify effects the developer missed, but should be interpreted critically. “The search is confusing” suggests an issue; observing which step fails reveals a more actionable cause. Compare feedback with requirements and test evidence. One user's request for extra features does not automatically expand the agreed problem.