Why good software estimates are impossible
Subscribe to Engineering Impact, the weekly newsletter for managers of software teams.
This week: Turn mistakes into teachable moments, why good software estimates are impossible, a maturity model for safe deploys, questioning the value of unit tests.
Brian Gilham discusses an important lesson in project leadership that he learned early in his career. He had just implemented some new features for a client’s Rails app before leaving town for the weekend. Kevin, his project lead, called Brian on Friday night to inform him that the client’s site was down. Kevin added that he would take care of it and told Brian to enjoy his weekend.
Brian walked into the office on Monday fully expecting to be fired, but Kevin simply asked Brian what he had learned from this incident. His lessons learned included the need for proper QA, thoroughly testing changes and taking the time to ensure a job is done right. Kevin was satisfied with this answer and never brought up the incident again. Brian often uses this experience to look for the potential in his junior programmers after they screw up.
Solving a Rubik’s cube is similar to estimating the time needed to complete a software project, according to Christian M. Mackeprang. Someone who has never played with a Rubik’s cube is typically unable to solve at all without practice, whereas an expert can solve it in seconds. Likewise, a programmer attempting to solve a task for the first time must first determine a solution before implementing it, which greatly increases the time required to complete the task.
This similarity is based on the idea of unexpected complexity, meaning that a programmer needs to be aware of the code’s situational context before making decisions about modifying it. Unlike mechanical tasks that can be automated to some extent, it isn’t really possible to estimate the time needed to complete a creative task like programming with reasonable accuracy. This rule applies even when the person making the estimation has significant general experience and proficiency in the required tools. All this person can do is base the estimation on historical metrics, which may not be worth very much.
Julia Evans discusses her maturity model for a programmer’s understanding of software operations, which consists of three basic stages. Stage 1 is when the programmer believes that the software generally works fine, even when bugs are occasionally pushed to production. Stage 2 occurs when the programmer realizes that a site can break at any time, especially when it has a high uptime requirement. Programmers enter stage 3 when they acquire the knowledge needed to keep the site up almost all of the time, even when it has high traffic.
Specific solutions that help to achieve the third stage in software operations include a dashboard showing the database’s current status, migrating complex code to a separate web server and removing code that’s unnecessarily slow. This process involves making operations part of the developer’s job, which is a departure from the traditional model of the development team throwing new code over the wall to the production team. Julia adds that the merging of development and operations into “DevOps” provides developers with a better opportunity to learn how to safely deploy changes and understand what their code actually does for the users.
On HackerNews this week, there was a lively discussion about an article on RBCS discussing the usefulness of unit testing. No one questioned the value of unit testing during the early years of programming, when functions generally represented units of computation. However, the use of object-oriented code made it impractical to determine the code’s run-time behavior by simply inspecting it, since classes typically encapsulate multiple functions. Object-oriented programs therefore require a dramatic increase in the number of lines of code needed for unit testing, causing software engineers to debate its value.
Some engineers recommend discarding test cases that rarely fail as a means of reducing the test mass. They argue that the information provided by these cases may not be worth the cost of maintaining and executing them. Other engineers point out that a case may test implementation instead of behavior, which means that removing the case could actually increase maintenance costs if it causes the test to report a false negative.
Our recommendation? Try collecting behavioral insights by measuring the nature of the source code itself. That’ll give you a perspective of which kind of code works best for you – with or without unit testing and why.
If you enjoy Engineering Impact, sign up to get it weekly.
Ben Thompson is a co-founder at GitPrime where he leads design and customer experience. He is a Y Combinator alumni, with a background in product design, branding, and UX design. Follow @thebent on Twitter.
Get Engineering Impact: the weekly newsletter for managers of software teams
Keep current with trends in engineering leadership, productivity, culture, and scaling development teams.