Levels of details in requirements and their relationship with bug-rate

There's a tl;dr at the end.

 I was recently having a conversation with 2 coworkers in which we were discussing whether it was realistic to expect that not-so-obvious details of a product had to be inferred either by a Tech Lead or a Product Manager (or anyone in the team that's not the product owner), we mainly focused on the former because the specific situation was with that role involved. That conversation encouraged me to write this article and go deeper into details in this topic.

 The main problem I want to address in this article is how to know which level of details we should use for our projects. Although level of details in requirements is a pillar for the success of a project there's another crucial factor that could make a project fail spectacularly and that's bugs, I will talk about their correlation with level of details later.

 I will refer to clients, product managers or project managers that are defining requirements as product owner. But there will be cases where we consider product owners as actors that don't directly define requirements in the professional sense, meaning, involved in the creation of the artifacts.

 We are going to be referring to some level of details when it comes to requirements, each row is a level of details and columns are the artifacts that are present or not in levels. Let's match them up briefly so we aren't lost in a sea of possibilities and perceptions. Any time we refer to level of details we are talking about the level of details in the requirements.

• Tasks List assumes that tasks have a minimal description. It also refers to whatever tasks manager the team uses.
• Even though Tasks List is set to YES in None, there are cases where developers are just working by their own and they know what they are doing so they don't even require that.
• Being on a level doesn't mean that everything on the project must be done with those artifacts exclusively, other requirements can be specified in lower levels (not a higher one though, because that would mean we are in the higher level instead).
• Gherkin is a language used to specify scenarios/use cases, their inputs, their conditions and their outputs. It's a highly verbose tool to document behavior.
• Level of details are presented with artifacts in them but it doesn't mean that you cannot use Reqs Doc and AC ignoring design. The important thing is the needs of the project, so this levels representation is just a simplification to use as baseline for this article's argument that must not be consider as a methodology or anything similar.

 Now let's talk about some case studies in which level of details are affected by either actors or conditions. This will help us explore why different levels are necessary and what criteria should be used to consider what level should we be working in.

Case Study 1: The Lone-wolf developer as its own Product Owner

 This case is pretty straightforward. The most likely level of details that's going to be use here is None. Why? Because the developer only needs a tasks list to remember all the things that he/she planned (sometimes, the tasks list is also non-existent). It doesn't mean they cannot move to Basic or Moderate, if they do, they are probably already working with a team and not by themselves. The perception of progress in this scenario is very fast, the developer doesn't have any constraints related to requirements, design or uses cases that must be considered, the constraints are on their mind and dropping/changing them is just a matter of thinking.

Case Study 2: The Lone-wolf developer with a non-technical Product Owner

 In this situation, the developer has to discuss with the product owner to both extract and agree on requirements. This is very common in small projects and the developer will have to change hats constantly between the technical side of things and bringing down to earth the ideas that the product owner is presenting. The most probable level of details in this one is from Basic to Detailed. None level could also happen if the developer is not experience enough to communicate the product owner that a requirements document is necessary and why. Not everything is on the shoulders of the developer though, it could happen that the product owner is somewhat experience in software development projects planning and it could come with the artifacts of a level of details of Moderate already created.

Case Study 3: The Lone-wolf developer with a technical Product Owner

 The difference between this case study and the previous one is that the product owner is technical. As a consequence, the level of details could range from None to Basic or None to Detailed. Very Detailed is unlikely to happen because product owners often look for Senior Developers so they can defer technical decision-making to them so it kind of defeats the purpose.

 The product owner will probably be doing code reviews and following the process more closely (at least at the beginning), when that's the case, adjustments could happen sooner in the development process and that's why a level of details higher than Detailed in this scenario would most likely be a waste of time. The reason behind this is that both the developer and the product owner will learn and adapt requirements while doing the actual work.

Case Study 4: Developers with a non-technical Product Owner

 The level of details of requirements in this case will have to slightly increase in comparison to the previous, not only because of the non-technical product owner but also for having multiple people in the team.

 Another factor that will determine what level of details we need in the requirements is the average seniority of the developers. For example, if you have a team of 5 Junior Developers and 1 Senior Developer (who's going to take care of the communication with the product owner and requirement gathering) you will most likely need a level of details between Detailed and Very Detailed. Simply put, Junior Developers will struggle inferring logic from a requirements document or a task with just a brief description. On the other hand if most developers on the team are senior you can lower the level of details to Basic or Moderate depending on the degree of understanding that these developers have in the domain of the system.

Case Study 5: Developers with a technical Product Owner

 Under this scenario it's highly possible that the product owner will be writing requirements themselves, or he/she will be in constant communication with the team lead of the developers so they can help them define requirements. Again, this situation is similar to the previous, the level of details depends on the average seniority of the team and it might range between Basic and Very Detailed. Another factor enters into play though, the product owner being technical will also affect the level of details drastically. Some people have a preference over high or low detailed requirements so this will affect what artifacts will be considered as requirements depending on the inclination that the product owner has.

What about scenarios with other roles?

 Of course a team is not only developers, and the product owner is not only the product owner (they could also have it's own product manager or a CTO that helps in the technical side, etc). We won't try to make a case for each of the different configurations of teams with different roles but we do have to talk about 3 roles that are also part of software development teams and have a huge impact in the level of details. In this case, we refer to these roles as additional members of the team apart from the product owner.

 The Product Manager role affects requirements a lot, if this role is in a team it means that requirements are going to be well defined (that's a given) and the level of details practically defaults to Very Detailed, as a minimum, they will be writing acceptance criteria.

 The Project Manager role doesn't necessarily affects requirements like the previous role, but if the product manager is not in the team it's very common that the Project Manager will, in most cases, wear the hat of that role. Keep in mind that this is only related to requirement gathering/defining and documentation, there are other activities that a Product Manager does that a Project Manager doesn't even have to think of. That said, the level of details will most likely be in Very Detailed.

 Finally, the QA/Tester role, when this role is in the team the presence of requirements becomes a must, the level of details has to be Very Detailed and not lower than that. The reason for this is that for QA/Tester to do their job they must know what they are going to test and if you have requirements that are basically just a ticket with a description you are leaving a lot of room for QA/Tester to assume details (this also applies for developers) this not only make their work harder and in some cases even impossible, it's dangerous, because they might "confirm" things under those assumptions which will let tasks be moved to done without really being done. I know this won't happen under good QA/testers, they will simply refuse to make assumptions in the first place.

What about Highly Detailed level?

 We haven't attribute this level to any of the example, this was intentional. I wouldn't recommend Highly Detail level unless you either have a ridiculous high complex domain logic or you are going to use the Gherkin notation that was created in the definition phase of requirements as part of your behavior test suite in development. I'm sure there might be a plethora of cases where Highly Detail level is a good fit but I'll leave that to you, the reader, to figure out in the different contexts you'll encounter in your job environment.

Bug-rate: so is there really a relationship?

 There is definitively a relationship, one would think that Highly Detail level would reduce bug-rate, and the opposite could also be true. I think that statement is reasonable, but then, why doesn't waterfall works flawlessly? In that case a lot of definition is done upfront, a lot of thinking, design, wireframes, and still projects fail miserably with a high bug-rate and low quality under that methodology.

 What if we use agile? That would save us, right? I have also seen projects using agile (not Agile™) and still fail with a high bug-rate. I still think the relationship is there, but not all bugs (or most of them) comes from a low level of details or a high one. let's explore where bugs come from to shed some light to the bug-rate topic.

Lack of details in requirements

 There are things that people can infer from requirements that are incomplete and other things that they cannot or should not. let's expose 2 examples that will help us understand the point better.

First Example: Requirements explain a functionality where the developers must create a form to capture some data and then process it in the backend. The requirements don't specify anything about validation, still, developers will consider validation as a mandatory basic feature the form must do even if it's not in the requirements.

This is a positive case in which people working in the implementation can infer logic that's required to deliver quality results. We have to be careful though, developers should infer things that don't require too much time or effort to accomplish and prevent doing so too much. This could lead to problems with product owner because the developers end up working on things that aren't useful from the product owner perspective.

Second Example: Requirements explain a functionality where developers must create some tables to visualize some data. In the requirements' artifacts it's implied that the table is practically read-only, that there's nothing the user can interact with, which seems fine. After developers implement this functionality and present it to the product owner they explain that they use Airtable and that they expected the functionality to be like Airtable's tables (with custom filters, search, sort, pagination, highlight of column+row, etc).

In this case, there's no way that developers can infer the missing details from the requirements, it has to be explicitly stated in the requirements and even if they find a way to infer them, implementing them would be wrong because it could happen that the product owner actually wanted/needed a read-only table and working in all those Airtable's functionality would be a waste of time.

This is a negative case in which all the lacking features from Airtable are going to be perceived as more work to be done or "bugs", but in reality, it's just work that has to be done, the same work that had to be done whether the requirements were correct in the first place or not.

Known unknowns

 Another point that could make bug-rate of a project increase is known unknowns, even if we know the unknowns there will be new information that will expose itself in the development process that will potentially end up being unknown unknowns resulting in unexpected behaviors or edge cases that developers couldn't spot before implementing the functionality and now that same feature that was "correct" (under the assumption of the known unknown) in the requirements, it's incomplete or with flaws in reality. I think some of these bugs can be avoided but not all of them.

Unknown unknowns

 These I can say with confidence that are the bugs that are very difficult to prevent, these bugs come out of nowhere and you only get to notice them because they are happening in front of you under a circumstance that you never even thought about it. Unknown unknowns will appear for multiple reasons, lack of requirements, domain complexity, code base complexity and finally just by the mere act of implementing ideas through software development.

Domain Complexity

 There are projects that are very simple, for example just a CRUD of some entities and then just render them with some HTML. On the other hand there are projects where business logic requires a lot of relationship and communication between the different entities. Bug-rate in the simpler project would likely be lower than in the one that has a high complexity in their business logic.

Developers skill

 Of course, developers skill will also affect the bug-rate of the project. If developers are in their early Juniors days and not under supervision of a Senior developer this could easily become the largest source of increase for the bug-rate metric and even make a project simply not feasible to be continued.

Is bug-rate inversely proportional to the levels of details?

 I would say yes, the thing is that is not necessarily the only source that increments it and not even the most important one.

 I don't have data to back this up so I'll go with gut feeling here, I would say that most bugs comes from known unknowns and unknown unknowns (this on the case where developers skill is not an issue),  developers fix some of these bugs before they reach the public (be presented to the product owner) and a lot of them do pop in the final result.

 The problem with bugs that are generated because of low level of details is that they are going to make it almost always to the public, because if the developers don't infer them, that's work that won't be done and will be pointed out as missing or as bugs by the product owner and so the perception that low level details is the source of most bugs (yes, of those that make it to the final result) gains strength, falsely so.

Low level of details - Probably higher bug-rate

When a team is working under a low level of details it will naturally happen that the product owners will find a lot of problems on what was delivered and it's highly probable that a lot of fixes will have to be made. This is basically the price that has to be paid for being in the lower levels of details. We don't have to perceive this as wrong, it is simply a trade-off, and under some circumstances or conditions of a project this is actually better than sitting and producing a Highly Detailed level that no one knows if they will pass the test of reality (developers could try and guess details, but it doesn't guarantee correctness in the product owner's eyes). Although it's crucial that the product owner considers a position of high tolerance or the process of setting expectations helps them get into that same place. The why behind high tolerance position is that we as a team are consciously deciding to work on a low level of details so problems of synchronicity between what the product owner expects and reality are very likely to happen.

High level of details - Probably lower bug-rate

On the other hand when a team is working under a high level of details the problems that the product owner could find might be less, one could say that problems/issues are inversely proportional to the level of details (sort of). In this case something happens, the tolerance to problems is decrease (it just happens), because the team assumes that since everything was defined then it means that the development of the product must be, if not flawless, very near to that. Sadly, reality hits and there's no such thing, there are always going to be issues and bugs that must be fixed, changes, re-work or even waste time on work that wasn't required. What a high level of details does in this case is reduce all those possible issues but never takes them to zero, the difficult part is to provide visibility of this prevention work, there's no simple way to expose the value of what was prevented, naturally, the wrong didn't happen.

Conclusion

I mentioned at the beginning of the article that bugs could be a reason for the project to fail spectacularly. A product owner that's frustrated by the software development process and all it's little details, problems, issues, changes and fixes will suddenly implode by it's own frustration losing credibility in the team and even losing hope that the implementation of the initial idea is feasible.

I have seen different levels of details be successful in different teams setup so the answer to what level of details should we use is the classic it depends. In my humble opinion opting for a Detailed or Very Detailed level is what I would suggest for most scenarios, and when in doubt just sit back and think about the context of your team setup and the nature of the project. Projects with high complexity in their domain will benefit the most from high level of details, projects with low complexity in their domain could probably do just fine in the lower levels.

Make sure that you don't end up in situations where the level of details is low and the tolerance is also low as that particular configuration will create a lot of friction and frustration for the whole team. Furthermore, when you are on the higher levels of details situation, communicate properly that problems may still arise, that there will be less of them of course and a lot will be prevented, but the process of learning and adapting for both the technical and business side of things must and will happen as that's an unavoidable aspect of creating products through software development.

tl;dr

• Check level of details table at the beginning of article.
• Working between Detailed and Very Detailed is a "safe bet".
• Levels of details are inversely proportional to bug-rate (doesn't apply to the solo developer who is its own product owner).
• Avoid being in low level of details and low tolerance.
• Don't fall in the trap that high level of details results into perfect software, it does improve it, but there are other factors that could still impact the project in irreversible ways.
• Learning and adapting is necessary and unavoidable in software development. It's orthogonal to the levels of details.