2 Processes 🔗

In software development, perfect is a verb, not an adjective. There is no perfect process. There is no perfect design. There are no perfect stories. You can, however, perfect your process, your design, and your stories. – Kent Beck

This chapter begins with a discussion on the importance of software processes (Section 2.1). Next, we explore general and foundational aspects of agile processes (Section 2.2), including an examination of the historical context that motivated their emergence. The subsequent sections focus on three prominent agile methods: Extreme Programming (Section 2.3), Scrum (Section 2.4), and Kanban (Section 2.5). Following this, we consider scenarios where agile methods might not be the most suitable approach (Section 2.6). Lastly, in Section 2.7, we provide a brief overview of some traditional processes, such as the Unified Process.

2.1 Introduction 🔗

Car production in an automobile factory follows a well-defined process. First, sheets of steel are cut and pressed into shapes for doors, roofs, and hoods. Next, the car is painted and components such as the dashboard, seats, seat belts, and wiring are installed. Finally, the mechanical parts, including the engine, suspension, and brakes, are fitted.

Analogously, software production should also follow a process, although it is far less mechanized and more dependent on intellectual endeavor. A software development process defines a sequence of steps, tasks, events, and practices that developers must follow while creating a software system.

Critics of software processes often question their need, asking: Why should I follow a process?or What process did Linus Torvalds or Donald Knuth use when implementing the Linux operating system or the TeX text formatter?

However, the second question above is not entirely relevant to our context, as both Linux (in its initial versions) and TeX were individual projects led by a single developer. In such scenarios, following a process is less important. More accurately, the process followed in those projects was a personal one, reflecting the principles, practices, and experience of a single developer.

Nevertheless, contemporary software systems are often too complex to be developed by a single developer. As a result, systems implemented by individual developers are increasingly rare today. In practice, modern systems—the focus of this book—are developed by teams.

These teams require at least a basic set of guidelines to produce quality software effectively. This is why companies invest heavily in development processes. They serve as tools for companies to organize, coordinate, motivate, and evaluate their developers, ensuring productivity and alignment with the organization’s goals. Without a process, even a lightweight and simplified one such as the agile methods we will study in this chapter, teams risk working in an uncoordinated way, potentially resulting in products that lack business value. Moreover, processes benefit both the company and developers by clarifying expected tasks and outcomes, thus reducing misalignment among team members.

In this chapter, we explore various software processes. In fact, in Chapter 1, we previously presented an overview of Waterfall and Agile methods. In the following sections, we will continue this discussion and cover three well-known Agile methods.

2.2 Agile Manifesto 🔗

The earliest software development processes, such as the Waterfall process proposed in the 1970s, followed a linear, sequential approach. Usually, projects began with a requirements specification phase and progressed through implementation, testing, deployment, and maintenance phases.

Given the historical context, the adoption of this strictly sequential approach was understandable. Indeed, projects in traditional engineering fields typically follow a sequential process based on detailed and up-front planning. Therefore, it’s not surprising that the emerging software engineering field initially mirrored the processes of longer-established domains, such as electronics, civil, mechanical, and aeronautical engineering, among others.

However, by the 1980s, it became increasingly evident that software development differs significantly from other engineering products. The recurrent problems faced by software projects during this period reinforced this conclusion. For instance, such projects frequently overshot schedules and budgets. In extreme cases, some were even abandoned after years of effort without delivering a functioning system.

The CHAOS Report, published in 1994 by the Standish Group consulting firm, provided quantitative evidence about the state of software projects at the time. This report revealed that over 55% of projects surpassed their deadlines by 51% to 200%; and at least 12% exceeded the deadlines by more than 200%, as shown in the next figure.

Regarding costs, almost 40% of the studied projects overshot their budgets by 51% to 200%, as indicated in the following figure.

Therefore, in 2001, seventeen software industry professionals gathered in Snowbird, Utah, to discuss and propose an alternative to the prevailing Waterfall-based processes. They argued that, due to its unique nature, software requires a distinct development process compared to traditional engineering products.

For instance, software requirements change more frequently than those of other engineering projects, such as airplanes or bridges. Moreover, customers often lack a clear understanding of their needs. As a result, there is a constant risk that a product might become obsolete by the time it’s finished, due to changing circumstances or customer needs. The professionals who met in Utah also identified issues with the document-centric approach advocated by Waterfall. Specifically, the detailed requirements documents common at that time tended to become obsolete quickly, as developers often did not update them to reflect changes in requirements. In response, the group proposed a new paradigm for software development processes, which was described in a document they named the Agile Manifesto. The manifesto reads:

Through this work, we have come to value:

Individuals and interactions over processes and tools

Working software over comprehensive documentation

Customer collaboration over contract negotiation

Responding to change over following a plan.

Agile processes are characterized by short and iterative development cycles. Systems are built incrementally, starting with the most critical features, according to customers. Initially, a first version of the system is created, implementing only high-priority functionality. This version is then validated by the customer. If approved, a new cycle, also called iteration or sprint, begins, during which a few more features are added. These cycles are typically short, lasting about two weeks, for example. This allows the software to be incrementally built, with each feature increment receiving explicit approval by the customer. The development process ends when all the customer’s prioritized requests have been implemented.

The next two figures provide a comparison between Waterfall and Agile methods:

However, the previous figures might suggest that in agile development, each iteration replicates a mini-Waterfall process, encompassing all the Waterfall phases. This interpretation isn’t accurate; generally, iterations in agile methods do not follow a strict sequence of tasks as in Waterfall (more details will be provided in the following sections). The figure may also imply that a system must be put into production at the end of each iteration. This is also incorrect. Indeed, the objective is to deliver a functional system that performs useful tasks for the customer. However, the decision to launch the software involves other factors such as business risks, hardware availability, marketing campaigns, user training, and so on. Other key characteristics of agile processes include:

• Less emphasis on documentation: Only essential information should be documented, focusing on what is necessary for understanding and maintaining the system.

• Less emphasis on detailed plans: At the start of a project, often neither the customer nor the developers have a clear understanding of all requirements that should be implemented. This understanding gradually unfolds as sprints are completed and their outcomes validated. In essence, the core principle of agile is to progress even when we have incomplete, partial, and changing requirements.

• No dedicated design phase, i.e., no Big Design Up Front (BDUF): The design of the software is also incremental, evolving with each iteration.

• Small teams: Teams typically consist of about a dozen developers or, as Amazon CEO Jeff Bezos phrased it, teams that can be fed with two pizzas. This concept can also be compared to the size of sports teams, ranging from a basketball team (5 members) to a soccer team (11 members).

• Emphasis on innovative programming practices (as of the early 2000s), such as automated testing, refactoring, and continuous integration.

Due to these characteristics, agile processes are considered lightweight, with few prescriptive elements and minimal documentation.

However, these characteristics, while important, remain quite generic and broad. Thus, to make agile principles more concrete and actionable, several specific methods were proposed. Notably, many of these methods were proposed before the 2001 Agile Manifesto’s meeting. In this chapter, we will specifically examine three prominent agile methods:

• Extreme Programming (XP): Kent Beck proposed this method in a book published in 1999 (link). A second and heavily revised edition was released in 2004.

• Scrum: This agile method was introduced by Jeff Sutherland and Ken Schwaber in 1995 (link).

• Kanban: This method originated from a production control system implemented in Toyota’s factories in the 1950s (link). In recent years, Kanban has been gradually adapted for software development.

It’s also important to understand that all development methods are essentially a set of recommendations. Each organization should carefully evaluate these methods and determine their suitability for its specific context. As a result, organizations may find it necessary to adapt existing methods to their needs. In fact, it’s uncommon to see two organizations that follow exactly the same development process. This observation applies even to organizations that claim to be using the same method, such as Scrum.

In-Depth: The terms process and method are often used interchangeably. However, there are also some subtle differences between them. In the context of software development, a process is a set of steps, events, and tasks used to construct software. Every organization employs a process to develop their systems, which can be Agile, Waterfall, or even ad hoc. Regardless of its nature, a process always exists. In contrast, a method specifies a particular process (the term originates from Greek, meaning the means to achieve a goal). Therefore, XP, Scrum, and Kanban are examples of agile methods or, in other words, they define practices, activities, events, and techniques that align with agile principles.

Real World: The widespread success and impact of agile methods are remarkable. Currently, an overwhelming majority of organizations, regardless of their size or business domain, adopt agile principles to varying degrees. A compelling statistic comes from the 2018 Stack Overflow survey, which included a question about the most used development methods (link). Among over 57,000 professional developers who responded, a majority identified with agile methods or practices. This included methodologies we will study in this chapter—specifically, Scrum (63% of responses), Kanban (36%), and Extreme Programming (16%). In contrast, only 15% of participants identified Waterfall as their development method.

2.3 Extreme Programming 🔗

Extreme Programming (XP) is a lightweight method for developing software with ever-changing or unclear requirements, such as the Type B systems (Business), according to the classification described in Chapter 1. Like any agile method, XP embodies characteristics like short iterative development cycles, less emphasis on comprehensive documentation, incremental design, and development in small teams.

Thus, XP is not a prescriptive method that outlines a step-by-step plan for software construction. Rather, XP is defined by a set of values, principles, and practices. The values and principles are important in shaping the culture of software development teams. They are key components of the method that give meaning to the practices proposed by XP. Thus, if an organization is unprepared to embrace the XP mindset, as represented by its values and principles, it should not invest time in adopting the practices.

In this chapter, we will begin by presenting the values and principles of XP. Here is a list of them:

• Values: Communication, simplicity, feedback, courage, respect, and quality of life.

• Principles: Humanity, economics, mutual benefits, continuous improvements, acknowledging failures, baby steps, and personal responsibility.

After that, we will explore the practices, which are organized into three groups: process practices, programming practices, and project management practices. Here is a list of the practices within each group:

• Process Practices: Customer representative, user stories, iterations, releases, release planning, iteration planning, planning poker, and slack.

• Programming Practices: Incremental design, pair programming, test-driven development (TDD), automated builds, and continuous integration.

• Project Management Practices: Metrics, working environment, and negotiated scope contracts.

2.3.1 Values 🔗

XP advocates that three main values should primarily guide software projects: communication, simplicity, and feedback. Indeed, these values are important in any effort involving human collaboration. Effective communication is important not only to avoid mistakes but also to learn from them. The second value emphasized by XP is simplicity, as within every complex and challenging software system there is a simpler system that is sometimes overlooked. The third value, continuous feedback from stakeholders, serves to mitigate risks such as changing requirements and evolving programming technologies.

Indeed, it is often challenging to develop the right software at the first attempt. Frederick Brooks has a well-known recommendation regarding this challenge:

Plan to throw away parts of your system, because you will.

Thus, feedback is an essential value to ensure that the software parts that are going to be discarded are identified as early as possible, reducing waste and rework. In addition, XP also promotes other values such as courage, respect, and quality of life.

2.3.2 Principles 🔗

XP’s values are abstract, while its practices are concrete and pragmatic. Thus, to bridge this gap, XP advocates following a set of principles. We can envision this as a river with the values on one bank and the practices on the other. The principles serve as a bridge connecting both sides. The following are the primary principles of XP:

Humanity. Software development heavily relies on human capital. The principal resource of software companies is not their physical assets, such as computers, buildings, chairs, or Internet connections, but rather their developers. A term that nicely reflects the importance of this principle is peopleware, which was coined by Tom DeMarco in a book with the same title (link). It emphasizes that managing people, including their expectations, growth, motivation, and responsibilities, is essential for the success of software projects.

Economics. Software development requires a substantial financial investment. Therefore, delivering economic results is very important, particularly for Type B (Business) systems, according to the classification discussed in Chapter 1.

Mutual Benefits. XP advocates for project decisions that benefit multiple stakeholders. For instance, a software company should ensure a good work environment and, in return, the team should deliver software that adds value to the business. Another example is the writing of tests: they benefit the developer by helping to detect bugs in their code. Additionally, tests benefit other developers by providing confidence that their code does not introduce regressions, meaning bugs in previously working code. A final example is refactoring, which makes code cleaner and easier to understand for both the original developer and future maintainers.

Continuous Improvements. As expressed in the opening quote by Kent Beck, no software development process is perfect. Therefore, we should strive for a process that is under continuous improvement, at each iteration, incorporating feedback from customers and team members. For this reason, XP discourages investing significant time in a comprehensive initial design. Instead, software design should be incremental, evolving with each iteration. Moreover, the programming practices themselves should be subject to improvement; teams should allocate time to discuss and reflect on them.

Failures Happen. Software development is inherently risky. As mentioned in Chapter 1, software represents one of the most complex human creations. Consequently, failures are inevitable in software projects, including major bugs, features that do not attract users, and issues with non-functional requirements such as performance, privacy, and availability. Evidently, XP does not advocate covering up these failures. However, they should not be used to punish developers or other team members. Rather, confronting failures is an integral part of the process for teams aiming to deliver timely solutions to problems through software.

Baby Steps. Secure, tested, and validated progress, even if incremental, is preferable to large software increments that risk being discarded by users. This principle applies equally to tests (which are useful even when assessing small units like individual methods), code integration (daily integration is preferable to the stress of major integration after weeks of work), and refactorings (which should occur in small steps, interspersed with running tests). In summary, the focus should be on continuous improvements, regardless of scale, as long as it advances the project in the right direction.

Personal Responsibility (or Accepted Responsibility). According to this principle, it is essential for developers to have a clear understanding of their role and responsibility within the team. The rationale is that responsibility cannot be transferred without the other party accepting it. For a similar reason, XP advocates that the developer who implements a user story—i.e., a small feature increment—should also be responsible for its maintenance.

Real World: One of the earliest systems developed using XP was the Chrysler Comprehensive Compensation (C3) system, a payroll system for the car manufacturer Chrysler (link). The project initiated in early 1995 but failed to deliver concrete results, leading to its relaunch the following year under Kent Beck’s leadership. Martin Fowler, another prominent figure in the agile community, participated in the project as a consultant. During the C3 system’s implementation, many aspects of what would soon be named XP were defined and tested.

2.3.3 Process Practices 🔗

Like other agile methodologies, XP advocates for direct customer involvement in the project. For example, teams should include one customer representative who is an expert on the domain of the system under development. This member is responsible for proposing and writing the user stories, which are XP’s version of requirement specification documents. However, user stories are concise and lightweight documents, typically two to three sentences long, in which the customer representative outlines the features the system should implement.

User stories will be explored in depth in Chapter 3. For now, it should be noted that they are typically written on cards, either handwritten or using project management tools. As mentioned earlier, user stories are concise documents that highlight the key features of the system from a user perspective, in contrast to the complex requirement documents commonly employed in Waterfall-based approaches.

Here is an example of a user story for a Question & Answer forum application, similar to Stack Overflow, which we will use to present XP in this chapter.

This story has a title (Post Question) and a brief description in two sentences. User stories serve as reminders for subsequent detailed verbal specifications provided by the customer representative. Thus, developers can consult the customer representative at any time to clarify issues about user stories.

After being created by the customer representative, developers should estimate the effort required to implement the stories. This estimation is often performed using story points, rather than hours or person-days. In this approach, an integer scale is used to assign a certain number of story points to stories. The simplest stories are assigned one story point. If a story is deemed twice as complex, it is assigned two story points, and so on. In fact, story points are commonly determined using a Fibonacci sequence, such as 1, 2, 3, 5, 8, 13. The aim is to use a scale that progressively reflects the increasing difficulty of the stories and also enables the team to make comparisons like the following: in terms of effort, is a task with 8 story points equivalent to a task with 5 story points plus a task with 3 points (i.e., the two preceding values in the scale)? If yes, 8 story points is a good estimate. If not, it is better to estimate it as having 5 story points.

In-Depth: One technique for estimating story points is called Planning Poker. Here is how it works: the customer representative selects a story and reads it aloud to the developers, who can then discuss it with the representative to clarify any doubts. Afterwards, each developer independently estimates the story’s size in story points, and then all participants simultaneously reveal their estimates by using previously distributed cards with numbers such as 1, 2, 3, 5, etc. If there is a consensus, the team moves on to the next story. If not, further discussion and voting occur until a consensus is reached.

The user stories are implemented in iterations, which have a fixed duration, usually ranging from one to three weeks. These iterations are also organized into longer cycles, referred to as releases, which typically last two to three months. A team’s velocity refers to the number of story points it can implement within an iteration. The customer representative is expected to propose stories that take at least one release to implement. Therefore, in XP, the planning horizon is one release, typically spanning a few months.

Note: The term release in XP differs from its usage in configuration management, where it refers to a version of a system that is made available for use. However, the version produced at the end of an iteration or release in XP does not necessarily have to be put into production.

To summarize, to initiate the use of XP, an organization needs to:

• Define the duration of each iteration.
• Define the number of iterations within a release.
• Create a list of user stories, as proposed by the customer representative.
• Obtain estimates for each story from the developers.
• Define the team’s velocity, i.e., the number of story points that the team can implement per iteration.

After defining the above aspects, the customer representative must prioritize the stories. To achieve this, the representative must select the stories to be implemented in each iteration of the first release, taking into account the team’s velocity For example, assume that a team has a velocity of 25 story points per iteration. In this case, the customer representative cannot allocate stories to an iteration if their combined story points exceed this limit. The process of assigning stories to iterations and releases is known as release planning or the planning game, which was the term used in the first edition of the XP book.

Consider the Q&A forum we mentioned earlier. The following table summarizes the outcome of a possible release planning. In this example, we assume that the customer representative wrote 8 stories, that each release has two iterations, and that the team’s velocity is 21 story points per iteration (note that the sum of the story points of each iteration is 21).

The table emphasizes two points: (1) the stories in XP represent the main features of the intended system; (2) developers do not control the order of story implementation; this is decided by the customer representative, who must be someone capable and with authority to define the most urgent and important features for the organization contracting the development.

After the release planning, the iterations begin. At the beginning of each iteration, the team must meet to conduct an iteration planning. The purpose of this planning is to decompose the stories of an iteration into tasks that can be assigned to the team’s developers. For example, the Post Questions story can be decomposed into the following tasks:

• Design the web interface, including layout, CSS templates, etc.
• Install the database, and design and create the database tables.
• Implement the data access layer.
• Install and test the web server framework.
• Implement the control layer, with endpoints to register, remove, and update questions.
• Implement the web interface.

Generally, these tasks should not be complex and should be completed within a few days, for example.

In summary, an XP project encompasses:

• Releases, which are groups of iterations spanning a few months.
• Iterations, which are groups of stories and respective tasks, lasting a few weeks.
• Tasks, which take a few days to complete.

Once the tasks are created, the team should also decide who is responsible for each one. With this done, the iteration begins and the developers start working on their respective tasks.

An iteration concludes when all the stories have been implemented and validated by the customer representative. Therefore, at an iteration’s end, the stories should be demonstrated to the customer representative, who must verify and confirm their implementations.

XP also proposes that teams should program slack tasks, which are tasks that can be postponed during an iteration. Examples include studying a new technology, taking an online course, improving documentation, or even developing a side project. In fact, companies like Google are renowned for allowing developers to use 20% of their time for personal projects (link). In XP, slack tasks have two primary objectives: (1) to create a buffer that can be used if a task requires more time than estimated; and (2) to allow developers some downtime from the demanding and intense project pace.

Frequently Asked Questions 🔗

Let’s answer some questions about the XP practices we have just studied.

How long should an iteration be? There isn’t a definitive answer, as numerous factors such as team characteristics, system domain, and requirements all play a part. Short iterations—typically a week long—offer quick feedback. However, they also require more customer involvement, as new software increments need weekly validation. Moreover, stories must be simple and straightforward to fit into a 1-week iteration. Conversely, longer iterations—such as a month—grant the team more time to plan and execute tasks with reduced pressure. The trade-off is delays in receiving customer feedback, which is particularly relevant when the requirements are unclear. As a result, a 2-3 week timeframe tends to provide a balance between rapid feedback and less pressure on the team. Another useful recommendation is to experiment, testing and evaluating different durations before settling on one.

What role does the customer representative play during iterations? At the beginning of a release, the customer representative drafts the stories to be included in that release. They then validate and approve the implementation of these stories at the end of each iteration. The representative also needs to be readily available during iterations to address questions the team may have. Since user stories contain only a few sentences, such questions are likely to arise.

How do we select the customer representative? The most important criteria are that the person has a solid understanding of the system’s domain and the authority to prioritize user stories. There are at least three potential scenarios for customer representatives:

• In a scenario where a team is working on a system for a department within their own company, the customer representative might be an employee from that department.

• When a team is working on a system for a different company, the customer representative should ideally be an employee from that contracting company who has knowledge of the software domain.

• For a team working on a product that can be contracted or subscribed to by any client, the ideal customer representative should come from the marketing, sales, or business departments. They should be closely aligned with the problem but distant from the solution. For this reason, it’s preferable not to have a developer in this role. This type of representative is often referred to as a user proxy.

How to define the team’s velocity? There is no silver bullet for this issue. The definition depends on the team’s experience. If the team members have already participated in projects similar to the one they are starting, this is a less difficult task. Otherwise, the team needs to test and calibrate their velocity throughout successive iterations.

Can stories refer to software engineering activities? No, because the customer representative—who is not a software engineering expert—is the one responsible for defining the stories. However, a story could trigger a task like install and test the database. In other words, stories refer to functional requirements, but the tasks created for implementing these stories may relate to functional requirements, non-functional requirements, or technical tasks such as database or framework installation.

What if story X is prioritized before Y, despite depending on Y? For instance, suppose the customer representative has allocated the story Post Question to iteration 2 and Post Answer to iteration 1. Although it seems contradictory, the team should respect this order as the final decision always lies with the customer representative. But then the dilemma might be How can we post responses without having questions? To resolve this issue, placeholder questions that can’t be edited by users can be used. In iteration 1, these questions will appear by default, and users will be able to respond to them.

When does an XP project conclude? The project concludes when the customer representative determines that the already implemented stories are sufficient, and there are no more relevant stories to be implemented.

2.3.4 Programming Practices 🔗

The name Extreme Programming was chosen due to the novel programming practices the method proposed at the end of the 1990s. The essence of XP revolves around these programming practices and on the production of running code since the first weeks of a project. The importance of this fact can be better understood within the context of the ’90s where distinct roles for analysts and programmers were the norm. Analysts at that time were responsible for the high-level design of the system, defining its main components, classes, and interfaces using modeling languages, like UML, which we will discuss in Chapter 4. Only upon completion of the analysis and design phases, would the coding phase start, under the responsibility of programmers. This process inherently created a hierarchy where analysts were often regarded as having the more prestigious roles. However, XP challenged this hierarchy by advocating for coding to begin at a project’s outset.

Simultaneously, to mitigate the risk of producing low-quality code, XP introduced a novel set of programming practices, including pair programming, automated testing, test-driven development (TDD), automated builds, and continuous integration, among others. Most of these practices were widely accepted within the software industry, and are extensively used nowadays even by projects not entirely based on agile principles.

We will now look at XP’s programming practices.

Incremental Design. XP projects do not have a traditional design phase, known as Big Design Up Front (BDUF). Instead, XP promotes the idea of making design a continuous and incremental activity. This approach aims to address issues that arise when a big design phase occurs at the beginning of a project. At this early stage, the project’s requirements are often unclear both to the team and to the customers, leading to inaccurate estimations of their value. Moreover, as the project progresses, new requirements may emerge, potentially making the initial design obsolete.

For this reason, XP contends that the optimal time to consider design is when it becomes essential. Two phrases often used to encourage and justify this practice are: Do the simplest thing that could possibly work and you aren’t going to need it, with the latter known by the acronym YAGNI.

To better understand XP’s incremental approach to design, two key points should be highlighted. First, proficient teams often have a high-level design in mind before the first iteration of a project. This might include knowing that the system will have a web interface, a set of backend components for implementing complex business logic, and also components for accessing relational databases. These decisions, by themselves, already cover and solve major design concerns. Second, during any iteration, the team can also propose a technical task to discuss and refine the system’s design.

Finally, it’s important to note that incremental design should be used in conjunction with other XP practices, especially refactoring. XP insists on the importance of refactoring to enhance the design’s quality. Therefore, teams should seize every opportunity to refactor without delay.

Pair Programming. Accompanied by incremental design, pair programming stands out as one of the most controversial practices proposed by Extreme Programming. The concept is straightforward: every coding task, whether it’s implementing a user story, writing a unit test, or fixing a bug, should be conducted by two developers working together, using the same keyboard and monitor. One developer takes on the role of driver, controlling the keyboard and mouse, while the other acts as a navigator or reviewer. The navigator monitors the driver’s decisions, providing feedback and direction, similar to a car rally team.

Pair programming aims to enhance code and design quality, guided by the principle that two heads are better than one. It also aids in spreading knowledge about the code within a team, preventing silos where only one developer understands a specific part of the code. The practice also provides valuable training opportunities for novice developers, offering exposure to various programming technologies, algorithms, data structures, design patterns and principles, and testing and debugging techniques.

However, employing two developers per task has economic implications, as often noted by pair programming critics. Furthermore, some developers find it emotionally and cognitively uncomfortable to discuss each line of code and implementation decision with a colleague. To mitigate this, XP recommends rotating pairs at every session. These sessions are typically 50 minutes long, followed by a 10-minute break. In the subsequent session, both pairs and roles (driver vs. navigator) switch.

Real World: In 2008, Microsoft Research’s Andrew Begel and Nachiappan Nagappan surveyed 106 company developers about their experiences with pair programming (link). Approximately 65% of the respondents reported finding pair programming beneficial. When asked about the advantages, the most cited were reduced bugs (62%), better code quality (45%), knowledge spread (40%), and peer learning opportunities (40%). However, 75% of respondents considered the associated cost a significant disadvantage. Interestingly, the majority (38%) preferred to pair with colleagues who have complementary skills to compensate for their own weaknesses.

In recent years, the practice of code review has been embraced by many software companies. This practice involves another developer checking the code asynchronously, eliminating the need for two developers to work simultaneously on the same code.

Collective Code Ownership. This practice allows any developer or pair of developers can modify any part of the code, whether to implement a new feature, fix a bug, or perform a refactoring, without needing approval from the original developer or the last person who maintained it.

Automated Testing. This programming practice, advocated by XP, has achieved significant success. The rationale is that manual testing, where a tester runs the program, provides inputs, and checks the outputs, is a costly and non-reproducible procedure. As an alternative, XP proposes implementing programs, called automated tests, that call small code units, such as methods, and verify if the results match the expected outcomes. The advent of automated testing coincided with the development of the first unit testing frameworks, including JUnit (developed by Kent Beck and Erich Gamma in 1997), which played a key role in promoting the practice. A more detailed discussion about automated tests is reserved for Chapter 8.

Test-Driven Development (TDD). TDD, another innovative programming practice proposed by XP, advocates writing a test for every method before implementing the method itself. This practice serves two main purposes: (1) it ensures developers do not forget to write tests by implementing them first, and (2) by writing tests first, developers adopt the perspective of the method’s user. For example, they must think about the method’s interface and its usage before starting the actual implementation. This encourages the creation of user-friendly methods. A more detailed discussion of TDD will be presented in Chapter 8.

Automated Build. This practice involves the automated generation of an executable system that is ready for production. Several tools can facilitate this process, including the Make system, part of Unix distributions since the 1970s, and newer tools like Ant, Maven, and Gradle. XP emphasizes two key concepts regarding the build process. First, the process should be fully automated to eliminate manual intervention by developers. Second, the process should be as quick as possible to provide developers with immediate feedback on potential issues, such as compilation errors or failing tests.

Continuous Integration. As we discussed in Chapter 1, version control systems (VCS) such as Git are widely used nowadays. A VCS stores the source code and related files, including configuration files, images, and documentation. To work on a task, developers pull the source code to their local machines, make modifications, and then push the updated code back to the VCS repository. This step of uploading and merging the code is referred to as integration.

However, integration is subject to conflicts when multiple developers change the same lines of code. In such cases, the VCS prevents the integration from proceeding. For example, if Developer A assigns a value of 10 to a variable x, while Developer B, working independently, assigns x a value of 20, a conflict arises. To resolve this conflict, A and B must agree on the correct value for x. While this is a simple example, conflicts often involve larger sections of code and more developers, making them time-consuming and challenging to resolve and leading to situations known as integration hell.

To minimize integration conflicts, XP encourages developers to integrate their code frequently—daily, if possible—under a practice known as continuous integration. The goal is to reduce both the number of conflicts and the time developers spend resolving them.

To ensure the quality of integrated code, a continuous integration service should be configured. Before carrying out any integration, this service builds the code and executes the tests. Its purpose is to verify that the new code has no compilation errors and passes all tests. Examples of these services include GitHub Actions, Jenkins, and TravisCI. Chapter 10 will provide a detailed study of Continuous Integration.

Real World: In 2010, Laurie Williams, a professor at the University of North Carolina in the USA, surveyed 326 developers about their experiences with agile methodologies (link). Participants ranked the importance of various agile practices on a scale of 1 to 5, with 5 denoting essential practices. Three practices tied for first place with an average score of 4.5 and a standard deviation of 0.8: continuous integration, short iterations (less than 30 days), and definition of done criteria (i.e., the criteria used to consider the implementation of user stories finished). In contrast, planning poker and pair programming were ranked lower, with average scores of 3.1 and 3.3, respectively.

2.3.5 Project Management Practices 🔗

Working Environment. XP advocates for the development of software by small, dedicated teams, ideally composed of fewer than 10 developers. It also discourages situations where team members split their time between multiple projects during the week.

XP also emphasizes the importance of maintaining sustainable work schedules. It’s well known that software companies often demand long working hours, extensive overtime, and weekend commitments. However, XP challenges this practice, advocating instead for sustainable work schedules, with around 40 hours per week, even before important deliveries. Notably, XP was proposed by seasoned professionals with experience in managing real-world software projects, who had firsthand experience of the effects of prolonged working hours. Long hours can be detrimental to developers’ physical and mental health and can also lead to high turnover, with developers frequently considering changing jobs.

Negotiated Scope Contracts. When outsourcing development, two types of contracts are possible: negotiated scope and fixed-scope contracts. Fixed-scope contracts require the hiring company to provide a specification of the requirements. Based on that, the outsourced company establishes a cost and a delivery date. XP questions these contracts due to the volatile nature of software requirements and the customer’s inability to define them in advance. Consequently, the outsourced company may deliver a system with several quality issues, including bugs, solely to avoid penalties.

In contrast, negotiated scope contracts (or open scope contracts) are based on a pay-per-hour model. For instance, the contract might prescribe the allocation of a full-time development team from the outsourced company, which will work in accordance with XP practices. The customer proposes the stories and validates their implementation at the end of each iteration. These contracts also offer the flexibility for termination or renewal every few months, enabling customers to switch companies whenever they find the quality unsatisfactory. As is typical in XP, the objective is to foster communication and feedback streams between customers and suppliers, rather than compelling the latter to deliver a low-quality product just to fulfill the contract. In conclusion, negotiated scope contracts align better with the Agile Manifesto, which recommends customer collaboration over contract negotiation.

Process Metrics. To efficiently track an XP project, managers are encouraged to use two key metrics: the number of bugs detected in production (ideally a few per year) and the time interval between project start and the project’s first financial returns (ideally within one year).

2.4 Scrum 🔗

Scrum is an agile, iterative, and incremental method for project management. It was first proposed by Jeff Sutherland and Ken Schwaber in 1995 (link). Among various agile methods, Scrum is arguably the most popular and broadly applied one. This widespread acceptance can be partly explained by the proliferation of related books, educational courses, professional consultancies, and certifications.

To better understand Scrum, let’s address a frequently asked question: How do Scrum and XP differ? While there are a number of minor variations, the key difference lies in their purpose:

• XP is an agile method specifically designed for software development projects. As such, it introduces a set of programming practices, such as unit testing, pair programming, continuous integration, and incremental design, which we have explored in the previous section.

• Scrum, however, is an agile method that isn’t confined to software projects. For example, the creation of this book, as we will explain shortly, was based on Scrum concepts. Given its broader scope compared to XP, Scrum doesn’t specify any programming practices.

Furthermore, among the agile methods, Scrum stands out as the most structured, including a well-defined set of roles, artifacts, and events, which are listed below. In the following sections, we will explain each of them in detail.

• Roles: Product Owner, Scrum Master, Developer.

• Artifacts: Product Backlog, Sprint Backlog, Scrum Board, Burndown Chart.

• Events: Sprint Planning, Sprint, Daily Meetings, Sprint Review, Retrospective.

2.4.1 Roles 🔗

A Scrum team is composed of a Product Owner, a Scrum Master, and developers. The Scrum Guide 2020 states that the Scrum team is small enough to remain nimble and large enough to complete significant work within a Sprint, typically 10 or fewer people.

The Product Owner (PO) is equivalent to the customer representative in XP. This role requires a clear vision for the product being developed, with the aim of maximizing the return on investment for the project. The Product Owner’s responsibilities include writing user stories, and
they should always be available to address the team’s questions.

The role of the Scrum Master is unique to the Scrum method. They serve as the team’s Scrum expert, responsible for ensuring adherence to the Scrum framework. The Scrum Master continuously trains and educates the team members on Scrum principles and acts as a facilitator to remove obstacles. For instance, if a team encounters issues with faulty hardware, it is the Scrum Master’s role to ensure this issue is resolved promptly to avoid hindering the team’s progress. However, the Scrum Master is not a traditional project manager. They do not, for example, lead the team, as Scrum teams are self-organizing.

Scrum teams are typically described as cross-functional or multidisciplinary, meaning they should have all necessary specialists for developing the product, thereby avoiding dependencies on external members. In software projects, this includes specialists such as front-end developers, back-end developers, database specialists, and UI designers. These team members are responsible for the technical decisions during the project, including defining the programming language, architecture, and frameworks to be used in the implementation. Additionally, they are tasked with estimating the size of the user stories proposed by the Product Owner, using units such as story points, a concept similar to what we studied in Extreme Programming.

2.4.2 Artifacts and Events 🔗

Scrum defines two key artifacts: the Product Backlog and the Sprint Backlog. The key events include Sprints and Sprint Planning. Next, we will detail these artifacts and events.

• The Product Backlog is a prioritized list of user stories (along with other relevant work items). As in XP, the Product Owner is responsible for proposing these stories, which are brief descriptions of the features the system should have. Importantly, the Product Backlog is not static. It requires constant updates to capture changes to requirements or to the product vision. For instance, as the project advances, new feature ideas may emerge while others may no longer hold relevance. These changes are the responsibility of the Product Owner—a role which derives its title precisely from this task of maintaining the Product Backlog

• Sprint refers to a time-boxed iteration in Scrum. Like all agile methods, Scrum employs an iterative approach, organizing the development into sprints, each lasting up to a month. Every sprint should yield a product increment that has tangible value for the customer, called a potentially shippable product. The term potential indicates that deployment the product to production is not mandatory, as explained in Section 2.2.

• Sprint Planning is a preparatory meeting that serves as the launchpad for every sprint. During this meeting, the team decides which stories will be incorporated in the forthcoming sprint. The meeting consists of two main parts. First, the Product Owner suggests stories for the upcoming sprint, while the remaining team members evaluate if they have enough velocity (or capacity) to implement them. In the second part, led by the developers, the selected stories are decomposed into tasks and their execution times are estimated. The Product Owner should be present in both parts of the Sprint Planning to answer questions about the selected stories. In some cases, the team may decide to remove a story if it proves to be more complex when broken down into tasks.

• The Sprint Backlog is an artifact generated at the conclusion of the Sprint Planning. It is a list of tasks for the sprint, each with its respective duration. Similar to the Product Backlog, the Sprint Backlog is also a dynamic entity. Tasks may be considered unnecessary or new ones may be created throughout the sprint, and the estimated effort for each task may also be adjusted if needed. However, the sprint goal—the stories selected by the Product Owner for the sprint and that the development team has committed to implement—should remain unchanged. Thus, while Scrum is adaptable to changes, such changes should occur between sprints. During a sprint, the development team should have the stability of working on a fixed list of stories.

When the sprint planning concludes, the sprint begins and the team starts working on the tasks in the sprint backlog. It’s important to note that Scrum teams are not only cross-functional but also self-organizing, meaning they have the autonomy to decide how and who will implement the selected stories.

To complement the sprint backlog, teams can also use a board with columns denoting individual story stages, such as to do, in progress, testing, and done. This board, known as a Scrum Board, provides a visual aid for tracking the daily progress of the sprint. See the next figure for an example.

Defining the criteria for marking a story or task as done is a crucial decision in Scrum projects. The team should reach a consensus on these criteria, and every member should follow them. For instance, the team might define a story as completed only if all its unit tests pass. A code review by another team member could also be required. Additionally, the code should be successfully merged into the project repository. These criteria help prevent team members from prematurely moving their tasks to the done column with substandard code.

The Burndown Chart is another common artifact in Scrum. It displays the number of work hours remaining to complete the unfinished tasks on each day of the sprint. Specifically, on day x, it shows that the unfinished sprint tasks require y hours of work. Ideally, the burndown chart should show a continuous decline, reaching zero by the end of the sprint (assuming success). Next, we present a burndown chart for a sprint that spans 15 days.

2.4.3 Additional Scrum Events 🔗

Next, we’ll examine three additional Scrum events: Daily Stand-Up, Sprint Review, and Retrospective.

Scrum recommends holding Daily Stand-Up meetings, typically lasting 15 minutes and involving all team members. As the name suggests, these meetings are usually held with participants standing up, leading to alternative names such as Stand-Up Meetings or Daily Scrum. During these meetings, each team member should address the following three key questions: (1) What tasks were completed the previous day? (2) What tasks are planned for the current day? (3) Are there any significant obstacles hindering their progress? The purpose of these meetings is to facilitate communication and information sharing among team members, and to provide a platform to keep everyone updated on the project’s progress. For instance, during the Daily Stand-Up, two developers may discover they are about to work on the same code section. In this scenario, they might arrange a separate meeting to discuss their planned changes, thereby reducing potential conflicts during the integration of their modifications.

The Sprint Review is designed to showcase the results of a sprint. It involves all team members, as well as any additional stakeholders relevant to the sprint’s goal. During this meeting, the team demonstrates the finished product to the customers. Ideally, the Product Owner should approve all stories implemented during the sprint. However, if any issues are identified with a story, it is returned to the Product Backlog to be reworked in a future sprint. Any story the team was unable to complete are also moved back to the Product Backlog.

The Retrospective marks the conclusion of a sprint. This meeting focuses on reflecting on the completed sprint and identifying potential improvements related to the process, team dynamics, and the tools and practices used. For example, as an outcome of the retrospective, the team might emphasize the importance of punctuality and consistent attendance at Daily Stand-Up meetings, particularly if tardiness has been a recurrent issue. However, the purpose of a retrospective is not to provoke complaints or to foster heated discussions among team members. Rather, critical issues should be addressed separately in other meetings or under the supervision of managers. Following the retrospective, the project progresses to the next sprint.

A key aspect of Scrum events is their fixed duration, also known as the time-box of the event. This term frequently appears in Scrum literature. For example, consider a quote from the official Scrum Guide (2017 version, page 9): The heart of Scrum is a sprint, a time-box of one month or less during which a done, usable, and potentially releasable product increment is created. The purpose of defining time-boxes is to maintain a steady workflow, encourage commitment from the team, and prevent deviation from the sprint goal.

The following table presents the time-boxes associated with Scrum events. When events have a maximum time-box (like sprint planning), this maximum value applies to a one-month sprint. For shorter sprints, the respective time-box should be adjusted accordingly.

2.4.4 Example: Writing a Book using Scrum 🔗

This book was written using various elements of Scrum. For example, at the book’s inception, we planned the chapters, thus establishing a Product Backlog. After that, each chapter was written within a sprint. During the Sprint Planning meetings, we partitioned each chapter into sections, treating these sections as tasks. Subsequently, the writing of each chapter began, marking the start of a sprint. Generally, the sprints were scheduled to last two months. Below, we show the backlog of the current sprint, whose goal is to write Chapter 2. We can also see the status of each task/section.

The author decided on an agile approach when writing this book to minimize risks associated with developing a product that doesn’t meet the needs of our primary audience—students and instructors of Software Engineering courses, mainly at the undergraduate level. As such, at the end of each sprint, a chapter was shared and announced publicly, with the aim of receiving feedback. Our objective was to avoid a Waterfall approach, which would involve writing the book over years without receiving any input.

To conclude, let’s examine the criteria used for considering a chapter as done. This process involved the author thoroughly reading and reviewing the chapter. Once this review was completed, the chapter was tentatively released to the members of the Applied Software Engineering Research Group at DCC/UFMG for a second round of review and feedback. Only after incorporating this feedback was the chapter considered final.

2.4.5 Frequently Asked Questions 🔗

To conclude this section, here are answers to some common questions about Scrum:

What does Scrum mean? Scrum is not an acronym but a term borrowed from rugby, where it refers to a field meeting where rugby teams compete for ball possession after an accidental infringement.

What is a squad? A squad is another term for an agile or a Scrum team. The term was popularized by Spotify. Similar to Scrum teams, squads are small, cross-functional, and self-organizing units. A tribe typically refers to a group of squads.

Can the Product Owner role be assigned to a committee? In other words, can a Scrum team have more than one Product Owner? The answer is no. This role must be assigned to a single individual to avoid committee-based decision-making, which often leads to products overloaded with features, implemented mainly to satisfy specific committee members. However, the Product Owner can and should act as a link between the team and other experts in the product being developed. This facilitation is expected, as some requirements might be specific to the domain knowledge of certain individuals within the organization.

Is the Scrum Master considered a full-time role? Ideally, a Scrum Master should be committed on a full-time basis. However, for mature teams with vast experience in Scrum, a full-time Scrum Master might not be necessary. In such scenarios, two alternatives can be considered: (1) Allow the Scrum Master to operate in multiple teams, or (2) Delegate the Scrum Master role to a team member. If the latter approach is used, the chosen member should not simultaneously act as the Product Owner. This is because one of the responsibilities of a Scrum Master is to support the PO in writing and prioritizing the user stories.

Does a Scrum Master need a degree in computer science? No. A degree in computer science is not a prerequisite, as a Scrum Master’s role primarily involves removing obstacles and ensuring the team follows the Scrum principles. Thus, they are not expected to resolve technical issues like debugging, using frameworks correctly, or implementing features. However, some companies may require Scrum Master certifications.

In addition to stories, what other items can be included in the Product Backlog? In fact, the Product Backlog is not limited to user stories—it can also comprise other work items like complex bugs and major refactoring efforts, for example.

Do Scrum organizations employ managers? Yes. Although Scrum teams have autonomy to implement the stories defined by the Product Owner, additional decisions typically associated with managerial levels are required in a software development project. Examples include:

• Team composition: Managers are responsible for allocating developers to teams. In most software organizations, developers typically do not have the freedom to choose their own teams.

• Definition of goals and responsibilities: Managers designate priorities and goals for each team, including the systems they will work on. For example, a team doesn’t decide by itself that a new accounting system is needed and then start its development. Such strategic decisions are the responsibility of the organization’s managers and executives.

• Human resources management: Managers also handle human resource tasks, such as recruiting new employees, dismissals, promotions, and training initiatives.

• Performance evaluation: Managers are also tasked with assessing whether the Scrum teams are providing value to the organization.

2.5 Kanban 🔗

The Japanese term kanban literally translates to visual card. Since the 1950s, the term has also been associated with the just-in-time production process of Japanese factories, particularly at Toyota where it was initially implemented. This process is also recognized as the Toyota Production System (TPS) or, in more contemporary terms, as lean manufacturing.

In the context of software development, Kanban was first introduced at Microsoft in 2004 by David Anderson (link). Anderson describes Kanban as a method that promotes a sustainable work pace for development teams by eliminating waste, delivering consistent value, and fostering a culture of continuous improvement.

To better understand Kanban, we will compare it with Scrum. Primarily, Kanban is simpler than Scrum because it does not prescribe events like sprints. Furthermore, strictly speaking, Kanban does not define particular roles like Product Owner or Scrum Master. Lastly, unlike Scrum, Kanban doesn’t include several artifacts, with the exception of one main artifact: the task board. In Kanban, this artifact is referred to as the Kanban Board, which also includes the product backlog.

The Kanban Board is organized into specific columns, as described below:

• The first column represents the product backlog, closely mirroring the same artifact in Scrum. User stories are proposed and subsequently added to this backlog.

• The next columns represent the steps required to transition a user story into a functioning feature. These columns typically have labels such as Specification, Implementation, and Code Review. The stories from the backlog are processed sequentially, mirroring the workflow of an assembly line, progressing from left to right. Each column is further sectioned into two subcolumns labeled in progress and ready. For instance, the Implementation column contains two subcolumns: tasks under implementation and tasks that have been implemented. Completed tasks from one step await being pulled into the next step by a team member. This is why Kanban is commonly referred to as a pull system.

Let’s examine the structure of a Kanban Board using an example.

In this board, we can see stories and tasks at various stages. In the backlog, we have one story labeled H3. Another story, labeled as H2, has been picked up by a team member and is under Specification. Four tasks from a previous story, labeled as T6 to T9, have already been specified. Next, there are two tasks, T4 and T5, which are being implemented. Task T3 has been implemented and is now awaiting Code Review. Lastly, Task T2 is currently under review, while T1 has been fully implemented and is now considered done. You may also have noticed the acronym WIP appearing at all steps, with the exception of the backlog. Don’t worry about this for now; we will explain it shortly. It’s worth noting that, in this example, we used the letters H and T to represent stories and tasks, respectively. However, in real-world Kanban boards, tasks and stories are represented as cards that carry a brief description of each item.

Days later, the Kanban Board evolved into the following state (tasks that moved forward are distinguished by being underlined and in red).

Note that story H2 is no longer visible, as it was decomposed into three tasks (T10, T11, and T12). This decomposition is precisely what occurs during the specification phase, whose objective is to convert a story into an actionable task list. Furthermore, T6 and T7—which were previously on hold—are now under Implementation. Meanwhile, T3 has transitioned into Code Review. Finally, the review of T2 has been completed. It’s also important to note that currently there are no tasks waiting to enter the review stage.

Similar to other agile methods, Kanban teams are self-organizing. This means they have the autonomy to decide which tasks they will work on. Additionally, they are cross-functional—meaning the team consists of members who collectively have the skills to complete all the tasks on the Kanban Board.

We will now discuss the concept of WIP Limits (short for Work in Progress). Project management methods typically strive to maintain a sustainable workflow. To achieve this, two scenarios need to be avoided:

1. Teams having no tasks to work on and consequently staying idle most of the time.
2. Teams being overwhelmed with an excess of tasks, thus negatively affecting the quality of the software product.

To prevent the second scenario, Kanban proposes setting maximum task limits for each step on a board. This cap, referred to as the WIP Limit, represents the maximum number of tasks—or cards—that can be present in each step, including the tasks in progress (first subcolumn) and those that are ready (second subcolumn). The exception is the final step, where the WIP limit applies only to the first subcolumn, as it is meaningless to limit the number of completed tasks.

Next, we show a Kanban Board that incorporates these WIP limits. We can see the limits below each step’s name. Our board allows a maximum of 2 stories in Specification, 5 tasks in Implementation, and 3 tasks in Code Review. The reason for the particular WIP limit in Specification will be explained later. Currently, note that 4 tasks are under Implementation, thus remaining below the step’s WIP limit of 5. Similarly, in Code Review, the limit is also observed as only one task is under review. Remember that to comply with the WIP limit, tasks from both subcolumns— in progress and ready tasks—are taken into account, except for the final step, where only tasks from the first subcolumn count (T3, in this example).

We will now explain the concept of Work in Progress for the Specification step. To compute the WIP for this step, we need to sum up the stories in Specification (which is zero in our previous example) with the stories that have already been specified. For example, in the previous board, we’ve specified two stories: tasks T8 and T9 are products of the same story’s specification, while tasks T10, T11, and T12 resulted from the specification of another story. Therefore, in this step, for the purpose of WIP computation, we have two stories, which is equal to the limit of 2. To ease visualization, we display the tasks emerging from the same story specification on a single line. Hence, to compute the WIP of the Specification step, we should add the number of stories in the first subcolumn (zero in our case) to the number of lines in the second subcolumn (two in this case).

Regarding the previous board, we have:

• Story H3, in the Backlog, can’t be moved into Specification because this step’s WIP has reached its maximum limit.

• One of the tasks in Specification (T8 to T12) can be moved to Implementation as the WIP for this step is currently 4, with a limit of 5.

• One or more tasks in Implementation (T4 to T7) can be moved to Ready, which doesn’t affect that step’s WIP.

• The reviewing of T3 can be finalized.

An important point to note here is that WIP limits are in place to avoid overloading the team with tasks. When a team is burdened with numerous tasks—that bypass the WIP limits—the probability of producing low-quality software increases. Kanban understands this issue and provides an automatic lock to prevent teams from accepting tasks beyond their delivery capacity. These locks, represented by the WIP limits, have internal and external uses. For example, they serve as an instrument for the team to decline extra tasks that may be pushed by top management.

2.5.1 Determining WIP Limits 🔗

In this section, we will explain the steps involved in defining Work in Progress (WIP) limits. While several methods exist, we will adapt an algorithm proposed by Eric Brechner, a Microsoft engineer. This algorithm is detailed in his book on using Kanban in software development (link). Next, we outline the algorithm’s steps.

Initially, we need to estimate the average time it takes to complete a task in each step of the Kanban Board, known as the lead time (LT) of the step. For our example, we assume these values:

• LT(specification) = 5 days
• LT(implementation) = 12 days
• LT(review) = 6 days

These estimated times reflect an average task’s duration, taking into account that there will be both complex and simple tasks. Note that the lead time includes the queue time, i.e., the task’s waiting time in the ready subcolumns of the Kanban Board before progressing to the next step.

Next, we should estimate the throughput (TP) of the step with the longest lead time. In our example, this step is Implementation (which is typical for most software development projects). Assuming the team can implement eight tasks per month, this step’s throughput is: 8 / 21 = 0.38 tasks/day (we’ve assumed that a month has 21 workdays).

Lastly, each step’s WIP is calculated as:

WIP(step) = TP * LT(step)

Here, TP refers to the throughput of the slowest step, as calculated in the previous step. Therefore, using this formula, we derive:

• WIP(specification) = 0.38 * 5 = 1.9
• WIP(implementation) = 0.38 * 12 = 4.57
• WIP(review) = 0.38 * 6 = 2.29

After rounding, we get:

• WIP(specification) = 2
• WIP(implementation) = 5
• WIP(review) = 3

Eric Brechner also suggests adding a 50% margin of error to the estimated WIPs to account for variations in task size and tasks blocked due to external factors. However, we won’t apply this adjustment in our example, as it is purely illustrative.

It’s important to remember that the purpose of WIP limits, as proposed by Kanban, is to maintain a sustainable work pace and thereby deliver quality software. These limits prevent developers from becoming overburdened with tasks, which would reduce their work quality. Virtually all development methods offer such mechanisms. For instance, Scrum uses sprints with a fixed time-box to avoid overcommitting and to limit the workload based on the team’s velocity. Once started, the sprint’s goals can’t be changed, thus shielding the team from priority shifts. In Waterfall, the strategies to ensure a sustainable workflow involve a comprehensive phase of requirement specifications. This phase aims to provide developers with a clear understanding of the system they will implement.

2.5.2 Little’s Law 🔗

The procedure used for calculating WIP limits is considered an application of Little’s Law, a prominent result from Queueing Theory (link). Little’s Law states that the number of items in a queueing system is the product of the arrival rate of these items and the time each item spends within the system.

When applied to our context, where the items we refer to are tasks, the law can be broken down as follows:

• WIP: Number of tasks in a particular step of the Kanban Board.

• Throughput (TP): Rate at which tasks arrive at this step.

• Lead Time (LT): Time each task spends at this step.

Thus, according to Little’s Law, WIP = TP * LT. The following figure illustrates Little’s Law.

2.5.3 Frequently Asked Questions 🔗

To conclude this section, let’s answer some common questions related to Kanban:

What roles exist in Kanban? In contrast to Scrum, Kanban doesn’t prescribe a rigid set of roles. The responsibility rests on the team and the organization to determine the necessary roles, such as a Product Owner (PO), testers, and so on.

How are user stories prioritized? Kanban’s approach is lightweight compared to Scrum and even XP. One reason is the absence of explicitly defined criteria for prioritizing user stories. As mentioned in the previous answer, even the existence of a Product Owner—responsible for prioritization—isn’t mandatory. Thus, alternative solutions, such as external prioritization by a manager, can also be used.

Can Kanban teams conduct Scrum events, such as daily meetings, reviews, and retrospectives? Yes, they can. While Kanban does not prescribe these events, it doesn’t explicitly prohibit them either. The decision regarding which events are essential, as well as their timing and duration, is left to the team.

2.6 When Not to Use Agile Methods? 🔗

Although agile methods are very popular today, it’s important to remember that there’s no one-size-fits-all solution in software engineering. As such, this section examines scenarios and domains where agile practices may not be the most suitable choice.

However, the question posed in this section cannot be answered simply by stating that systems from areas X, Y, and Z should avoid agile methods, while others should adopt them. In reality, systems from all areas could potentially benefit from at least some agile practices. Conversely, some practices might not be advisable for specific types of systems, organizations, and contexts. Therefore, we will address the proposed question at a finer granularity. That is, we will comment on when certain agile practices should not be used.

• Incremental Design: This approach is effective when the team has an initial understanding of the system’s design. However, if the team lacks this vision, or the domain of the system is new and complex, or if future changes have high costs, it may be better to have a design phase prior to starting the implementation of features.

• User Stories: As a lightweight method for specifying requirements, stories are typically clarified with customer participation during the sprints. However, certain projects may require a detailed requirements specification phase at the beginning, particularly for domains unfamiliar to the developers.

• Customer Involvement: If the requirements are stable and thoroughly understood by the developers, constant involvement of a Customer Representative or Product Owner may be unnecessary. For instance, when developing a compiler for a language with well-established grammar and semantics, this role may not be as important.

• Lightweight Documentation: In specific domains, detailed requirements and comprehensive documentation are important. For example, we can mention systems where failures can result in human fatalities, such as systems in the medical and transportation sectors. These systems often require certification from an external entity, which may mandate the availability of extensive documentation.

• Self-organizing Teams: Agile teams have the autonomy to work without interference during a sprint. For example, they do not need to report daily to the organization’s managers. However, this autonomy can conflict with the values and culture of some organizations, particularly those with a hierarchical structure and rigid controls.

• Negotiated Scope Contracts: In such contracts, remuneration is defined on an hourly basis. Thus, organizations may hesitate to sign contracts with an open scope, especially if they lack experience with agile development or harbor reservations about the contracted company.

To conclude, it’s important to highlight that two agile practices have become prevalent in a majority of software projects:

• Small teams, because communication and coordination challenges increase significantly for large teams.

• Iterations (or sprints), even if they are longer than typical agile sprints. For instance, iterations may last two or three months, as opposed to the usual sprint time-box of 15 to 30 days. Interestingly, even before the rise of agile methods, some iterative methods were suggested, i.e., methods with validation points throughout the project. We will explore two of these methods in the next section.

2.7 Other Iterative Methods 🔗

The shift from the Waterfall model—highly prevalent during the 1970s and 1980s—to Agile approaches, which gained significant traction in the late 2000s, was a gradual process. During this transition, some methods emerged that incorporated the concept of iterations, although with a longer duration than those typically seen nowadays in Agile. Rather than lasting a few weeks, these iterations could extend over months. Furthermore, these transition methods also retained many elements from Waterfall, like the focus on documentation and an initial requirements specification phase followed by a design phase.

One such method formulated during this transition period is the Spiral Model, proposed by Barry Boehm in 1986 (link). In this model, a system is developed through a spiral of iterations. Each iteration, which completes a full round of the spiral, comprises four stages, as follows (see also the following figure):

1. Defining objectives and constraints, including costs, schedules, etc.

2. Evaluating alternatives and conducting risk analysis. For instance, considering whether it is more cost-effective to purchase a system instead of developing it in-house.

3. Development and testing, potentially using the Waterfall model. By the end of this stage, a prototype should be produced for demonstration to users.

4. Planning for the next iteration or deciding to conclude the project if the current implementation already meets the organization’s needs.

The Spiral Model generates a more comprehensive version of a system after each iteration. Each iteration, encapsulating all four phases, can take anywhere from 6 to 24 months—a duration significantly longer than in XP and Scrum. Another interesting feature of this model is the risk analysis phase, where concrete steps to mitigate project risks are considered.

The Unified Process (UP), proposed in the late 1990s, is another example of an iterative method. Developed by professionals affiliated with Rational, a software consulting company later acquired by IBM in 2003, UP also has a specific version known as the Rational Unified Process (RUP).

Due to its origins, RUP is associated with two specific technologies:

• Unified Modeling Language (UML), as many of the RUP outcomes are documented using UML diagrams (which will be discussed in detail in Chapter 4). It’s worth noting that both UML and RUP were proposed by the same group of professionals who intended to define a unified process and modeling language.

• CASE (Computer-Aided Software Engineering) tools, which are analogous to CAD (Computer-Aided Design) tools used in other fields such as Civil Engineering, Mechanical Engineering, and Architecture. RUP proposes that the design of software should rely on UML diagrams. However, these diagrams aren’t drawn on paper, but are created using CASE tools, as illustrated in the following figure. As a result, besides proposing the method, Rational was also responsible for developing its supporting tools.

When using RUP, development is divided into the following phases:

• Inception: This phase encompasses tasks such as feasibility analysis, budget definition, risk assessment, and system scope determination. At the end of this phase, the business case for the system should be well established. At this point, we can also conclude that it is more cost-effective to purchase a system off the shelf rather than to build one.

• Elaboration: This phase involves tasks such as requirements specification, software architecture definition, and the proposal of a development plan. Upon completion of this phase, the risks identified during the inception phase should be effectively managed and mitigated.

• Construction: This phase deals with lower-level design, implementation, and testing. Upon its completion, working software, including documentation and manuals, should be available.

• Transition: In this phase, the system is moved to production. This includes defining all necessary deployment procedures, such as backup and recovery policies, data migration from legacy systems, and training for the operations teams.

RUP also defines various engineering disciplines, including business modeling, requirement definition, analysis and design, implementation, testing, and deployment. These disciplines, also referred to as process workflows, can occur in any phase. However, certain disciplines are more prominent during specific phases, as illustrated in the following figure. For instance, business modeling tasks are primarily performed during the initial phases (inception and elaboration). Conversely, most of the implementation occurs during the construction phase.

Bibliography 🔗

Kent Beck, Cynthia Andres. Extreme Programming Explained: Embrace Change. Addison-Wesley, 2nd edition, 2004.

Kent Beck. Embracing Change with Extreme Programming. IEEE Computer, 1999.

Kent Beck, Martin Fowler. Planning Extreme Programming. Addison-Wesley, 2000.

Ken Schwaber, Jeff Sutherland. The Scrum Guide, 2020.

Kenneth Rubin. Essential Scrum: A Practical Guide to the Most Popular Agile Process. Addison-Wesley, 2012.

Eric Brechner. Agile Project Management with Kanban. Microsoft Press, 2015.

David Anderson. Kanban. Blue Hole Press, 2013.

Ian Sommerville. Software Engineering. Pearson, 10th edition, 2019.

Hans van Vliet. Software Engineering: Principles and Practice. Wiley, 3rd edition, 2008.

Armando Fox, David Patterson. Engineering Software as a Service: An Agile Approach Using Cloud Computing. 1st edition, 2014.

Exercises 🔗

1. How are software development contracts structured when adopting XP?

2. Describe the key differences between XP and Scrum.

3. Scrum teams are cross-functional and self-organizing. Define these terms.

4. In Scrum, what is the difference between stories at the top and at the bottom of the Product Backlog?

5. What are story points, and what are they used for?

6. What is the difference between a sprint review and a retrospective?

7. Can a sprint be canceled? If so, by whom and for what reasons? To answer this question, check the Scrum Guide (link).

8. Consider a system within your area of expertise. (a) Propose a user story for this system. (b) Break this story into tasks, similar to what we did with the Q&A forum in the section about XP. (c) Do these tasks have dependencies, or can they be implemented in any order?

9. Suppose two teams, A and B, are working on different projects, for different companies with no interconnections. Both teams follow 15-day sprint cycles and both have five developers. In their respective projects, team A estimates their velocity at 24 story points, while team B estimates theirs at 16 story points. Can we conclude that team A is 50% more productive than team B? Justify your answer.

10. Describe the primary differences between Scrum and Kanban.

11. What are the differences between a Scrum Board and a Kanban Board?

12. What is the error in the following Kanban Board?

13. Consider the following Kanban Board. At this moment, the team cannot work on the specification of new stories, as the WIP limit for this step is filled by items waiting to move to Implementation. What is more recommended at this moment: Bypassing the WIP limit and pulling a new story from the Backlog to Specification, or working on the tasks in Validation?

14. Why is it usually suggested to increase the WIP limits computed using Little’s Law, for example, by 50%? What circumstances can lead to errors when computing the WIP limits?

15. Consider a Kanban Board with four stages. The following table provides the lead time for each stage and the throughput for stage C, which is the slowest one. Using these values, calculate the WIP for each stage.

16. Describe the approaches provided by Waterfall, Scrum, and Kanban for managing risks and promoting the development of high-quality software.

17. Why are RUP and Spiral not categorized as agile methods? How do they differ from Waterfall?

18. Consider the following chart, which illustrates how the costs of changes vary according to the development phase in which they occur for a particular software project. (a) Which process would you recommend for this project, and why? (b) Give examples of systems that might have a similar change cost curve.

19. The article Development and Deployment at Facebook (link) presents the development practices employed at Facebook. In the first section (pages 2-3; figure 2), the authors distinguish between several development methods based on their release frequency. Fill in the following table with the release frequency for the methods and policies discussed in that part of the article.