All rights reserved. Version for personal use only.

This web version is subjected to minor edits. To report errors or typos, use this form.

Home | Dark Mode | Cite

Software Engineering: A Modern Approach

Marco Tulio Valente

2 Processes 🔗

In software development, perfect is a verb, not an adjective. There is no perfect process. There is no perfect design. There are no perfect stories. You can, however, perfect your process, your design, and your stories. – Kent Beck

This chapter begins with a discussion of the importance of software processes (Section 2.1). Next, we explore general and foundational aspects of agile processes (Section 2.2), including a discussion of the historical context that motivated the emergence of these processes. The next sections focus on three agile methods: Extreme Programming (Section 2.3), Scrum (Section 2.4), and Kanban (Section 2.5). Following that, we discuss scenarios where agile methods might not be the most recommended processes (Section 2.6). Lastly, in Section 2.7, we briefly present some traditional processes, such as the Unified Process.

2.1 Introduction 🔗

Car production in an automobile factory follows a well-defined process. In simple terms, sheets of steel are initially cut and pressed to shape doors, roofs, and hoods. Next, the car is painted and components such as the dashboard, seats, seat belts, and wiring are installed. Finally, the mechanical parts, including the engine, suspension, and brakes, are fitted.

Analogously, software production should also follow a process, although less mechanized and more dependent on intellectual endeavor. A software development process defines a series of steps, tasks, events, and practices that developers must observe while developing a software system.

There are those who criticize software processes and question their need, often asking: Why should I follow a process? or What process did Linus Torvalds or Donald Knuth use when implementing the Linux operating system or the TeX text formatter?

In reality, the second question above does not apply to our context, as both Linux (in its initial versions) and TeX were individual projects led by a single developer. In such scenarios, following a process is less important. Stated in better terms, the process followed in those projects was a personal one, reflecting the principles, practices, and experience of a single developer.

However, contemporary software systems are often too complex to be developed by a single developer. Thus, systems implemented by individual developers are increasingly rare nowadays. In practice, modern systems—the focus of this book—are developed by teams.

These software development teams require at least a basic set of guidelines to produce quality software efficiently. That’s why companies invest heavily in development processes. They serve as tools for companies to organize, coordinate, motivate, and evaluate their developers, ensuring productivity and alignment with the organization’s goals. Without a process, even a lightweight and simplified one such as the agile methods we will study in this chapter, teams risk working in an uncoordinated way, which can lead to the creation of products without business value. Finally, processes benefit both the company and developers by clarifying expected tasks and outcomes, thus reducing misalignment among team members.

In this chapter, we explore various software processes. In fact, in Chapter 1, we already presented an introduction to Waterfall and Agile methods. In the following sections, we will continue this discussion and cover three well-known Agile methods.

2.2 Agile Manifesto 🔗

The earliest software development processes, such as the Waterfall process, proposed in the 1970s, followed a sequential approach. Usually, projects began with a requirement specification phase and ended with the implementation, testing, deployment, and maintenance phases.

Given the historical context, the adoption of this strictly sequential approach was understandable. Indeed, projects in traditional engineering fields are developed sequentially and are based on detailed and up-front planning. Therefore, it’s not a surprise that the emerging Software Engineering field mirrored the processes of longer-established domains like Electronics, Civil, Mechanical, and Aeronautical Engineering, among others.

However, by the 1980s, it became increasingly recognized within the industry that software is different from other engineering products. The recurrent problems faced by software projects at this time reinforced this conclusion. For instance, such projects routinely overshot schedules and budgets. Some were even abandoned after years of effort without delivering a functioning system to customers.

The CHAOS Report, published in 1994 by the Standish Group consulting firm, provided more details about the state of software projects at the time. This report revealed that over 55% of projects surpassed their deadlines by 51% to 200%; and at least 12% exceeded the deadlines by more than 200%, as shown in the next figure.

In terms of costs, almost 40% of the studied projects overshot their budgets by 51% to 200%, as indicated in the following figure.

Therefore, in 2001, seventeen software industry professionals gathered in Snowbird, Utah, to discuss and propose an alternative to the prevailing Waterfall-based processes. They argued that, being different from traditional engineering products, software requires a distinct development process.

For instance, software requirements change more frequently than those of other engineering projects, such as airplanes or bridges. Moreover, customers commonly lack a clear understanding of their needs. As a result, there is a constant risk that a product might become obsolete by the time it’s finished, due to changing circumstances or customer needs. The professionals who met in Utah also identified issues with the document-centric approach advocated by Waterfall. The detailed and dense requirements documents common at that time tended to become obsolete quickly, as developers often did not update them to reflect changes in requirements.

In response, the group proposed a new paradigm for software development processes, which was described in a document they named the Agile Manifesto. The manifesto reads:

Through this work, we have come to value:

Individuals and interactions over processes and tools

Working software over comprehensive documentation

Customer collaboration over contract negotiation

Responding to change over following a plan.

Agile processes are characterized by short and iterative development cycles. Systems are built incrementally, starting with the most relevant features, according to customers. Initially, a first version of the system is created, which implements only high-priority functionality. This version is then validated by the customer. If approved, a new cycle, also called iteration or sprint, begins, during which a few more features are added. Typically, these cycles are short, lasting two weeks, for example. This allows the software to be incrementally built, with each feature increment receiving explicit approval by the customer. The development ends when all the customer’s requests have been implemented.

The next two figures provide a comparison between Waterfall and Agile methods:

However, the previous figures may suggest that in agile development, each iteration replicates a mini-Waterfall process, encompassing all the Waterfall phases. This isn’t accurate; generally, iterations in agile methods do not follow a strict sequence of tasks as in Waterfall (more details in the following sections). The figure may also suggest that a system must be put into production at the end of each iteration. This is also not correct. Indeed, the objective is to deliver a functional system that performs useful tasks for the customer. However, the decision to launch the software involves other variables such as business risks, hardware availability, marketing campaigns, user training, and so on.

Other key characteristics of agile processes include:

• Less emphasis on documentation: Only essential information should be documented, focusing on what is necessary for understanding and maintaining the system.

• Less emphasis on detailed plans: At the start of a project, often neither the customer nor the developers have a clear understanding of all requirements that should be implemented. This understanding gradually unfolds as sprints are completed and their outcomes validated. In other words, the essence of agile is to advance even when we have incomplete, partial, and changing requirements.

• No dedicated design phase, i.e., no Big Design Up Front (BDUF): The design of the software system is also incremental, evolving with each iteration.

• Small teams: Teams typically consist of about a dozen developers or, as Amazon CEO Jeff Bezos phrased it, teams that can be fed with two pizzas. We also like to compare the size of agile teams to that of sports teams, ranging from a basketball team (5 members) to a soccer team (11 members).

• Emphasis on innovative programming practices (as in the early 2000s), such as pair programming, automated testing, refactoring, and continuous integration.

Due to these characteristics, agile processes are considered lightweight, with few prescriptive elements and minimal documentation.

However, these characteristics, while important, remain quite generic and broad. Thus, to make agile principles more concrete and actionable, several methods were proposed. Interestingly, they were initially proposed before the 2001 Agile Manifesto’s meeting. Particularly, in this chapter, we will study three agile methods:

• Extreme Programming (XP): Kent Beck proposed this method in a 1999 book (link). A second and heavily revised edition was released in 2004.

• Scrum: This agile method was introduced by Jeffrey Sutherland and Ken Schwaber in 1995 (link).

• Kanban: This method originated from a production control system implemented in Toyota’s factories in the 1950s (link). Over the past years, Kanban has been gradually adapted for software development.

It’s also important to understand that all development methods are essentially a set of recommendations. Every organization should analyze each one and decide if it’s suitable for their particular context. As a result, organizations might find it necessary to adapt existing methods to their needs. In fact, it’s rare to see two organizations that follow exactly the same development process. This applies even to organizations that claim to be using Scrum.

In-Depth: The terms process and methods are often used interchangeably. However, there are also subtle differences between them. In our context, a process is a set of steps, events, and tasks used to construct software. Every organization employs a process to develop their systems, which can be Agile, Waterfall, or even a chaotic one. However, a process always exists. On the other hand, a method specifies a particular process (the term originates from Greek, meaning the means to achieve a goal). Therefore, XP, Scrum, and Kanban are agile methods or, put another way, they define practices, activities, events, and techniques compatible with agile principles.

Real World: The widespread success and impact of agile methods are noteworthy. Currently, an overwhelming majority of organizations, irrespective of their size or business domain, follow agile principles to varying degrees. Here’s an interesting statistic: in 2018, the Stack Overflow survey included a question about the most used development methods (link). Over 57 thousand professional developers responded, and a majority identified with agile methods or practices. This includes methodologies we will study in this chapter—such as Scrum (63% responses), Kanban (36%), and Extreme Programming (16%). Only 15% of participants answered Waterfall as their development method.

2.3 Extreme Programming 🔗

Extreme Programming (XP) is considered a lightweight method for developing software with ever-changing or unclear requirements, such as the Type B systems (Business), according to the classification described in Chapter 1. As any agile method, XP embodies characteristics like short iterative development cycles, less emphasis on comprehensive documentation, incremental design, and development in small teams.

Thus, XP is not a prescriptive method that outlines a step-by-step plan for software construction. Instead, XP is defined by a set of values, principles, and practices. The values and principles are important to shape the culture of software development teams. For this reason, they are key components of the method and they give meaning to the practices proposed by XP. To be clear, if an organization is unprepared to embrace the XP mindset, as represented by its values and principles, it probably should not invest time in adopting the practices.

In this chapter, we will begin by presenting the values and principles defended by XP. Here is a list of them:

• Values: Communication, simplicity, feedback, courage, respect, and quality of life.

• Principles: Humanity, economics, mutual benefits, continuous improvements, acknowledging failures, baby steps, and personal responsibility.

Next, we will explain the practices, which are organized into three groups: process practices, programming practices, and project management practices. Here is a list of the practices within each group:

• Process Practices: Customer representative, user stories, iterations, releases, release planning, iteration planning, planning poker, and slacks.

• Programming Practices: Incremental design, pair programming, test-driven development (TDD), automated builds, continuous integration.

• Project Management Practices: Metrics, working environment, negotiated scope contracts.

2.3.1 Values 🔗

XP advocates that software projects should be primarily guided by three main values: communication, simplicity, and feedback. Indeed, these values are crucial in any effort involving human collaboration. Effective communication is important not only to avoid mistakes but also to learn from them. The second value XP emphasizes is simplicity, as within every complex and challenging software system there is a simpler system that is sometimes overlooked. Lastly, continuous feedback from stakeholders serves to counterbalance risks such as changing requirements and evolving programming technologies.

In fact, it is difficult to develop the right software at the first attempt. Frederick Brooks has a well-known recommendation regarding this problem:

Plan to throw away parts of your system, because you will.

Thus, feedback is an essential value to ensure that the software parts that are going to be discarded are identified as early as possible, to reduce waste and rework. Besides, XP also defends other values such as courage, respect, and quality of life.

2.3.2 Principles 🔗

While XP’s values are abstract, the practices are concrete and pragmatic. To bridge this gap, XP encourages following a set of principles. Picture it as a river with the values on one side and the practices on the other. The principles act as a bridge between both sides. The primary principles of XP are:

Humanity. Software development is an activity that heavily depends on human capital. The principal resource of software companies is not their physical assets, such as computers, buildings, chairs, or Internet connections, but rather their developers. A term that nicely reflects the importance of this principle is peopleware, which was coined by Tom DeMarco in a book with the same title (link). The idea is that managing people, including their expectations, growth, motivation, and responsibilities, is essential for the success of software projects.

Economics: Software development requires a substantial financial investment. Hence, delivering economic results is also important, particularly for Type B (Business) systems, according to the classification we studied in Chapter 1.

Mutual Benefits: XP advocates for decision-making in a project to benefit multiple stakeholders. For instance, a software company should ensure a good work environment and in return the team should deliver software that adds value to the business. Another example: by writing tests a developer benefits, as tests help to detect bugs in their code. But tests also benefit other developers by providing confidence that their code does not introduce regressions, meaning bugs in previously working code. A final example: refactoring makes code cleaner and easier to understand for both the original developer and any future maintainers.

Continuous Improvements: As expressed in the opening quote by Kent Beck, no software development process is perfect. Therefore, we should aim for a process that is under continuous improvement, at each iteration, with feedback from customers and team members. For this reason, XP advises against investing a large amount of time in a complete initial design. Instead, the design of the software is also incremental, improving with each iteration. Finally, the programming practices themselves can be subject to improvement; the team should allocate time to discuss and reflect on them.

Failures Happen: Software development is not a risk-free endeavor. As mentioned in Chapter 1, software represents one of the most complex creations of humans. Thus, failures are expected in software projects, including major bugs, features that do not attract users, and issues with non-functional requirements such as performance, privacy, and availability. Evidently, XP does not advocate covering up these failures. However, they should not be used to punish developers or other team members. On the contrary, facing failures is part of the process if a team aims to deliver timely solutions to problems through software.

Baby Steps: Progress that is secure, tested, and validated, even if small, is better than large software increments with risks of being discarded by the users. The same applies to tests (which are useful even when the units under test are small, such as individual methods), code integration (it’s better to integrate daily, than to go through the stress of a major integration after weeks of work) and refactorings (which should occur in small steps intercalated with running the tests). In summary, the focus should be on continuous improvements, no matter how small, as long as they move the project in the right direction.

Personal Responsibility (or Accepted Responsibility): According to this principle, it is essential for developers to have a clear understanding of their role and responsibility in the team. The reason is that responsibility cannot be transferred without the other party accepting it. For a similar reason, XP advocates that the developer who implements a user story—i.e., a small feature increment—should also be also the one who will test and maintain it.

Real World: One of the first systems to be developed using XP was a payroll system for the car manufacturer Chrysler, called Chrysler Comprehensive Compensation (C3) system (link). The project began in early 1995 but failed to deliver concrete results, leading to its restart the following year under the leadership of Kent Beck. Another well-known member of the agile community, Martin Fowler, participated in the project as a consultant. While implementing the C3 system, many facets of the soon-to-be-named XP method were defined and experimented with.

2.3.3 Process Practices 🔗

Similar to other agile methodologies, XP recommends direct customer involvement in the project. For example, teams should include one customer representative who is an expert on the domain of the system under development. This member is responsible for proposing and writing the user stories, which are XP’s version of requirement specification documents. However, user stories are rather short and lightweight documents, comprising two to three sentences, in which the customer representative describes the features the system should implement.

We will examine user stories in more detail in Chapter 3. For now, it should be noted that they are typically written on cards, either by hand or using project management tools. As previously mentioned, they are simple documents that focus on the key features of the system from a user perspective, in contrast to the complex requirement documents that are typically used in Waterfall-based approaches.

Here is an example of a user story for a Question & Answer forum application, similar to Stack Overflow, which we will use to present XP in this chapter.

Post Question

A logged-in user should be able to post questions. Since it’s a programming forum, questions may incorporate code blocks, which must be presented in a differentiated layout.

This story has a title (Post Question) and a brief description in two sentences. These stories serve as reminders for subsequent detailed verbal specifications by the customer representative. In other words, developers can at any time consult the customer representative to clarify issues about user stories.

After being created by the customer representative, developers should estimate the effort required to implement the stories. This estimation is often performed using story points, rather than hours or person-days. In these cases, an integer scale is used to assign a certain number of story points to stories. The simplest stories are assigned one story point. If a story is deemed twice as complex, it is assigned two story points, and so on. In fact, story points are commonly determined using a Fibonacci sequence, such as 1, 2, 3, 5, 8, 13. The aim is to use a scale that progressively reflects the increasing difficulty of the stories and also enables the team to make comparisons similar to the following one: in terms of effort, is a task with 8 story points equivalent to a task with 5 story points plus a task with 3 points (i.e., the two preceding values in the scale)? If yes, 8 story points is a good estimate. If not, it is better to estimate the story as having 5 story points.

In-Depth: One technique for estimating story points is called Planning Poker. Here is how it works: the customer representative selects a story, reads it aloud to the developers, who can then discuss it with the representative to clarify any doubts. Afterwards, each developer independently estimates the story’s size in story points, and then all participants simultaneously reveal their estimates by using previously distributed cards with numbers such as 1, 2, 3, 5, etc. If there is a consensus, the team moves on to the next story. If not, a discussion and further voting occur until a common ground is found.

The user stories are implemented in iterations, which have a fixed duration, usually ranging from one to three weeks. These iterations are also organized into longer cycles, referred to as releases, which typically last two to three months. A team’s velocity refers to the number of story points it can implement within an iteration. The customer representative is expected to propose and create stories that take at least one release to implement. Therefore, in XP, the planning horizon is one release, which may span few months.

Note: The term release in XP differs from its usage in configuration management, where it refers to a version of a system that is made available for use. However, the version produced at the end of an iteration or release in XP does not necessarily have to be put into production.

To summarize, to initiate the use of XP, an organization needs to:

• Define the duration of each iteration.
• Define the number of iterations within a release.
• Set a list of user stories, as proposed by the customer representative.
• Obtain estimates for each story from the developers.
• Define the team’s velocity, i.e., the number of story points that the team can implement per iteration.

After defining the above aspects, the customer representative must prioritize the stories. To achieve this, the representative must select the stories to be implemented in each iteration of the first release. In this process, the team’s velocity should be taken into account. For example, assume that a team has a velocity of 25 story points per iteration. In this case, the customer representative cannot allocate stories to an iteration if their combined story points exceed this limit. The process of assigning stories to iterations and releases is known as release planning or the planning game, which is the term used in the first edition of the XP book.

For example, consider the Q&A forum we mentioned earlier. The next table summarizes the outcome of a possible release planning. In this example, we are assuming that the customer representative wrote 8 stories, that each release has two iterations, and that the team’s velocity is 21 story points per iteration (note that the sum of the story points of each iteration is 21).

Story	Story Points	Iteration	Release
Register user	8	1	1
Post questions	5	1	1
Post answers	3	1	1
Opening screen	5	1	1
Gamify questions/answers	5	2	1
Search questions/answers	8	2	1
Add tags	5	2	1
Comment on questions/answers	3	2	1

The table emphasizes two points: (1) the stories in XP represent the main features of the intended system; (2) developers have no control on the order of story implementation; this is decided by the customer representative, who must be someone capable and with authority to define the most urgent and important features for the organization contracting the development.

After the release planning, the iterations start. At the beginning of each iteration, the team must meet to conduct an iteration planning. The purpose of this planning is to decompose the stories of an iteration into tasks that can be assigned to the team’s developers. For example, the Post Questions story can be decomposed into the following tasks:

• Design the web interface, including layout, CSS templates, etc.
• Install the database, as well as design and create the database tables.
• Implement the data access layer.
• Install and test the web server framework.
• Implement the control layer, with endpoints to register, remove, and update questions.
• Implement the web interface.

Generally, these tasks should not be complex and should be completed in a few days, for example.

So, an XP project encompasses:

• Releases, which are groups of iterations spanning a few months.
• Iterations, which are groups of stories and respective tasks, lasting a few weeks.
• Tasks, which take a few days to complete.

Once the tasks are created, the team should also decide who is responsible for each one. With that, the iteration begins and the developers start to work on their respective tasks.

An iteration concludes when all the stories have been implemented and validated by the customer representative. Therefore, at an iteration’s end, the stories should be demonstrated to the customer representative, who must check and confirm their implementations.

XP also proposes that teams should program slack tasks, or tasks that can be postponed during an iteration. Examples include studying a new technology, taking an online course, improving documentation, or even developing a side project. In fact, companies like Google are renowned for allowing developers to use 20% of their time for personal projects (link). In XP, slack tasks have two primary objectives: (1) create a buffer that can be used should a task require more time than estimated; and (2) allow developers some downtime from the demanding and intense project pace.

Frequently Asked Questions 🔗

Now let’s answer some questions about the XP practices we have just studied.

How long should an iteration be? There isn’t a definitive answer as numerous factors such as team characteristics, system domain, and requirements all play a part. Short iterations—typically a week long—offer quick feedback. However, they also require more customer involvement as new software increments need weekly validation. Moreover, stories must be simple and straightforward to fit in a 1-week iteration. Conversely, longer iterations—like a month—grant the team has more time to plan and execute tasks with less pressure. The trade-off here is delays in receiving customer feedback, which is particularly relevant when the requirements are unclear. As a result, a 2-3 week time box tends to provide a balance between rapid feedback and less pressure on the team. Another useful recommendation is to experiment, meaning testing and evaluating different durations before settling on one.

What role does the customer representative play during iterations? At the beginning of a release, the customer representative drafts the stories to be included in that release. They then validate and approve the implementation of these stories at the end of each iteration. The representative also needs to be readily available during iterations to address questions the team may have. Given that user stories contain only a few sentences, these questions might naturally arise.

How do we select the customer representative? The most important criteria are that the person has a solid understanding of the system’s domain and authority to prioritize user stories. There are at least three potential scenarios for customer representatives:

• Consider a team working on a system for a department within their own company. In this scenario, the customer representative might be an employee from that client department.

• Consider a team working on a system for a different company. Here, the customer representative should ideally be an employee from that contracting company who has knowledge of the software domain.

• Consider a team working on a product that can be contracted or subscribed to by any client. The ideal customer representative in this case should come from the marketing, sales, or business departments. They should be closely aligned with the problem but distant from the solution. For this reason, it’s preferable not to have a developer in this role. This type of representative is often referred to as a user proxy.

How to define the team’s velocity? There is no silver bullet for this issue. This definition depends on the team’s experience. If the team members have already participated in projects similar to the one they are starting, this is a less difficult question. Otherwise, the team needs to test and calibrate their velocity throughout successive iterations.

Can stories refer to software engineering activities? No, as the customer representative—who is a non-software engineering expert—is the one responsible for defining the stories. However, a story could trigger a task like install and test the database. In essence, stories refer to functional requirements. On the other hand, the tasks created for implementing these stories might relate to functional requirements, non-functional requirements or technical tasks such as database or framework installation.

What if story X is prioritized before Y, despite depending on Y? For instance, the customer representative has allocated the story Post Question to iteration 2 and Post Answer to iteration 1. Although it seems contradictory, the team should respect this order as the final decision always lies with the customer representative. But then the dilemma might be how can we post responses without having questions? To resolve this problem, placeholder questions that can’t be edited by users can be used. In iteration 1, these questions will appear by default and users will be able to respond to them.

When does an XP project conclude? The project concludes when the customer representative determines that the already implemented stories are sufficient, and there are no more relevant stories to be implemented.

2.3.4 Programming Practices 🔗

The name Extreme Programming was chosen due to the novel programming practices the method proposed at the end of the 90’s. The essence of XP revolves around these programming practices and also on the production of running code since the first weeks of a project. The importance of this fact can be better understood within the context of the 90’s where distinct roles for analysts and programmers were the norm. Analysts at that time were responsible for the high-level design of the system, defining its main components, classes, and interfaces using modeling languages, like UML, which we will discuss in Chapter 4. Only upon completion of the analysis and design phases, the coding phase would start under the responsibility of programmers. This process inherently created a hierarchy where analysts were often regarded as the more prestigious roles. However, with the introduction of agile methodologies, particularly XP, this hierarchy was dismantled, since XP advocates for starting coding at a project’s outset.

Simultaneously, to counteract the risk of low-quality code production, XP introduced a novel set of programming practices, including pair programming, automated testing, test-driven development (TDD), automated builds, and continuous integration, among others. Most of these practices were widely accepted within the software industry, and are extensively used nowadays even by projects not entirely based on agile principles.

We will now look at XP’s programming practices.

Incremental Design. XP projects do not have a traditional design phase, known as Big Design Up Front (BDUF). Instead, XP promotes the idea of making design a continuous and incremental activity. This approach aims to address issues that arise when a big design phase occurs at the beginning of a project. This is a stage where the project’s requirements are unclear both to the team and to the customers, leading to overestimating or underestimating their value. Moreover, new requirements may come up as the project progresses, making the initially planned design obsolete.

For this reason, XP contends that the best time to think about design is when it becomes crucial. Two phrases often used to encourage and justify this practice are: Do the simplest thing that could possibly work and you aren’t going to need it, the latter known by the acronym YAGNI.

For a better understanding of the incremental approach to design proposed by XP, two points need to be highlighted. Firstly, proficient teams often have a design in mind before the first iteration of a project. This might include knowing that the system will have a web interface, a set of backend components for implementing complex business logic and also components for accessing relational databases. These decisions, by themselves, already cover and solve major design concerns. Secondly, at any iteration, the team can also propose a technical task to discuss and refine the system’s design.

Finally, it’s crucial to note that incremental design should be used in conjunction with other XP practices, especially refactoring. XP insists on the importance of refactoring to enhance the design’s quality. Hence, every opportunity to refactor should be put in practice without delay.

Pair Programming. Accompanied by incremental design, pair programming stands out as one of the most controversial practices proposed by Extreme Programming. The concept is straightforward: every coding task, whether it’s implementing a user story, implementing a unit test, or fixing a bug, should be conducted by two developers working together, using the same keyboard and monitor. One developer takes on the role of driver, controlling the keyboard and mouse, while the other acts as a navigator or reviewer. The navigator monitors the driver’s decisions, providing feedback and direction, similar to a car rally team.

Pair programming aims to enhance code and design quality, guided by the principle that two heads are better than one. It also aids in spreading knowledge about the code within a team, preventing silos where only one developer understands a specific part of the code. The practice also provides valuable training lessons for novice developers, offering exposure to various programming technologies, algorithms, data structures, design patterns, and principles, testing and debugging techniques, etc.

However, employing two developers per task has economic implications, as usually commented by pair programming critics. Also, for some developers it is emotionally and cognitively uncomfortable to discuss each line of code and each implementation decision with a colleague. To mitigate this, XP recommends to rotate pairs at every session. These sessions are typically 50 minutes long, followed by a 10-minute break. In the subsequent session, both pairs and roles (driver vs. navigator) switch.

Real World: In 2008, Microsoft Research’s Andrew Begel and Nachiappan Nagappan surveyed 106 company developers about their experiences with pair programming (link). Approximately 65% of the respondents reported finding pair programming beneficial. When asked about the advantages, the most cited were reduced bugs (62%), better code quality (45%), knowledge spread (40%), and peer learning opportunities (40%). However, 75% of respondents considered the associated cost a significant disadvantage. Lastly, the majority (38%) preferred to pair with colleagues who have complementary skills to compensate for their own weaknesses.

In recent years, the practice of code review has been embraced by many software companies. In this case, another developer checks the code asynchronously, eliminating the need for two developers to work simultaneously on the same code.

Collective Code Ownership. According to this practice, any developer or pair of developers can modify any part of the code, whether to implement a new feature, fix a bug, or perform a refactoring, without needing approval from the original developer or the last person who maintained it.

Automated Testing. This programming practice, advocated by XP, has achieved significant success. The rationale is that manual testing, where a tester runs the program, provides inputs, and checks the outputs, is a costly and non-reproducible procedure. Therefore, XP proposes implementing programs, called automated tests, that call small code units, such as methods, and verify if the results match the expected outcomes. The advent of automated testing coincided with the development of the first unit testing frameworks, including JUnit (developed by Kent Beck and Erich Gamma in 1997), which played a key role in promoting the practice. We reserve a detailed discussion about automated tests for Chapter 8.

Test-Driven Development (TDD). TDD, another innovative programming practice proposed by XP, advocates that every method should have a test written before the implementation of the method. This practice serves two main purposes: (1) it ensures developers do not forget to write tests by implementing them first, and (2) by writing a test first, developers also adopt the perspective of the method’s user. For example, they should think about the method’s interface and its usage before starting the actual implementation. This encourages the creation of user-friendly methods. We will discuss TDD in Chapter 8.

Automated Build. This practice involves the automated generation of an executable system that is ready for production. Several tools can facilitate this process, including the Make system, part of Unix distributions since the 1970s, and newer tools like Ant, Maven, and Gradle. XP emphasizes two key concepts regarding the build process. First, to eliminate manual intervention by developers, the process should be fully automated. Second, the build process should be as quick as possible to provide developers with immediate feedback on potential issues, such as compilation errors or failing tests.

Continuous Integration. As we discussed in Chapter 1, version control systems (VCS) such as Git are widely used nowadays. A VCS stores the source code and related files, including configuration files, images, and documentation. To work on a task, developers pull the source code to their local machines, make modifications, and then push the updated code back to the VCS repository. This step of uploading and merging the code is referred to as integration.

However, integration is subject to conflicts when multiple developers change the same lines of code. In such cases, the VCS prevents the integration from proceeding. For example, if Developer A assigns a value of 10 to a variable x, while Developer B, working independently, assigns x a value of 20, a conflict arises. To resolve this conflict, A and B must agree on the correct value for x. This is a simple example, but conflicts often involve larger sections of code and more developers, making them time-consuming and challenging to resolve and leading to situations known as integration hell.

To minimize integration conflicts, XP encourages developers to integrate their code frequently—daily, if possible—under a practice known as continuous integration. The aim is to reduce the number of conflicts and the time developers spend on resolving them.

To check the quality of the code that is integrated, we should also configure a continuous integration service. Before carrying out any integration, this service builds the code and executes the tests. The aim is to ensure that the new code does not have compilation errors and that it passes all the tests. Examples of these services include GitHub Actions, Jenkins, TravisCI, and more. In Chapter 10, we will delve deeper into Continuous Integration.

Real World: In 2010, Laurie Williams, a professor at the University of North Carolina in the USA, surveyed 326 developers about their experiences with agile methodologies (link). Participants ranked the importance of various agile practices on a scale of 1 to 5, with five denoting essential practices. Three practices tied for first place with an average score of 4.5 and a standard deviation of 0.8: continuous integration, short iterations (less than 30 days), and definition of done criteria (i.e., the criteria used to consider the implementation of user stories finished). Conversely, planning poker and pair programming were ranked lower, with average scores of 3.1 and 3.3, respectively.

2.3.5 Project Management Practices 🔗

Working Environment. XP advocates for the development of software by small, dedicated teams, ideally composed of fewer than 10 developers. It also discourages scenarios where team members split their time between multiple projects during the week.

XP also emphasizes the importance of maintaining sustainable work schedules. It’s a known fact that software companies often demand long working hours, extensive overtime, and weekend commitments. However, XP contests this practice, arguing instead for sustainable work schedules, with around 40 hours per week, even before important deliveries. Notably, XP was proposed by seasoned professionals with experience in managing real-world software projects, and thus, they had firsthand experience of the effects of prolonged working hours. Long hours can be detrimental to developers’ physical and mental health and can also lead to high turnover, with developers frequently contemplating changing jobs.

Negotiated Scope Contracts. Two types of contracts are possible when outsourcing development: negotiated scope and fixed-scope contracts. Fixed-scope contracts require the hiring company to provide a specification of the requirements. Based on that, the outsourced company establishes a cost and a delivery date. XP questions these contracts due to the volatile nature of software requirements and the customer’s inability to define them in advance. Consequently, the outsourced company may deliver a system with several quality issues, including bugs, solely to avoid penalties.

Conversely, negotiated scope contracts (or open scope contracts) are based on a pay-per-hour model. For instance, the contract might prescribe the allocation of a full-time development team from the outsourced company, which will work in accordance with XP practices. The customer proposes the stories and validates their implementation at the end of each iteration. These contracts also offer the flexibility for termination or renewal every few months, enabling customers to switch companies whenever they find the quality unsatisfactory. As is usual in XP, the objective is to cultivate communication and feedback streams between customers and suppliers, rather than compelling the latter to deliver a low-quality product just to fulfill the contract. In summary, negotiated scope contracts align better with the Agile Manifesto, which recommends customer collaboration over contract negotiation.

Process Metrics. To efficiently track an XP project, managers and executives are encouraged to use two key metrics: the number of bugs detected in production (ideally a few per year) and the time interval between project start and the project yielding the first financial returns (ideally within one year).

2.4 Scrum 🔗

Scrum is an agile, iterative, and incremental method for project management. It was first proposed by Jeff Sutherland and Ken Schwaber in 1995 (link). Among various agile methods, Scrum is probably the most popular and broadly applied one. This widespread acceptance can be partly explained by the proliferation of related books, educational courses, professional consultancies, and certifications.

We begin this section by addressing a frequently asked question: How do Scrum and XP differ? There are a number of minor variations, but the key difference lies in their purpose:

• XP is an agile method specifically designed for software development projects. For this reason, it introduces a set of programming practices, such as unit testing, pair programming, continuous integration, and incremental design, which we have explored in the previous section.

• Scrum, however, is an agile method that isn’t confined to software projects. For example, the creation of this book, as we will explain shortly, was based on Scrum concepts. Having a broader scope than XP, Scrum doesn’t specify any programming practices.

Moreover, among the agile methods, Scrum stands out as the most structured, including a well-defined set of roles, artifacts, and events, which are listed below. In the rest of this section, we will explain each of them.

• Roles: Product Owner, Scrum Master, Developer.

• Artifacts: Product Backlog, Sprint Backlog, Scrum Board, Burndown Chart.

• Events: Sprint Planning, Sprint, Daily Meetings, Sprint Review, Retrospective.

2.4.1 Roles 🔗

A Scrum team is composed of a Product Owner, a Scrum Master, and between three and nine developers.

The Product Owner (PO) is equivalent to the customer representative in XP. Their role requires a clear vision for the product being developed, in order to maximize the return on investment for the project. Their responsibilities also include writing user stories, and hence, they should always be available to address the team’s questions.

The role of the Scrum Master is unique to the Scrum method. They are the team’s Scrum expert and their responsibility involves ensuring adherence to the Scrum methodology. They should continuously train and educate the team members on Scrum principles and work as a facilitator to remove obstacles. For instance, if a team encounters issues with faulty hardware, it is the Scrum Master’s role to ensure this issue is resolved promptly to avoid hindering the team’s progress. However, the Scrum Master is not a traditional project manager. For example, they do not lead the team, as Scrum teams operate on a non-hierarchical basis.

Scrum teams are often described as cross-functional or multidisciplinary, implying they should have all necessary specialists for developing the product, thus avoiding dependencies on external members. In software projects, this includes specialists such as front-end developers, back-end developers, database specialists, and UI designers. These experts are responsible for the technical decisions during the project, including defining the programming language, architecture, and frameworks to be used in the implementation. They are also responsible for estimating the size of the stories proposed by the Product Owner, using units such as story points, similar to what we studied in XP.

2.4.2 Artifacts and Events 🔗

Scrum defines two key artifacts: the Product Backlog and the Sprint Backlog. The key events include Sprints and Sprint Planning. Next, we detail these artifacts and events.

• The Product Backlog is a prioritized list of user stories (along with other relevant work items). Much like XP, the Product Owner is responsible for proposing these stories, which are just short descriptions of the features the system should have. It’s also worth noting that the Product Backlog is not static. It constantly requires updates to capture changes to requirements or to the product vision. For instance, as the project advances, new feature ideas may come up while some may no longer hold relevance. These changes are under the responsibility of the Product Owner—a role which has this title precisely for maintaining the Product Backlog.

• Sprint refers to a time-boxed iteration in Scrum. Like all agile methods, Scrum employs an iterative approach, organizing the development into sprints, each one lasting up to a month. Every sprint should yield a product increment that has tangible value for the customer, called a potentially shippable product. The term potential indicates that transitioning the product to production is not mandatory, as we previously elaborated in Section 2.2.

• Sprint Planning is a preparatory meeting that serves as the launchpad for every sprint. In this meeting, the team decides which stories will be incorporated in the forthcoming sprint. This meeting consists of two main parts. First, the Product Owner suggests stories for the upcoming sprint, while the remaining team members evaluate if they have enough velocity (or capacity) to implement them. In the second part, led by the developers, the selected stories are decomposed into tasks and their execution times are estimated. The Product Owner should be present in both parts of the Sprint Planning to answer questions about the selected stories. In some circumstances, the team may decide to remove a story, if it proves to be more complex when broken down into tasks.

• The Sprint Backlog is an artifact generated at the conclusion of the Sprint Planning. It is a list of tasks for the sprint, each with its respective duration. Like the Product Backlog, the Sprint Backlog is also a dynamic entity. Tasks may be considered unnecessary or new ones may be created throughout the sprint, and the estimated effort for each task may also be adjusted if needed. However, the sprint goal—the stories selected by the Product Owner for the sprint and that the development team has committed to implement—should not change. Thus, while Scrum is adaptable to changes, such changes should occur between sprints. During a sprint, the development team should have the stability and assurance of working on a fixed list of stories.

When the sprint planning concludes, the sprint begins, meaning the team starts working on the tasks in the sprint backlog. At this point, it’s also worth mentioning that Scrum teams are not only cross-functional but also self-organizing, that is, they have the autonomy to decide how and who will implement the selected stories.

In addition to the sprint backlog, teams can use a board with columns denoting individual story stages, such as to do, in progress, testing, and done. This board, known as a Scrum Board, provides a visual aid to tracking the daily progress of the sprint. See the next figure for an example.

Defining the criteria for marking a story or task as done is a crucial decision in Scrum projects. The team should reach a consensus on these criteria, and every member should follow them. For instance, the team can define a story as completed only if all its unit tests pass. A code review by another team member may also be required. Finally, the code should be successfully merged into the project repository. These criteria deter team members from prematurely moving their tasks to the done column by using substandard code.

The Burndown Chart is another familiar artifact in Scrum. It displays the number of work hours needed to complete the unfinished tasks on each day of the sprint. In other words, on day x, it shows that the unfinished sprint tasks require y hours of work. Ideally, the burndown chart should exhibit a continuous decline, hitting zero by the end of the sprint (assuming success). Next, we present a burndown chart for a sprint that spans 15 days.

2.4.3 Additional Scrum Events 🔗

We will now explore three additional Scrum events: Daily Stand-Up, Sprint Review, and Retrospective.

Scrum recommends holding Daily Stand-Up meetings, typically 15 minutes long, involving all team members. As the name suggests, these meetings are usually held with participants standing up, leading to alternative names such as Stand-Up Meetings or Daily Scrum. In these meetings, each team member should respond to the following three key questions: (1) What tasks were completed the previous day? (2) What tasks are planned for the current day? (3) Are there any significant obstacles hindering their progress? The intention of these meetings is to facilitate communication and information sharing among team members, and to provide a platform to keep each member updated on the project’s progress. For example, during the Daily Stand-Up, two developers may discover they are about to work on the same code section. In this scenario, it would be beneficial to arrange a separate meeting to discuss their planned changes, thereby reducing potential conflicts during the integration of their modifications.

The Sprint Review is intended to showcase the results of a sprint. All team members should participate, including additional stakeholders involved in the sprint’s goal. In this meeting, the team demonstrates the finished product to the customers. Ideally, the Product Owner should approve all stories implemented during the sprint. However, if any issues arise with a story, it is moved back to the Product Backlog to be reworked in a future sprint. Likewise, the stories the team was not able to finish are also returned to the Product Backlog.

The Retrospective marks the end of a sprint. This meeting aims to reflect on the completed sprint and to identify possible improvements related to the process, team interrelationships, and the employed practices and tools. For instance, as a result of the retrospective, the team might emphasize the importance of punctuality and consistent attendance at Daily Stand-Up meetings, particularly if lateness has previously been an issue. However, the aim of a retrospective is not to trigger complaints or to have heated discussions among team members. Instead, critical issues should be addressed separately in other meetings or under the supervision of managers. After the retrospective concludes, the project continues with the forthcoming sprint.

A key aspect of Scrum events is their fixed duration, also known as the time-box of the event. This is why this term frequently appears in Scrum literature. Consider, for instance, a quote from the official Scrum Guide (2017 version, page 9): The heart of Scrum is a sprint, a time-box of one month or less during which a done, usable, and potentially releasable product increment is created. The purpose of defining time-boxes is to maintain a steady workflow, encourage commitment from the team, and prevent diversion from the sprint goal.

Next, we show a table with the time-boxes associated with Scrum events. When events have a maximum time-box (like sprint planning), this maximum value applies to a one-month sprint. Thus, in the case of shorter sprints, the respective time-box should be adjusted accordingly.

Event	Time-box
Sprint Planning	maximum of 8 hours
Sprint	less than 1 month
Daily Stand-Up	15 minutes
Sprint Review	maximum of 4 hours
Retrospective	maximum of 3 hours

2.4.4 Example: Writing a Book using Scrum 🔗

This book was written using various elements of Scrum. For example, at the book’s inception, we planned the chapters, thus establishing a Product Backlog. After that, each chapter was written within a sprint. During the Sprint Planning meetings, we partitioned each chapter into sections, thus treating these sections as tasks. Subsequently, the writing for each chapter began, marking the start of a sprint. Generally, the sprints were scheduled to last two months. Below we show the backlog of the current sprint, whose goal is to write Chapter 2. We can also see the status of each task/section.

Story	To Do	In Progress	Done
Chapter 2 - Processes	Kanban When not Use Other Methods Exercises	Scrum	Introduction Agile Manifesto XP

We decided on an agile approach when writing this book to minimize risks associated with developing a product that doesn’t meet the needs of our primary audience—students and instructors of Software Engineering courses, mainly at the undergraduate level. As such, at the end of each sprint, a chapter was shared and announced publicly, with the aim of receiving feedback. Thus, our objective was to avoid a Waterfall approach, which would involve writing the book over years without receiving any input.

To conclude, it’s important to examine the criteria we used for considering a chapter as done. It involves the author of the book thoroughly reading and reviewing the chapter. Once this review was completed, the chapter was tentatively released to the members of the Applied Software Engineering Research Group at DCC/UFMG for a second round of review and feedback.

2.4.5 Frequently Asked Questions 🔗

Before we conclude, here are answers to some common questions about Scrum:

What does Scrum mean? Scrum is not an acronym but a term borrowed from rugby, where it refers to a field meeting where rugby teams compete for ball possession after an accidental infringement.

What is a squad? A squad is another term for an agile or a Scrum team. The term was popularized by Spotify. Similar to Scrum teams, squads are small, cross-functional, and self-organizing units. The term tribe is also commonly used to designate a group of squads.

Can the Product Owner role be assigned to a committee? In simpler terms, can a Scrum team have more than one Product Owner? The answer is no. This role must be assigned to a single individual to avoid committee-based decision-making, which often leads to products overloaded with features, implemented mainly to satisfy specific committee members. However, the Product Owner can and should act as a link between the team and other experts in the product being developed. Indeed, this facilitation is expected as some requirements might be specific to the domain knowledge of certain individuals within the organization.

Is the Scrum Master considered a full-time role? Ideally, a Scrum Master should be committed on a full-time basis. However, for mature teams with vast experience in Scrum, a full-time Scrum Master might not be necessary. In such scenarios, two alternatives can be considered: (1) Allow the Scrum Master to operate in multiple teams, or (2) Delegate the Scrum Master role to a team member. If the latter approach is used, the chosen member should not simultaneously act as the Product Owner. This is because one of the responsibilities of a Scrum Master is to support the PO in writing and prioritizing the user stories.

Does a Scrum Master need a degree in Computer Science? No. A degree in Computer Science is not a prerequisite, as a Scrum Master’s role primarily involves removing obstacles and making sure the team follows the Scrum principles. Thus, they are not expected to resolve technical issues like debugging, using frameworks correctly, implementing features etc. On the other hand, some companies may require Scrum Master certifications.

In addition to stories, what other items can be included in the Product Backlog? In fact, the Product Backlog is not limited to user stories—it can also comprise other work items like bug fixes (particularly complex ones) and major refactoring efforts, for example.

Do Scrum organizations employ managers? Yes. Although Scrum teams have autonomy to implement the stories defined by the Product Owner, additional decisions typically associated with managerial levels are required in a software development project. Examples include:

• Team composition: Managers are responsible for allocating developers to teams, as developers usually don’t have the authority to select their teams.

• Definition of goals and responsibilities: Managers designate priorities and goals for each team, which includes the systems they will be working on. For example, a team doesn’t decide by itself that a new accounting system is needed and then start its development. Such strategic decisions are the responsibility of the organizations’ managers and executives.

• Human resources management: Managers also handle human resource tasks, such as recruiting new employees, dismissals, promotions, and training initiatives.

• Performance evaluation: Managers are also tasked with assessing whether the Scrum teams are providing value to the organization.

2.5 Kanban 🔗

The Japanese term kanban translates to visual card. Since the 1950s, the term is also associated with the just-in-time production process of Japanese factories, particularly in Toyota where it was initially implemented. This process is also recognized as the Toyota Production System (TPS) or, in more contemporary terms, lean manufacturing.

In the context of software development, Kanban was first introduced at Microsoft in 2004 by David Anderson (link). Anderson describes Kanban as a method that promotes sustainable work pace for development teams by eliminating waste, delivering consistent value, and fostering a culture of continuous improvement.

To better understand Kanban, we will compare it to Scrum. Primarily, Kanban is simpler than Scrum because it does not prescribe events like sprints. Furthermore, strictly speaking, Kanban does not define particular roles like Product Owner or Scrum Master. Lastly, unlike Scrum, Kanban doesn’t include several artifacts, with the exception of one main artifact: the task board. In Kanban, this is referred to as the Kanban Board, which also includes the product backlog.

The Kanban Board is organized into specific columns as described below:

• The first column represents the product backlog, closely mirroring the same artifact in Scrum. User stories are proposed and subsequently added to this backlog.

• The next columns represent the steps required to transition a user story into a functioning feature. These columns typically have labels such as Specification, Implementation, and Code Review. The stories in the backlog are processed sequentially, mirroring the workflow of an assembly line, progressing from left to right. Furthermore, each column is sectioned into two subcolumns labeled in progress and ready. For instance, Implementation contains two subcolumns: tasks under implementation and tasks that have been implemented. Completed tasks from one step await to be pulled into the next step by a team member. This is why Kanban is commonly referred to as a pull system.

We will look at the structure of a Kanban Board using an example.

In this board, we can see stories and tasks at various stages. In the backlog, we have one story labeled H3. Another story, labeled as H2, has been picked up by a team member and is under Specification. Four tasks from a previous story, labeled as T6 to T9, have already been specified. Next, there are two tasks, T4 and T5, which are being implemented. Task T3 has been implemented and is now awaiting Code Review. Lastly, Task T2 is currently under review, while T1 has been fully implemented and is now considered done. You may also have noticed the acronym WIP appearing at all steps, with the exception of the backlog. Don’t worry about this for now; we will explain it shortly. It’s also important to note that, in this example, we used the letters H and T to represent stories and tasks, respectively. However, in real board setups, tasks and stories are represented as cards that carry a brief description of each.

Days later, the Kanban Board evolved into the following state (tasks that moved forward are distinguished by being underlined and colored red).

Note that H2 is no longer visible as it was decomposed into three tasks (T10, T11, and T12). This decomposition is exactly what occurs during the specification phase, whose objective is to convert a story into an actionable task list. Furthermore, T6 and T7—which were on hold—are now under Implementation. Simultaneously, T3 has transitioned into the Code Review step. Finally, the review of T2 has been completed. It’s also important to note that currently there are no tasks waiting for entering in the review stage.

Similar to other agile methods, Kanban teams are self-organizing. This implies they have the autonomy to decide which task they will work on. Also, they are cross-functional—meaning the team is composed of members who have the skillset to perform all the tasks in the Kanban Board.

We will now discuss the concept of WIP Limits (short for Work in Progress). Project management methods typically strive to maintain a sustainable work flow. To achieve this, two scenarios need to be avoided:

1. Teams having no tasks to work on and consequently staying idle most of the time.
2. Teams being overwhelmed with an excess of tasks, thus negatively affecting the quality of the software product.

To circumvent the second scenario, Kanban proposes setting maximum task limits for each step on a board. This cap, referred to as the WIP Limit, represents the maximum number of tasks—or cards—that can be present in each step, including the tasks in progress (first sub-column) and those that are ready (second sub-column). The exception is the final step, where the WIP limit applies only to the first sub-column as it is meaningless to limit the number of completed tasks.

Next, we show a Kanban Board that incorporates these WIP limits. We can see the limits below each step’s name. Our board allows a maximum of 2 stories in Specification, 5 tasks in Implementation, and 3 tasks in Code Review. The reason for the particular WIP limit in Specification will be elaborated later. For now, note that 4 tasks are under Implementation, thus remaining below the step’s WIP limit of 5. Similarly, in Code Review the limit also is followed as only one task is under review. Remember that to comply with the WIP limit, tasks from both sub-columns— in progress and ready tasks—are taken into account, except for the final step where only tasks from the first sub-column count (T3, in this example).

We will now explain the concept of Work in Progress for the Specification step. To compute the WIP for this step, we need to sum up the stories in Specification (which is zero in our previous example) with the stories that have already been specified. For instance, in the example, we’ve specified two stories, i.e., tasks T8 and T9 are products of the same story’s specification, while tasks T10, T11, and T12 resulted from the specification of another story. Therefore, in this step for the means of WIP computation, we have two stories, which is equal to the limit of 2. To ease visualization, we display the tasks emerging from the same story specification on a single line. Hence, to compute the WIP of the Specification step, we should add up the stories in the first sub-column (zero in our case) to the number of lines in the second sub-column (two in this case).

Also about the previous board we have:

• Story H3, in the Backlog, can’t be moved into the Specification because this step’s WIP has reached its maximum limit.

• One of the tasks in the ready subcolumn of Specification (T8 to T12) can be moved to Implementation as the WIP for this step is currently 4, but having a limit of 5.

• One or more tasks in Implementation (T4 to T7) can be moved to ready, which doesn’t affect that step’s WIP.

• Reviewing of T3 can be finalized.

An important point to note here is that WIP limits are in place to avoid overloading the team with tasks. When a team is burdened with numerous tasks—that bypass the WIP limits—the probability of producing low quality software increases. Kanban understands this issue and provides an automatic lock to prevent teams from accepting tasks beyond their delivery capacity. These locks, which are the WIP limits, have internal as well as external usage. For example, they serve as an instrument for the team to decline extra tasks that are possibly pushed by the top management.

2.5.1 Determining WIP Limits 🔗

In this section, we will explain the steps involved in defining Work in Progress (WIP) limits. While several methods exist, we will adapt an algorithm proposed by Eric Brechner, a Microsoft engineer. This algorithm is detailed in his book on using Kanban in software development (link). Next, we outline the algorithm’s steps.

Initially, we need to estimate the average time it takes to complete a task in each step of the Kanban board, known as the lead time (LT) of the step. For our example, we assume these values:

• LT(specification) = 5 days
• LT(implementation) = 12 days
• LT(review) = 6 days

These estimated times reflect an average task’s duration, taking into account that there will be both complex and simple tasks. Note that the lead time includes the queue time, i.e., the task’s waiting time in the ready subcolumns of the Kanban board before progressing to the next step.

After that, we should estimate the throughput (TP) of the step with the longest lead time. In our example (which also holds for most software development projects), this step is Implementation. Assuming the team can implement eight tasks per month, this step’s throughput is: 8 / 21 = 0.38 tasks/day (we’ve assumed a month has 21 workdays).

Finally, each step’s WIP is calculated as:

WIP(step) = TP * LT(step)

Here, TP refers to the throughput of the slowest step, as calculated in the previous step. So, using this formula, we derive:

• WIP(specification) = 0.38 * 5 = 1.9
• WIP(implementation) = 0.38 * 12 = 4.57
• WIP(review) = 0.38 * 6 = 2.29

And after rounding, we get:

• WIP(specification) = 2
• WIP(implementation) = 5
• WIP(review) = 3

Eric Brechner also suggests adding a 50% margin of error to the estimated WIPs to allow for variations in task size and tasks blocked due to external factors. However, we won’t apply this adjustment in our example, as it is purely illustrative.

Finally, it’s important to note that the purpose of WIP limits, as proposed by Kanban, is to maintain a sustainable work pace and therefore to deliver quality software. These limits prevent developers from becoming overburdened with tasks, which would reduce their work quality. Virtually all development methods offer such instruments. For instance, Scrum uses sprints with a fixed time-box to avoid overcommitting and to limit the workload based on the team’s velocity. Once started, the sprint’s goals can’t be changed to shield the team from priority shifts. For Waterfall methods, the strategies to ensure a sustainable workflow with quality outcomes involve a comprehensive phase of requirement specifications. This phase aims to provide developers with a clear understanding of the system they will implement.

2.5.2 Little’s Law 🔗

The procedure used for calculating WIP limits is an application of Little’s Law, a prominent result from Queue Theory (link). Little’s Law states that the number of items in a queue system is the product of the arrival rate of these items and the time each item spends within the system.

When applied to our context, where the items we referred to are tasks, the law can be broken down as follows:

• WIP: number of tasks in a particular step of the Kanban board.

• Throughput (TP): rate at which tasks arrive at this step.

• Lead Time (LT): time each task spends at this step.

Thus, according to Little’s Law, WIP = TP * LT. Below, we also visually represent Little’s Law for a clearer understanding.

2.5.3 Frequently Asked Questions 🔗

Before we conclude, let’s answer some common questions related to Kanban:

What roles exist in Kanban? In contrast to Scrum, Kanban doesn’t prescribe a rigid set of roles. The responsibility rests on the team and the organization to determine the necessary roles, such as a Product Owner, testers, and so on.

How are user stories prioritized? Kanban’s approach is lightweight compared to Scrum and even XP. One reason is the absence of explicitly defined criteria for prioritizing user stories. As mentioned in the previous answer, even the existence of a Product Owner—responsible for prioritization—isn’t mandatory. Thus, alternative solutions like external prioritization by a manager can also be used.

Can Kanban teams conduct Scrum events, such as daily meetings, reviews, and retrospectives? Yes, they can. While Kanban does not prescribe these events, it doesn’t explicitly prevent them either. The decision regarding which events are essential, their timing, duration, etc., is left to the team.

2.6 When Not to Use Agile Methods? 🔗

Although agile methods are very popular today, it’s important to remember that there’s no one-size-fits-all solution in Software Engineering. As such, this section examines scenarios and domains where agile practices may not be the most suitable choice.

However, the question posed in this section cannot be answered simply by stating that “systems from areas X, Y, and Z should avoid agile methods, while others should adopt them. In reality, systems from all areas could benefit from at least some agile practices. On the other hand, some practices might not be advisable for specific types of systems, organizations, and contexts. Therefore, we will answer the proposed question at a finer granularity. That is, we will comment on when to not use certain agile practices.

• Incremental Design: This approach is effective when the team has an initial understanding of the system’s design. However, if the team lacks this vision, or the domain of the system is new and complex, or if future changes have high costs, it may be better to have a design phase prior to starting the implementation of features.

• User Stories: As a lightweight method for specifying requirements, stories are typically clarified with customer participation in the sprints. However, certain projects may require a detailed requirements specification phase at the beginning, particularly if they originate from domains unfamiliar to the developers.

• Customer Involvement: If the requirements are stable and thoroughly understood by the developers, constant involvement of a Customer Representative or Product Owner may be unnecessary. For instance, when developing a compiler for a language with well-established grammar and semantics, this role may not be as important.

• Lightweight Documentation: For specific domains, detailed requirements and comprehensive documentation are important. As examples, we can mention systems where failures can result in human fatalities, such as systems in the medical and transportation sectors. These systems often require certification from an external entity, which may mandate the availability of extensive documentation.

• Self-organizing Teams: Agile teams have autonomy to work without interference during a sprint. For example, they do not need to report daily to the organization’s managers. Nonetheless, this autonomy can conflict with the values and culture of some organizations, particularly those with a hierarchical structure and rigid controls.

• Negotiated Scope Contracts: In such contracts, remuneration is defined on an hourly basis. Thus, some organizations may hesitate to sign contracts with an open scope, especially if they lack experience with agile development or harbor reservations about trusting the contracted company.

To conclude, it’s important to mention that two agile practices have become prevalent in a majority of software projects:

• Small teams, because communication and coordination challenges increase significantly for large teams.

• Iterations (or sprints), even if they are longer than typical agile sprints. For instance, iterations lasting two or three months, as opposed to the usual sprint time-box of less than 30 days. Interestingly, even before the rise of agile methods, some iterative methods were suggested, i.e., methods with validation points throughout the project. We will explore two of these methods in the next section.

2.7 Other Iterative Methods 🔗

The shift from the Waterfall model—highly prevalent during the 1970s and 1980s—to Agile approaches, which gained significant traction in the late 2000s, was a gradual process. For example, some methods proposed during this transition also incorporated the concept of iterations, although having a longer duration than those typically seen nowadays in Agile. Rather than lasting a few weeks, these iterations could extend over months. Furthermore, these transition methods also retained many elements from Waterfall, like the focus on documentation and an initial requirements specification phase followed by a design phase.

One such method formulated during this transition period is the Spiral Model, proposed by Barry Boehm in 1986 (link). In this model, a system is developed through a spiral of iterations. Each iteration, which includes a complete round of the spiral, comprises four stages, as elaborated next (see also the following figure):

1. Defining objectives and constraints, including costs, schedules, etc.

2. Evaluating alternatives and conducting risk analysis. For instance, considering whether it is more cost-effective to purchase a system instead of developing it in-house.

3. Development and testing, potentially using the Waterfall model. By the end of this stage, a prototype should be produced for demonstration to users.

4. Planning for the next iteration or deciding to conclude the project if the current implementation already attends the organization’s needs.

The Spiral Model generates a more comprehensive version of a system after each iteration. Despite this, each iteration, encapsulating all four phases, can take anywhere from 6 to 24 months—which is significantly longer than in XP and Scrum. Another interesting feature of this model is the risk analysis phase, where concrete steps to mitigate project risks are considered.

The Unified Process (UP), proposed in the late 90s, is another example of an iterative method. Proposed by professionals affiliated with Rational, a software consulting company later acquired by IBM in 2003, UP has also a specific version known as the Rational Unified Process (RUP).

Due to its origins, RUP is intrinsically associated with two technologies:

• Unified Modeling Language (UML), since many of the RUP outcomes are documented using UML diagrams (which will be discussed in detail in Chapter 4). However, it’s worth noting that both UML and RUP were proposed by the same group of professionals who intended to define a unified process and modeling language.

• CASE (Computer-Aided Software Engineering) tools, which are analogous to CAD (Computer-Aided Design) tools used in other fields such as Civil Engineering, Mechanical Engineering, and Architecture. RUP proposes that the design of software should rely on UML diagrams. However, these diagrams aren’t drawn on paper, but using CASE tools, as illustrated in the following figure. Thus, besides proposing the method, Rational was also responsible for its supporting tools.

When using RUP, development is divided into the following phases:

• Inception: this phase encompasses tasks such as feasibility analysis, budget definition, risk assessment, and system scope determination. At the end of this phase, the business case for the system should be well established. We can also conclude that it is more cost-effective to purchase a system in the market than to build one.

• Elaboration: this phase involves tasks such as requirements specification, software architecture definition, and the proposal of a development plan. Upon completion of this phase, the risks identified during the inception phase should be effectively managed and mitigated.

• Construction: this phase deals with lower-level design, implementation, and testing. Upon its completion, a working software, including documentation and manuals, should be available.

• Transition: in this phase, the system is moved to production. Thus, it includes defining all necessary deployment procedures, such as backup and recovery policies, data migration from legacy systems, and training the operations teams.

RUP also defines various engineering disciplines, including business modeling, requirement definition, design analysis, implementation, testing, and deployment. These disciplines, also referred to as process workflows, can take place in any phase. However, certain disciplines are more important during specific phases, as illustrated in the next figure. In this example, business modeling tasks are primarily performed during the initial phases (inception and elaboration). Conversely, most of the implementation happens during the construction phase.

Bibliography 🔗

Kent Beck, Cynthia Andres. Extreme Programming Explained: Embrace Change. Addison-Wesley, 2nd edition, 2004.

Kent Beck. Embracing Change with Extreme Programming. IEEE Computer, vol. 32, issue 10, p. 70-77, 1999.

Kent Beck, Martin Fowler. Planning Extreme Programming. Addison-Wesley, 2000.

Ken Schwaber, Jeff Sutherland. The Scrum Guide, 2017.

Kenneth Rubin. Essential Scrum: A Practical Guide to the Most Popular Agile Process. Addison-Wesley, 2012

Eric Brechner. Agile Project Management with Kanban. Microsoft Press, 2015.

David Anderson. Kanban. Blue Hole Press, 2013.

Ian Sommerville. Software Engineering. Pearson, 10th edition, 2019.

Hans van Vliet. Software Engineering: Principles and Practice. 3rd edition. Wiley, 2008.

Armando Fox, David Patterson. Engineering Software as a Service: An Agile Approach Using Cloud Computing. 1st edition, 2014.

Exercises 🔗

1. How are software development contracts structured when adopting XP?

2. Describe the key differences between XP and Scrum.

3. Scrum teams are cross-functional and self-organizing. Define these terms.

4. In Scrum, what is the difference between stories at the top and in the bottom of the Product Backlog?

5. What are story points and what are they used for?

6. What is the difference between a sprint review and a retrospective?

7. Can a sprint be canceled? If so, by whom and for what reasons? To answer this question, check out the Scrum Guide (link).

8. Consider a system within your area of expertise. (a) Propose a user story for this system. (b) Break this story into tasks, akin to what we did with the Q&A forum in the section about XP. (c) Do these tasks have dependencies or can they be implemented in any order?

9. Suppose two teams, A and B, working in different projects, from different companies with no interconnections. Both teams follow 15-day sprint cycles and both have five developers. In their respective projects, team A estimates their velocity as 24 story points, while team B estimates theirs at 16 story points. Can we conclude that team A is 50% more productive than team B? Justify your answer.

10. Describe the primary differences between Scrum and Kanban.

11. What are the differences between a Scrum Board and a Kanban Board?

12. What is the error in the following Kanban board?

13. Consider the following Kanban board. At this moment, the team cannot work on the specification of new stories, as the WIP for this step is filled by items waiting to move to Implementation. What is more recommended at this moment: Bypass the WIP limit and pull a new story from the Backlog to Specification? Or work on the tasks in Validation?

14. Why is it usually suggested to increase the WIP limits computed using Little’s Law, for example, by 50%? What circumstances can lead to errors when computing the WIP limits?

15. Describe the instruments provided by Waterfall, Scrum, and Kanban for managing risks and promoting the development of high-quality software.

16. Consider a Kanban board with four stages. The following table provides the lead time for each stage and the throughput for stage C, which is the slowest one. Using these values, calculate the WIP for each stage.

Step	Lead Time (average, days)	Throughput (tasks/day)	WIP
A	4	-
B	3	-
C	10	0.5
D	5	-

17. Consider the following chart, which illustrates how the costs of changes vary according to the development phase in which they occur for a particular software project. (a) Which development method would you recommend for this project, and why? Give examples of systems that might have a similar change cost curve.

18. The article Development and Deployment at Facebook (link) presents the development practices employed at Facebook. In the first section (pages 2-3; figure 2), the authors distinguish between several development methods based on their release frequency. Fill the following table with the release frequency for the methods and policies discussed in that part of the article.

Method	Frequency of new releases
Waterfall
Evolutionary
Agile
Facebook
Continuous Deployment

19. Why are RUP and Spiral not categorized as agile methods? How do they differ from Waterfall?