• Read more about Redesigning Agile: Part III - Putting Forge to Work

In the first two posts in this series, I explained some of the problems that we had with traditional Scrum and then laid out our new Agile process: Forge. In this post, I'm going to share a bit more about implementing Forge on our project and tell you about some of the benefits and challenges that we ran into.

The Benefits

Forge has solved a lot of problems for our team. Here are some of the most significant benefits we have seen.

• Getting rid of technical debt

  One of the biggest problems with any project is the accumulation of “technical debt” (TD). TD is simply bugs and complexity that get introduced as a project gets larger. Things get even worse if you aren’t starting from scratch and will be working with someone else’s code.

  The advantage of having a finisher (or many finishers towards the end of the iteration) is that you have a dedicated developer whose primary job is reducing TD. Few other processes even acknowledge that you accumulate TD, let alone insist on a mechanism for reducing it. Over time, the net result is a more stable, more maintainable project. Ideally the process will also result in a robust library of modules and/or open source components that you can tap for enhancements and future projects.

• Developer focus and ownership

  Since each new feature is assigned to a single developer, he/she develops a sense of ownership of that feature. There are several key benefits you get from this. First of all, since the implementation details are left entirely up to your developers, they are granted a level of autonomy and freedom of design that is very rewarding.

  Second, since each developer will begin an iteration as the sole person responsible for a new feature, by the time another developer is available to help carry it to completion, they will be very easily integrated into the task at hand. This dramatically reduces potential conflicts and improves productivity.

• Clearly defined release feature set

  After a final QA check at the end of each iteration, the product goes through a full release. This process gives the team the opportunity to roll out new features as part of a complete release that can be communicated to users and partners. This vastly improves transparency and gives the customer the opportunity to promote the new release and also alert users of potential downtime.

The Challenges

Despite these benefits, Forge is not a silver bullet. There are several challenges that we have run into and had to either work around or adjust our expectations.

• Delayed gratification

  One of the biggest roadblocks we faced was that our clients weren’t happy with the fact that they would need to wait for completed features. For example, if Sally completes a new feature four days into the iteration then it can be hard to explain why they have to wait another week for it to go to production. Over time the benefits of having a clear release announcement have outweighed this concern, but it can still be a hard sell for some organizations.

• Long-running features

  Some features can also be difficult to implement and thus take far too long to tie the delivery of an entire iteration on its completion. For example, if you have six features slated for an iteration and one of them is expected to take four weeks but the timelines for the others are measured in days, then it doesn’t make sense to have most of your development team acting as finishers for three weeks while you wait for the completion of the difficult feature.

  Features that are going to take a long time to complete should typically be known during the planning phases. When one of them is on the list, the simplest thing to do is to let it carry over multiple iterations. When the feature is finally completed, the developer working on it just becomes a finisher for the remainder of the iteration as they normally would.

• New projects

  Forge was created for a project that we inherited from another development company and so it came with a great deal of existing technical debt. In addition to simple code complexity, we had to deal with a large number of bugs from the first day on the project. This makes it easy to figure out what the finisher should be doing during that first iteration, but most projects aren’t "rescue" projects and the team will be starting from a blank canvas.

  Forge obviously still works in those situations, the only tricky thing is to figure out what the finisher should be doing early on. There are two different approaches that can be taken. First, you can leave off the finisher during the first iteration or two and only bring that role into play once you see yourself starting to accumulate technical debt. This can be risky because you may end up in a situation where the finisher never actually materializes, so we recommend that you explicitly limit the number of iterations without a finisher to one or two from the outset.

  The other possibility is to task the finisher early on with some of the infrastructure-oriented tasks. Configuring servers, deployment scripts, and continuous integration systems are all things that need to be done early on and may be a good place for the finisher.

• Small teams

  Forge assumes that you will always have one person dedicated to fixing bugs, but on projects with very small teams (e.g. 2-3 people) this can seem like a burden. The truth is that it probably is a burden for those projects. In these situations, you will need to tailor the process a bit to suit the circumstances.

  There are a number of ways to address the issue. Some ideas that pop to mind are only including the finisher role every other iteration or requiring that the first person to complete their task always becomes a finisher instead of rolling on to help with another feature. Each project should address these challenges in the way that suits the team and the client best without worrying about rigidly sticking to a process.

• Multi-finisher iterations and blitzes

  There were times on our project where we found ourselves falling behind on technical debt, even with a dedicated finisher on each iteration. This was most apparent when a new feature that we released turned out to have unforeseen consequences across the product. We found that sometimes we needed to run an iteration where we were less focused on features and more focused on finishing.

  These special iterations took one of two forms. The easiest form was simply to assign more than one person to the finisher role. When we might normally have five developers working features and one finisher, we would conduct an iteration with only two features and four finishers.

  The other form was what we called a blitz. We would designate an iteration length of a week and put the entire team on a finisher role. Blitzes also served an important planning role. The management team would spend blitzes reevaluating the whole feature wishlist and make sure that we were still on the right track. At the end of a blitz, we would have a fresh start with a renewed focus and direction for the product.

Conclusion

Forge has become a powerful weapon in Intridea’s arsenal. At the core, we are successful because we have extremely talented developers who can work miracles. But even the best developers in the world can run into roadblocks if the process they work under doesn’t help them get their jobs done. Forge helps our developers get the job done by making sure that the raw materials of a project, the ideas, are malleable enough to be worked into a finished product.

• Read more about Redesigning Agile: Part II - Introducing Intridea Forge

In the first post in this series, I talked about some of the challenges that we faced trying to implement Scrum on one of our large client projects. After failing to reap the benefits from Scrum, I designed a new process that was better suited to our project and our team. Thus Intridea Forge was born.

On the weekends, I help out at a local non-profit educational dairy farm doing blacksmithing. In the blacksmith's workshop, you start with raw materials and put them in the forge to heat up. The forge heats up the metal to the point where it's soft enough to be worked with tools like hammers and anvils.

This new process serves as a forge for ideas about the product. We use the process to take the ideas from cold, bare metal to a point where they are hot and maleable for the developers to work them into a finished product.

Intridea Forge in 200 words or less

In case you don't care about some of the details, here's the tl;dr version of Forge.

Forge is centered around three key concepts: release iterations, feature ownership, and the finisher.

Release iterations are variable-length, feature-based iterations that correspond to a full production release of the product. Iterations aren't tied to a fixed timeline and they only end when the last of the features is completed. Every iteration is a release iteration.

Feature ownership is the result of starting each iteration with a single developer being assigned to a single feature, even if it's huge and complex. As developers finish the easier features, they roll over and help on the bigger ones or work on code improvement.

The finisher is a role on each iteration filled by a single developer who is not tied to a specific feature. Instead they are dedicated to fixing bugs, code improvement, and reduction of technical debt (or any other task that makes it more of a finished product). The developer assigned to this role changes every iteration.

Intridea Forge in Detail

Since a picture is worth a thousand words, here's a graphical overview of how an iteration progresses.

• Release Iterations

  Iterations in Forge don't have a fixed length. Instead, iterations are tied to the completion of all of the features slotted for that iteration, regardless of how long it takes. This means that iterations will rarely be the same duration as they are pegged to functionality, not a specific schedule. At the end of each iteration you do a full production release, complete with feature announcements and all of the other fanfare.

  This is a bit unconventional in the Agile world and it may sound more like traditional waterfall, but I promise that it's not waterfall and brings quite a few benefits with it.

• Developer Ownership

  At the beginning of the iteration, all but one of the developers on the team are assigned to work exactly one feature each and to work it alone. The developers will then own the feature they start on for the duration of the iteration.

  Since not every feature in an iteration will require the same amount of time to complete, there will be developers who finish their features “early” (it's not really early and shouldn't be regarded that way; some features will be more complex and will take longer and no judgement should be placed on how long it takes to complete). When a developer finishes their feature (and all supporting test and documentation), then one of two things happen.

  If all of the other features are progressing without any issues, then the recently freed up developer simply gets added to the finisher pool (see below). If any of the features are lagging, complicated, tedious, or otherwise in need of help, then the project manager will reassign the developer to help with that feature.

  Let’s say Sally is working on a difficult feature and John completes his feature early and shifts over to help Sally finish. One of the advantages of this process is that by the time John comes in to help Sally she will have already have gotten started on the feature and should be able to easily guide John. At this point Sally will know what needs to be done and how it needs to be done and can make effective use of the additional help.

• Finishers

  Most of the development team is assigned a specific feature to work on for the iteration. The extra developer is assigned to a rotating role called the finisher.

  The primary responsibility of the finisher is to respond to critical popup issues. Such issues might include bugs that were recently introduced and missed during QA, a minor but critical new feature, or other similar items. Any time that the finisher isn’t working on these popups he or she can work on anything that they think needs to be done to make the project better. Examples may include:

  • Fixing old bugs
  • Adding documentation
  • Improving test coverage
  • Refactoring difficult code
  • Modularizing features into components that can be reused and/or released as open source

  Each iteration, the finisher should be a different member of the development team. The rotation of this role is essential and provides a few advantages. First, it keeps developers from getting stove-piped into a single part of the system or a single type of development. It gives everyone the opportunity to dig into other parts of the project. It also ensures that everyone has a chance to clean up or rework some piece of code they may have run across on a previous iteration.

Key Events

During the iteration, the management team has a responsibility to be setting up the next iteration for the team. This is handled by several important meetings.

• Iteration Kickoff
  Who: Development team, project manager

  At the start of the iteration, the project manager meets with the development team and describes the features for the upcoming iteration. During this meeting, the team members are assigned to the various features and the finisher is identified. This is also an opportunity for the development team to get details on the new features and clarify any questions they may have.

  At this meeting, the development team will also give their estimates for how long the features will take. This will be relayed back to the management team and other stakeholders so that they know when to expect the end of the iteration.

• Release announcement
  Who: Management team/stakeholders, project manager

  After all of the work from the previous iteration is complete, the management team meets to schedule the release and plan any downtime. They will review the new features, prepare newsletters, blog posts, etc. at this meeting. After the release is planned, the deployment team will perform the production deploy.

• Mid-Iteration Review
  Who: Management team/stakeholders, project manager

  Based on daily reports from the development team, at this point the project manager informs the management team of the status of the iteration so far. The estimate for completion will be updated and any features that are available on staging can be demonstrated. This meeting can occur multiple times during the iteration.

• Initial Planning Review
  Who: Management team/stakeholders, project manager

  Throughout the iteration, informal planning for the next iteration should be ongoing. The initial planning review is the first occasion where the management team meets to formalize plans for the next iteration. At this meeting, the project manager will collect inputs and rank upcoming features. He or she will also start to flesh out the feature requirements so that estimates can be made and dependencies can be addressed.

• Dependency Review
  Who: Project manager, technical management, lead developers

  After the first pass at defining the next iteration is completed, a dependency review is held. This is the opportunity for the various inter-dependent teams to ensure that everyone will be accommodating any upcoming changes. For example, if the web site team will be making a change or adding a feature that will affect an API used by the desktop application, then this meeting should be where the project managers for the two teams meet to discuss the implications and deployment strategies.

• Final Planning Review
  Who: Management team/stakeholders, project manager

  Leading up to the final planning review, the project manager should have placed initial estimates on all of the features and thoroughly fleshed out the requirements. The final planning review meeting gives the management team one last opportunity to change course on the next iteration before it is locked it. Timeline estimates are reviewed and everything is agreed upon.

Throughout the iteration, informal planning should be happening to lay out upcoming iterations. As the iteration progresses the management team will also be able to see previews of some of the new features as they are made available for testing on staging. This helps ensure that the capability is implemented as intended and satisfies the curiosity of the management team to see what is being built.

Conclusion

This post has given you an overview of Intridea Forge. In the final post in the series, we'll tell you about why it has worked well for our team and we'll share some of the challenges that we faced implementing it. We'll also give some tips and lessons learned about how to implement it on other projects.

• Read more about Redesigning Agile: Part I - Challenges

One of the best comments I've heard about Agile development is that "adopting Agile is itself an agile process". It sums up very nicely the fact that each team and each project will have their own needs when it comes to finding a process that works for everyone, including the stakeholders. It also means that no matter how much you like a specific methodology, it's virtually impossible to implement that process on your team following its strict definition.

As a Scrum Master I've always been partial to Scrum, but after spending six months trying to make it work on one of our large client projects, I reevaluated and started from the ground up to design a whole new Agile process. We'll be introducing our new process in a future blog post, but for now I wanted to talk about some of the challenges we faced implementing Scrum.

There was nothing specific that was wrong with Scrum for our project. We just ran into enough minor hiccups that we started to lose some of the benefits that we should have been able to reap by using it. Before starting, I needed to figure out exactly what wasn't working (and also what was working well) with Scrum so that those issues could be addressed by our new process.

Here are some highlights of our challenges:

• Missing or ambiguous roles

  Scrum works best when a strong Product Owner and a strong Scrum Master maintain a bit of tension between them. The PO needs to demand more and the SM needs to keep things reasonable. The result is an aggressive schedule with a great deal getting done, but development kept to a sustainable pace.

  This can break down if your Scrum Master needs to do a lot of the product planning and prioritization (or isn't involved enough in the planning process). Likewise you can also run into significant problems if you have trouble finding someone who is both empowered and knowledgable enough to be a strong Product Owner.

  Another potential issue is that as painful as it may be, priorities sometimes need to be set by a committee. If the client can’t abide by a single person who selects the features for the upcoming iteration, the process needs to be flexible enough to support that. This also means that if the person who is most responsible for setting priorities is unavailable, then one or more surrogates can easily step in.

• Self-organization doesn't work for every project

  Even when you have competent and self-motivated developers, not all teams work well when attempting to self-organize. For our team, there were a few factors that made it a challenge.

  First, we were regularly adding and removing people. As other projects spun up or down, we would have fairly regular shifts in team composition. While we always had the same number of developers, we frequently had new developers coming on who were new to the system. Since our product is highly complex, they simply could not figure out which of the backlog items from the current iteration were good ones to try to work on.

  Second, with a team spread out across the world, real-time collaboration is hard. In our case we have developers across the US and China and were working with client contacts in every US time zone as well as Spain and South Africa, making it impractical to rely on instant availability. The consequence is that your team needs to be more autonomous and focused on working independently. In my experience, self-organization works best when the whole development team is collocated and able to work with one another face-to-face.

  Last, the client will sometimes need more detailed insight into who is working on what. In traditional scrum, having a backlog story hanging out and unassigned at the beginning of a iteration can make some clients uneasy.

  To get around these issues, we found that the project manager for the development team needs to assign tasking to provide focus and avoid issues with non-real-time collaboration. This needs to be done in such a way that the clients are aware of the assignments, but the developers are still empowered with technical and design decisions without needing approval.

• Rigidly timed iterations can break velocity

  There are a number of situations where having a fixed-time iteration doesn’t work. In the web development world, you will often have short (e.g. 1 week) iterations. Many features easily fit into one iteration, but often a large feature needs to be spread across multiple iterations and simply cannot be broken down into concrete backlog items that can be deemed complete during a single iteration. Also, changes in client schedules, shifting personnel, and unexpected things like conferences, holidays, etc. can make iteration timing problematic.

  Scrum requires a stable team that has a consistent velocity. That velocity is used to determine how much to tackle in the next iteration. The problem is that a team’s velocity is based on the past performance of that team (i.e. how much they got done during the last few iterations). When you have a team that either changes frequently because of personnel or unstable iteration times and shifting customer requirements, then the velocity can become unstable.

  When these factors affect velocity, your most important planning metric becomes meaningless.

• Better transparency into release schedules

  With web development, you often end up in a cycle where major new features come out each iteration and marketing of those features can get lost in the noise. We need a way to say “the next major version of our site is online and it includes features foo, bar, and baz”. The client needs an opportunity to brag about progress and plan ahead for potential downtime.

• Incorporate a mechanism for QA and bug fix cycles

  There needs to be a clean way for the QA team to be fully integrated into the iteration cycle. The acceptance criteria and the planning of regression tests need to be a core part of the process while not holding up the development team. If you have separate QA and development teams then your developers can end up sitting around waiting for QA at the end of a sprint in Scrum.

  At the same time, there needs to be a real dedication to continuous code improvement and bug fixing that isn't dependent on the Product Owner deciding to prioritize those fixes.

While accommodating all of these changes, we still needed to harvest some of the benefits of Scrum and other Agile processes.

Even though we were looking for a process that would allow our clients more insight into who was working on what, we still wanted our developers to be able to focus on the task at hand and not be concerned with shifting priorities or popup crises. The client needs to remain hands-off and the process needs to be set up to make that comfortable for both the client and the development team.

Obviously one of the biggest advantages with Agile development is that you are able to respond to changing requirements (the idea of dropping “ready, aim, fire” in favor of “ready, fire, aim, aim, aim, aim, …”). The new process needs to support that as well or better than the process it is replacing.

These are some of the most important issues that we needed to incorporate into our new process. In the next post in this series, we'll actually describe our new process and how it works. After that, we'll include some tips for implementing it on your project and some of the challenges that might come up.