• Read more about The Pros and Cons of An Agile Engagement Model

Seventy-one percent of organizations use Agile to drive their app development process. By now, most of you have already heard agile-fanatics screaming from the rooftops and corporate execs using the word "agile" as if it's the most modish term on the planet. But what is agile, really? It can be difficult to wade your way through the dense forest of hype artists and the trendy alphabet soup of "agile-like" acronyms to discover the real, tangible meaning of agile.

So, let's cut through the red tape, ignore the trendy buzzwords, and uncover the truths of agile. No. It isn't a catch-all development solution, and it certainly isn't perfect for every development project. In fact, agile has some serious downfalls that seem to be excluded from those 25-page long "Agile Manifestos" that developers and companies are putting out at the speed of light.

To be clear, we're agile "fanatics." We're SAFe certified, and our entire motto (i.e., PUSH) is driven by agile-thinking, and we've used agile methodologies to execute massive-scale projects like NASA.gov 2.0 and The United Service Organizations' app. However, we're going to take a step back. Let's put away the biased hats, ignore the hype, and dig into the nitty-gritty of today's most popular engagement model. Here's everything you need to know about the agile engagement model.
 

What is an Agile Engagement Model?

Engagement models are frameworks that define the relationship between an outsourced development team and your company. In other words, engagement models are "how" your app is getting executed, and your project is being delivered by your development team. There are a wide variety of engagement models, and each of them has specific pros and cons.

Agile engagement involves rapid-fire iterations, immense flexibility, and plenty of collaboration. The overarching goal of agile is (as the name suggests) marketplace agility. So, when the environment outside of your development team changes, agile gives them the tools to quickly react to those changes. Typically, agile development teams have daily meetings, use tools like Kanban boards to execute bite-sized chunks of development, and have the flexibility to rapidly change requirements and needs based on internal and external factors.

In today's fast-paced, digitally-drenched software development lifecycle, agile brings a ton of value to the table — especially on long-term projects that have to cross that oh-so-scary "pit of scaling." According to PMI, 70% of organizations use agile development today, so it's certainly a popular and results-bearing approach to software development.
 

When Does Agile Make Sense?

Since agile requires strong cultural structures, unparalleled collaboration, and iterative-driven strategies, it's best for outsourced projects that leverage in-house teams. In internally-driven development cycles with in-house teams, agile is often used to execute from start-to-finish. However, when you're leveraging outsourced developers, start-to-finish projects may work better with more traditional engagement models like firm-fixed — since they prevent cost-traps and unnecessary cost scaling.

Agile really starts to shine when either:

In the first scenario, your outsourced team will send boots-on-the-ground to your location (or Zoom-on-platform in today's ecosystem). These are people who integrate themselves into your business, so you can push cultural and collaborative requirements onto them. In the second scenario, it doesn't matter as much if you have in-house or external outsourced teams — since you don't have to worry about cost-traps or over-the-scope project requirements. To be clear, agile is very powerful in these two situations. In fact, we would argue that agile can completely change your delivery cycle and help you create more fantastic customer experiences. But there are also plenty of situations where agile isn't strong.
 

Understanding the Pitfalls of Agile

We love Agile. Being SAFe certified, we leverage agile to execute massive government projects and huge client apps. But, despite the hype, there are very real situations where agile simply doesn't make sense. In particular, agile engagement models — meaning you are using outsourced development to execute and grow a project  — are tricky to leverage on from-the-ground-up projects. For starters, agile's iterative and flexible processes don't lend themselves well to budgetary projections. You don't get a firm, upfront quote like you would with firm-fixed. Since project requirements change throughout the lifecycle, agile can breed unpredictability and budgetary concerns into off-the-ground launches (i.e., costs change with the growth of the project and as new demands hit your outsourced team.)

Also, there's a tangible scope issue. Agile development doesn't have a set-in-stone scope. So, it's easy to overthink and over-scope projects with agile. You may start feeding new requirements to your outsourced team on a regular basis. And while these requirements may be things you desperately want, they also incur additional costs. Over time, your budget may grow out-of-control. This can also cause relationship frictions between you and your outsourced partner. The raw flexibility and scalability of agile are exactly what makes it so dangerous for some projects. Requirements can spiral out-of-control.
 

Which Engagement Model Works for Your Business?

There's no such thing as a perfect engagement model. But there is a perfect engagement model for your unique project. Obviously, agile is the most hyped, utilized, and championed engagement model in today's fast-paced development ecosystem. And for good reason. It's amazing. But it isn't ideal for every project. Unfortunately, the disadvantages of agile often get drowned out by the agile fanatics. Not every project fits with agile, especially full builds that leverage outsourced talent.

Will agile work for you? It depends! If you're looking to support an ongoing app or bring over an internal team of outsourced experts, agile could help you execute faster, smarter, and with more purpose. But if you want an off-the-ground app via outsourced talent, agile may put you in a dangerous cost-cyclone that's detrimental to your finances, forecasting, and relationship with your outsourced solution. Are you looking to create an amazing project?

Contact us. We have the skills, experience, and frameworks you need to execute projects across a variety of engagement models. From firm-fixed to agile and beyond, we're here to hand-pick the best solution for your business.

• Read more about How Can Cloud Computing Companies Transition From Agile To DevOps Principles?

Transitioning from Agile to DevOps 

Some ask if transitioning from Agile to DevOps principles requires a new consideration of deployment and hosting infrastructure. Many have been asking the question of "what are some best practices for companies that want to address the concern of how cloud computing companies transition from Agile to DevOp principles?" In actuality Agile and DevOps can be combined versus having to choose one methodology over another.

Check out the below Q & A to learn how they these two methodologies can complement each other. 

How can cloud companies best address the concerns of Agile and DevOps?

It is a misconception that Agile and DevOps principles have tension with each other. Rather, they each encompass a separate set of principles that apply to different parts of the software development life cycle.

Agile and DevOps principles can be successfully blended to ensure reliable, rapid and on-time deployments of working software.

Agile methodology stresses the immediacy of working software over comprehensive documentation, and embraces constant change pursued through short, rapid iterations of software development rather than well-defined "final" products.

DevOps, on the other hand, governs how the resulting software is tested, secured, deployed, and maintained as seamlessly as possible. DevOps is not an alternative or a response to Agile, but is best seen as a complement to Agile, which allows rapid release cycles that are secure, reliable and error-free.

What are the best practices and key advice for progressive IT teams?

Implementing DevOps often requires constructing a toolset based on cloud computing models, in that it advocates automation and repeatability of every aspect of the deployment process from the moment new code is committed to a project.

Continuous integration workflows are then constructed, using a variety of scripts and automation tools like Jenkins, which build the cloud components necessary to serve the application, configured from a central repository (using a configuration management suite like Ansible, Chef or Puppet) ensuring each deployment is identical, therefore minimizing the potential for human error.

Automated testing is another critical ingredient in the DevOps toolkit. From unit tests which confirm the functionality of individual snippets of code, to functional and integration tests, which verify that functional requirements are satisfied by the latest release without regressions.

DevOps thinking ensures that every deployment is as error-free as possible, all without the heavy workload of constant manual testing.

This focus on testing and infrastructure-as-code can align well with Agile's focus on rapid deployment in that it automates the most time-consuming portions of the software release process and allows developers to spend less time worrying about bug fixes and environmental differences, and more time implementing new features.

Simultaneously, it allows product owners to be confident in deployments, knowing that automated test cases are constantly checking for regressions, and CI/CD (continuous integration / continuous deployment) processes will reject a build that contains errors.

The fact that the cloud infrastructure is created from code at the time of the deployment is an important check that applications will function identically in test/development, staging/QA and production environments.

How can a new DevOps structure be best adapted for developers who have been using Agile for a while?

From a developer's perspective, implementing DevOps requires little change in workflow from an existing Agile mindset. The same focus on rapid deployment exists and work can be broken into Agile sprints or scrum periods according to the needs of the team and product owner.

The additional expectations that DevOps places on developers is that all committed code will be automatically tested for unit (functional and integration performance before being automatically deployed) and rejected if it does not pass all regression tests.

Thus, developers may need time to embrace a TDD (test-driven development) approach and write tests as a prerequisite to building the code that satisfies them.

Should there be an overlapping period of Agile and DevOps and if so, how long should the overlap be?

Since Agile and DevOps are not mutually exclusive, they can be blended together over time, with more DevOps thinking added onto a functioning Agile workflow to continue to narrow the gap between development and operations.

Both approaches stress communication within the team, although there are cultural disagreements about the level of specialization (Agile stresses that each member of the team be a jack-of-all-trades while DevOps tends to allow for more specialized roles such as systems architect, security expert, etc.) and the best way to schedule work: Agile favors dividing into short, rapidly repeated time chunks while DevOps focuses more on stability over the long term.

Again, these approaches are not contradictory but can be blended to ensure that software is delivered as rapidly and reliably as possible.

The level of automation involved in a DevOps workflow may be unfamiliar to developers who are new to the methodology, but it soon becomes apparent that all of the automated testing, code verification, and deployment processes can ultimately free developers up to do what they do best, which is build new features for the product owner.

What tools do you use or software solutions?

Mobomo works with both Amazon Web Services (AWS) and Microsoft Azure as our primary providers of cloud services. We utilize our deep experience in open-source technology to deliver DevOps toolsets based on Linux with Ansible code-based provisioning, and orchestration with Jenkins.

Automated functional testing is done with Selenium coupled with Gherkin-based test frameworks such as Behat and Lettuce. We rely heavily on scripting languages like Python, Ruby and Bash to connect these toolsets together and provide a robust DevOps workflow to deploy code seamlessly and reliably, on a velocity that is fully compatible with Agile best practices.

What are your yardsticks for success?

Time of release from code commit to production readiness; client acceptance of new features; number of bugs caught by automated testing frameworks for each release; number of bugs missed by automated testing and reported after the fact; total downtime, service interruptions and other SLA violations resulting from unanticipated infrastructure issues, either deployment-related or due to poor planning; etc.

How long do you pursue a DevOps strategy before calling it off as a failure and moving back to Agile? Is there no turning back?

Given that Agile and DevOps are not mutually exclusive, there is no need to "call off" a DevOps transition and go "back" to Agile. Rather, they can be combined in ways that reinforce each other, with Agile functioning as the development team's methodology while DevOps provides the backbone for the cloud deployment, security, networking and testing aspects of the engineering process. Thus, disruptions are minimized by DevOps which allow Agile to provide rapid iterations of working software.

• Read more about Agile Scrum Methodology

We were thrilled to do a Q & A with App Developer Magazine on Agile Scrum Methodology. Check out the piece below.

What is Agile Development Methodology?

Mobomo: The Agile Methodology is an alternative approach to project management that places emphasis on collaboration, iterative development and evolving requirements. It’s governing principles, commonly referred to as the Agile Manifesto, place focus on team interaction, customer involvement and adaptability to develop high quality products.

Who uses Agile scrum development?

Mobomo: While Agile is most commonly associated with software development, it’s leveraged across many industries in both the private sector and government and is not limited to just software. From General Motors and Spotify to NASA and the Department of Homeland Security, Agile can be applied to solve some of business’ most challenging needs.

What is the process of Agile scrum development?

Mobomo: Scrum is subset of Agile development that organizes a project in a consistent process that is repeatable each iteration of the project lifecycle. The most widely used Agile practice, Scrum is ideal for managing complex software and product development as it enables organizations to more easily adapt to rapidly-changing requirements. Scrum also provides a better framework for estimating tasks and measuring productivity, thus allowing for managing the project schedule more effectively.

What are the key positions and what are they responsible for during process? (Scrum master, product owner etc.)

Mobomo: Within the Agile Scrum framework, the key positions and their role within the project are as follows:
Product Owner: The Product Owner is the main stakeholder in an Agile Scrum project who is committed to ensuring the vision of the project is reflected in the final product delivered without impeding upon the Agile Scrum process. The Product Owner assists in transcribing the product vision into well-defined development user stories and prioritizing the user stories to be worked on by the Development Team.

Scrum Master: The Scrum Master is the facilitator for an Agile Scrum Team, managing the process by which information is exchanged amongst members of the team. The primary responsibilities of the Scrum Master are to help the team determine what can be achieved during each iteration cycle, manage the daily scrum meetings, keep the team focused by removing obstacles to their progress and protecting the team from outside distractions.

Development Team: The Development Team, or Scrum Team, is a collection of individuals responsible for all work delivered to the customer. The team works together each iteration, or sprint, to incrementally deliver working components of the product.

How do federal projects benefits from using Agile scrum development?

Mobomo: Within the Federal Government, implementing the Agile Scrum methodology for software development enables the government to solicit services using more broad requirements about the tools and technologies involved the development of the desired product as opposed to specific product requirements. This allows the Development Team to evaluate the business requirements of the product as part of the contract and identify the best technical solution to meet business goals, leveraging both licensed and open-source software. Agile Scrum also provides a more structured framework that supports efficient estimation of the tasks required to deliver the product used to effectively develop an accurate project schedule.

Lastly, the incremental development and delivery of the product to key stakeholders ensures that the end product meets or exceeds the expectations of the government customer. Ultimately, the key benefit is that the government is able to get a better software product through iterative development as opposed to a software product that merely complies to functional requirements that were not completely understood at the time of the RFP.

How does this methodology help to save the government money and time?

Mobomo: The Agile Scrum methodology directly benefits Federal customers both financially and with respect to project schedule in that the project team and the customer work together on estimating the level of effort required for delivery of the project. As a team, they and are able to shape the project requirements and technical approach to fit within the constraints of the contract’s period of performance and budget. The iterative development showing product progress every week provides a high level of transparency between the project team and the customer mitigates the risk of the project extending beyond the scope of the engagement and ensures the project team isn’t developing features that do not provide value to the product.

What type of projects use the Agile Scrum Methodology?

Mobomo: The Agile Scrum methodology can be applied to any project where the development of the final product can be managed through incremental development and delivery of the product to the customer. Whether it’s mobile applications, websites, microwaves, or automobiles, the principles of Agile Scrum can be applied to achieve positive results. The most common application of the Agile Scrum methodology is for the development of highly complex software products that require both a strong process-oriented management structure and flexibility of the development team to respond to fast changing requirements.

What are some tips to start using agile scrum development?

Mobomo: The best thing an organization can do to start adopting Agile Scrum for the development of their products is to have an open dialogue with their team to share the principles of the framework and illustrates the benefits of implementing Scrum. Without buy-in from the team, it would be extremely difficult to institute such changes effectively. Next, avoid trying to incorporate all of the Agile Scrum practices overnight. Agile Scrum is a framework through which a project is managed following the core principles of Agile. It is by no means a turnkey solution that works the same for every organization, rather a set of guidelines and tools to be leveraged for impactful results. Lastly, engage an individual or organization experienced in leading Agile Scrum projects on how they leverage Scrum for their needs and gain as much knowledge as you can from them.

In order to run a project using agile scrum methodology, do you need to be certified, if so what is the process?

Mobomo: It’s not necessary for every project manager in your organization to be certified Agile Scrum practitioners, nor is it necessary for anyone within the organization to be certified. However, it's highly beneficial to have a few members of your team familiar with the principles of Agile and specifically Scrum. There is a multitude of open source documentation and training materials available to the public. For certification, organizations such as Scrum Alliance manage online examinations and membership. There are also a number of organizations that offer formal training sessions with high success rates for individuals that complete the course and the certification exam.

What advantages does your project receive with agile scrum development? What are the disadvantages?

Mobomo: The advantages of your project adopting Agile Scrum to govern the development of your products include instituting a repeatable process that seeks to continually improve each iteration of a project and beyond to future projects, promote self-organizing teams who are able to delegate tasks amongst themselves in a way that conforms to their style so long as it doesn’t encroach upon the team’s ability to deliver each iteration, and maintains transparency between the project team and the customer. Agile Scrum also mitigates the risk of scope creep and delays to the project schedule by identifying obstacles proactively and remediating them accordingly. The disadvantages of Agile Scrum are that it requires a certain degree of maturity within the organization to empower its development teams to be self-organizing, which has the potential to negatively impact a project if the team isn’t closely monitored by an experienced Scrum Master. Additionally, an Agile Scrum project requires having a healthy backlog of tasks for the Development Team to ensure no lapses in productivity occur, which is dependent upon the Product Owner having a clear vision of the product and his/her ability to effectively communicate and prioritize the tasks accordingly.

How do you organize your tasks and then prioritize within the process?

Mobomo: Within Agile Scrum, the Product Owner works with the Development Team to translate the goals and needs of the product as identified by the key customer stakeholders into development user stories. The user stories are then organized within the product backlog based on priority and underlying dependencies to identify the order in which tasks need to be worked on. After the priorities are established, the Scrum Team estimates the level of effort required to complete the individual user stories and begin organizing them into iterative sprints based on the amount of time the team has available to work in each sprint.

What happens if an unexpected task occurs, how do you organize that task within the current sprint?

Mobomo: In the event that an unexpected task is identified by either the Development Team or the Product Owner during an active sprint, the task is communicated to the entire team who then conduct an assessment of the implications, priority and level of effort involved for the task. If the task warrants being addressed immediately within the sprint, the Product Owner will determine which task(s) that were originally scoped for the sprint will be removed to accommodate the new task being included. If the task does not warrant immediate attention, it is moved to the product backlog for prioritization and inclusion in a subsequent sprint.

• Read more about Agile Development Methodologies

While agile development methodologies and practices have long been part of common practice in private tech and product companies, many U.S. government agencies are still getting their feet wet with fully embracing the concept and finding ways to make it work within their complex contracting arrangements with private vendors.

Under Mobomo’s contract working with the U.S. Geological Survey (USGS) under our contract with the U.S. Department of Interior (DOI), we’ve been fortunate to work for a client who not only understands agile development, but requires it for the development of the USGS.gov agency website, and the underlying AWS cloud-hosted, Drupal-based content management system (CMS), called Palladium.

We’ve been fortunate to work for U.S. Geological Survey (USGS), a client who not only understands agile development, but requires it for the development process. Having a willing client partnership is key to making our scrum processes successful. It’s an evolving path forward as we work with USGS to continue to improve on our roles of scrum master and development team as well as coach the USGS Web Re-engineering team in their role as product owners. But, the process is working! Not only is it working but we’re able to show how the scrum process is helping us to get closer to continuous development/continuous integration (CD/CI), allow us to deliver new features on a regular and frequent basis, and is saving the government money and time.

Our Path to Agile at USGS: Transparency and Building Trust 

When we first started this project, the only real semblance of agile processes was the Jira ticketing system. It served as a backlog, but there was no process in place to identify which tickets would be included in an upcoming release and no definition of what a sprint and release process looked like within the project scope. It was difficult to comprehend, there was a huge rush to get the first version of the CMS and the new rebrand of USGS.gov up and running - but it was also a bit unmanageable given the system that was in place.

First, we worked out a sprint cycle which has evolved from a sprint every two weeks to a schedule where we have two, two-week sprints that overlap. This allows for ongoing development during code freezes and more QA time. The schedule flows like this:

The next step was to improve our communication requirements with our product owners. We were all onboard with using Jira as our ticketing system, reviewing tickets from the backlog to move them into sprints, and then pushing releases based on the completed tickets. But, too often, we found that tickets had to be re-opened after the release. The problem boiled down to communication. We needed a consistent format for how bugs and features were being communicated to us, so that we were fixing the right problems and building features based on user needs.

To solve this, we created templates for how a bug would be entered as a ticket in Jira. A sample format of questions would be as follows:

What is the URL?

How is it behaving now?

Steps to reproduce this issue.

How should it behave?

This was a huge improvement! But, we were still not where we wanted be on our releases. Next, we tackled how QA fits into our process, we revised our workflow to make QA an integral part of the process from the time a ticket is self-assigned by one of our developers, and we built out our Jira workflow to represent this new process.

Do you use agile scrum development? If you do, tell us what you think about our process!

[contact-form-7 id="12689" destination-link="https://mobomo.s3.amazonaws.com/uploads/2016/11/mobomo_brandbook_.pdf"]

• Read more about Agile Project Management with Scrum

Agile, sprints, scrum - all these words are thrown around endlessly in software development. So, what are they and how do they work? Before embarking on a 10 hour passenger ride to Georgia, I picked up Microsoft’s Agile Project Management with Scrum from our Mobomo library to give a better perspective.

What is Agile and why should I use it?

No secrets here, agile is what it says - agile. Agile is a type of project management different from the traditional waterfall style in which company X is obligated to produce a product for customer Y by a set deadline. Unlike Agile, the waterfall method doesn’t allow much, if any, room for changes within the dedicated timeline. As a Project Manager, I know that customer and business needs change with no regard for your timeline. At Mobomo, we understand that change isn’t bad, it’s awesome so we should make room for it. Agile helps your development team balance changes with your customer and business needs.

Ok, cool - now I know that agile is flexible, but why is it good for business?

Agile saves you time and money. Software development is complex, but it’s predictable in itself. People, on the other hand, are not. Agile allows for this margin of unpredictability with a defined, empirical process called Scrum. Using Scrum, your product can be built with a focus on a minimum viable product. Creating a product with a minimum set of features provides value by testing your business’s concept before continuing development. This iterative approach builds on your highest priority features (i.e. sprints) and saves you time and money by building a product that people want to use and pay for over a product filled with excessive, unvalidated features.

I particularly enjoyed Schaber’s book because it describes real, lessons-learned use cases without having to experience failure first hand.

Tell me more about Scrum!

I love Scrum because it’s intuitive and fact-based. It’s foundation empowers the development team while also putting customers first. Managers love it because its style is rooted in helping others achieve their goals in a common-sense learning process.

Here at Mobomo, we’re always employing Scrum in our Agile Methodology to get our clients the best product possible with value in mind. I could go on and on about Scrum and Agile, but I hope this a valuable insight to our goals and processes at Mobomo!

• Read more about Intridea Client, Demosphere, Launches Mobile Score Reporting App

Demosphere, a leading provider of web-based administrative tools for youth sports organizations approached our mobile development team to help give their users mobile accessibility. For some time, they have provided an IVR interface to youth sports organizations to phone in results of soccer matches via cell phone voice commands. Demosphere wanted to break into the mobile app space because they knew that's where their users were; so, they asked us to assist with creating custom mobile applications for Android and iOS devices.

One of the primary directives was to give the mobile apps functionality and aesthetics similar to the system Demosphere users were already accustomed to. Our team worked closely with Demosphere to define a look and feel that would mimic the experience offered by the already popular voice system. We were able to utilize the structured interactions of that system to quickly and elegantly develop an Android and iOS solution that provided the same login and interaction system as the voice interface, coupled with a familiar and stylized input system.

The UX team worked directly with the client to create blueprint documents that covered the use cases and interactions that would be most straightforward for their users. Following that phase, they worked with their marketing department to define a UI that was on target with their brand.

In working with our vision for the user experience, German and Yincan engineered the development of the Android and iOS applications.

We employed a Kanban agile process for design and development; this lean development practice allowed us to consistently deliver in-process working versions of the applications and adjust to Demosphere's feedback as we developed. Following a few short weeks of development, Demosphere recently announced PhoneItIn for Android and iPhone.

PhoneItIn™ is Demosphere's revolutionary score reporting system. Rather than reporting scores in front of a computer, you can now report scores on the go from your mobile device. For those with iPhone or Android devices, a free app allows easy score reporting with no phone call required!

Ashley Ralph, Demosphere Blog

The whole process went smoothly thanks to utilizing web-born development practices and a tight feedback loop. Congratulations to the Demosphere team on penetrating the mobile app market!

• Read more about Always Be Releasing

It's hard to be "on" all the time as a developer. Despite its engineering aspects, I find software development to be a fundamentally creative act, and creativity requires inspiration. Looking back on the times when I've felt in a slump, when I just couldn't get the energy to make the next big push I needed for whatever project I was working on, I came to a simple realization: I burn out when I'm not releasing.

For me, if I'm not releasing something new at a pace of about once a week, my creative fire is slowly fizzling out. It doesn't really matter what it is; it could be a new open source library, a great (and complete) feature for a product I'm working on, or even just an in-depth blog post. What I need from releasing is the release that it gives me: it gives me license to take a breath, feel satisfied, and let my brain tackle new problems.

If you're feeling burnt out, take a step back and think about something you can release this week. Then do it. If you're really burned out, make it something small, maybe even just a Tweeted gist of some interesting code. Release it, feel that satisfaction, and you might just find that you suddenly have the inspiration and energy you needed to get over your harder, more complicated challenges.

When you really think about it, agile development isn't simply a method for building better software with a tighter feedback loop. Fast iterations and constant releases also serve as an energizing force for a development team, a way to feel that at least something is finished. Perhaps this insight is nothing more than common knowledge, but I know that it's going to change the way I think about my schedule. It's not "what am I working on this week" but "what am I going to release this week". Maybe it'll help you, too.

• Read more about Redesigning Agile: Part III - Putting Forge to Work

In the first two posts in this series, I explained some of the problems that we had with traditional Scrum and then laid out our new Agile process: Forge. In this post, I'm going to share a bit more about implementing Forge on our project and tell you about some of the benefits and challenges that we ran into.

The Benefits

Forge has solved a lot of problems for our team. Here are some of the most significant benefits we have seen.

• Getting rid of technical debt

  One of the biggest problems with any project is the accumulation of “technical debt” (TD). TD is simply bugs and complexity that get introduced as a project gets larger. Things get even worse if you aren’t starting from scratch and will be working with someone else’s code.

  The advantage of having a finisher (or many finishers towards the end of the iteration) is that you have a dedicated developer whose primary job is reducing TD. Few other processes even acknowledge that you accumulate TD, let alone insist on a mechanism for reducing it. Over time, the net result is a more stable, more maintainable project. Ideally the process will also result in a robust library of modules and/or open source components that you can tap for enhancements and future projects.

• Developer focus and ownership

  Since each new feature is assigned to a single developer, he/she develops a sense of ownership of that feature. There are several key benefits you get from this. First of all, since the implementation details are left entirely up to your developers, they are granted a level of autonomy and freedom of design that is very rewarding.

  Second, since each developer will begin an iteration as the sole person responsible for a new feature, by the time another developer is available to help carry it to completion, they will be very easily integrated into the task at hand. This dramatically reduces potential conflicts and improves productivity.

• Clearly defined release feature set

  After a final QA check at the end of each iteration, the product goes through a full release. This process gives the team the opportunity to roll out new features as part of a complete release that can be communicated to users and partners. This vastly improves transparency and gives the customer the opportunity to promote the new release and also alert users of potential downtime.

The Challenges

Despite these benefits, Forge is not a silver bullet. There are several challenges that we have run into and had to either work around or adjust our expectations.

• Delayed gratification

  One of the biggest roadblocks we faced was that our clients weren’t happy with the fact that they would need to wait for completed features. For example, if Sally completes a new feature four days into the iteration then it can be hard to explain why they have to wait another week for it to go to production. Over time the benefits of having a clear release announcement have outweighed this concern, but it can still be a hard sell for some organizations.

• Long-running features

  Some features can also be difficult to implement and thus take far too long to tie the delivery of an entire iteration on its completion. For example, if you have six features slated for an iteration and one of them is expected to take four weeks but the timelines for the others are measured in days, then it doesn’t make sense to have most of your development team acting as finishers for three weeks while you wait for the completion of the difficult feature.

  Features that are going to take a long time to complete should typically be known during the planning phases. When one of them is on the list, the simplest thing to do is to let it carry over multiple iterations. When the feature is finally completed, the developer working on it just becomes a finisher for the remainder of the iteration as they normally would.

• New projects

  Forge was created for a project that we inherited from another development company and so it came with a great deal of existing technical debt. In addition to simple code complexity, we had to deal with a large number of bugs from the first day on the project. This makes it easy to figure out what the finisher should be doing during that first iteration, but most projects aren’t "rescue" projects and the team will be starting from a blank canvas.

  Forge obviously still works in those situations, the only tricky thing is to figure out what the finisher should be doing early on. There are two different approaches that can be taken. First, you can leave off the finisher during the first iteration or two and only bring that role into play once you see yourself starting to accumulate technical debt. This can be risky because you may end up in a situation where the finisher never actually materializes, so we recommend that you explicitly limit the number of iterations without a finisher to one or two from the outset.

  The other possibility is to task the finisher early on with some of the infrastructure-oriented tasks. Configuring servers, deployment scripts, and continuous integration systems are all things that need to be done early on and may be a good place for the finisher.

• Small teams

  Forge assumes that you will always have one person dedicated to fixing bugs, but on projects with very small teams (e.g. 2-3 people) this can seem like a burden. The truth is that it probably is a burden for those projects. In these situations, you will need to tailor the process a bit to suit the circumstances.

  There are a number of ways to address the issue. Some ideas that pop to mind are only including the finisher role every other iteration or requiring that the first person to complete their task always becomes a finisher instead of rolling on to help with another feature. Each project should address these challenges in the way that suits the team and the client best without worrying about rigidly sticking to a process.

• Multi-finisher iterations and blitzes

  There were times on our project where we found ourselves falling behind on technical debt, even with a dedicated finisher on each iteration. This was most apparent when a new feature that we released turned out to have unforeseen consequences across the product. We found that sometimes we needed to run an iteration where we were less focused on features and more focused on finishing.

  These special iterations took one of two forms. The easiest form was simply to assign more than one person to the finisher role. When we might normally have five developers working features and one finisher, we would conduct an iteration with only two features and four finishers.

  The other form was what we called a blitz. We would designate an iteration length of a week and put the entire team on a finisher role. Blitzes also served an important planning role. The management team would spend blitzes reevaluating the whole feature wishlist and make sure that we were still on the right track. At the end of a blitz, we would have a fresh start with a renewed focus and direction for the product.

Conclusion

Forge has become a powerful weapon in Intridea’s arsenal. At the core, we are successful because we have extremely talented developers who can work miracles. But even the best developers in the world can run into roadblocks if the process they work under doesn’t help them get their jobs done. Forge helps our developers get the job done by making sure that the raw materials of a project, the ideas, are malleable enough to be worked into a finished product.

• Read more about Redesigning Agile: Part II - Introducing Intridea Forge

In the first post in this series, I talked about some of the challenges that we faced trying to implement Scrum on one of our large client projects. After failing to reap the benefits from Scrum, I designed a new process that was better suited to our project and our team. Thus Intridea Forge was born.

On the weekends, I help out at a local non-profit educational dairy farm doing blacksmithing. In the blacksmith's workshop, you start with raw materials and put them in the forge to heat up. The forge heats up the metal to the point where it's soft enough to be worked with tools like hammers and anvils.

This new process serves as a forge for ideas about the product. We use the process to take the ideas from cold, bare metal to a point where they are hot and maleable for the developers to work them into a finished product.

Intridea Forge in 200 words or less

In case you don't care about some of the details, here's the tl;dr version of Forge.

Forge is centered around three key concepts: release iterations, feature ownership, and the finisher.

Release iterations are variable-length, feature-based iterations that correspond to a full production release of the product. Iterations aren't tied to a fixed timeline and they only end when the last of the features is completed. Every iteration is a release iteration.

Feature ownership is the result of starting each iteration with a single developer being assigned to a single feature, even if it's huge and complex. As developers finish the easier features, they roll over and help on the bigger ones or work on code improvement.

The finisher is a role on each iteration filled by a single developer who is not tied to a specific feature. Instead they are dedicated to fixing bugs, code improvement, and reduction of technical debt (or any other task that makes it more of a finished product). The developer assigned to this role changes every iteration.

Intridea Forge in Detail

Since a picture is worth a thousand words, here's a graphical overview of how an iteration progresses.

• Release Iterations

  Iterations in Forge don't have a fixed length. Instead, iterations are tied to the completion of all of the features slotted for that iteration, regardless of how long it takes. This means that iterations will rarely be the same duration as they are pegged to functionality, not a specific schedule. At the end of each iteration you do a full production release, complete with feature announcements and all of the other fanfare.

  This is a bit unconventional in the Agile world and it may sound more like traditional waterfall, but I promise that it's not waterfall and brings quite a few benefits with it.

• Developer Ownership

  At the beginning of the iteration, all but one of the developers on the team are assigned to work exactly one feature each and to work it alone. The developers will then own the feature they start on for the duration of the iteration.

  Since not every feature in an iteration will require the same amount of time to complete, there will be developers who finish their features “early” (it's not really early and shouldn't be regarded that way; some features will be more complex and will take longer and no judgement should be placed on how long it takes to complete). When a developer finishes their feature (and all supporting test and documentation), then one of two things happen.

  If all of the other features are progressing without any issues, then the recently freed up developer simply gets added to the finisher pool (see below). If any of the features are lagging, complicated, tedious, or otherwise in need of help, then the project manager will reassign the developer to help with that feature.

  Let’s say Sally is working on a difficult feature and John completes his feature early and shifts over to help Sally finish. One of the advantages of this process is that by the time John comes in to help Sally she will have already have gotten started on the feature and should be able to easily guide John. At this point Sally will know what needs to be done and how it needs to be done and can make effective use of the additional help.

• Finishers

  Most of the development team is assigned a specific feature to work on for the iteration. The extra developer is assigned to a rotating role called the finisher.

  The primary responsibility of the finisher is to respond to critical popup issues. Such issues might include bugs that were recently introduced and missed during QA, a minor but critical new feature, or other similar items. Any time that the finisher isn’t working on these popups he or she can work on anything that they think needs to be done to make the project better. Examples may include:

  • Fixing old bugs
  • Adding documentation
  • Improving test coverage
  • Refactoring difficult code
  • Modularizing features into components that can be reused and/or released as open source

  Each iteration, the finisher should be a different member of the development team. The rotation of this role is essential and provides a few advantages. First, it keeps developers from getting stove-piped into a single part of the system or a single type of development. It gives everyone the opportunity to dig into other parts of the project. It also ensures that everyone has a chance to clean up or rework some piece of code they may have run across on a previous iteration.

Key Events

During the iteration, the management team has a responsibility to be setting up the next iteration for the team. This is handled by several important meetings.

• Iteration Kickoff
  Who: Development team, project manager

  At the start of the iteration, the project manager meets with the development team and describes the features for the upcoming iteration. During this meeting, the team members are assigned to the various features and the finisher is identified. This is also an opportunity for the development team to get details on the new features and clarify any questions they may have.

  At this meeting, the development team will also give their estimates for how long the features will take. This will be relayed back to the management team and other stakeholders so that they know when to expect the end of the iteration.

• Release announcement
  Who: Management team/stakeholders, project manager

  After all of the work from the previous iteration is complete, the management team meets to schedule the release and plan any downtime. They will review the new features, prepare newsletters, blog posts, etc. at this meeting. After the release is planned, the deployment team will perform the production deploy.

• Mid-Iteration Review
  Who: Management team/stakeholders, project manager

  Based on daily reports from the development team, at this point the project manager informs the management team of the status of the iteration so far. The estimate for completion will be updated and any features that are available on staging can be demonstrated. This meeting can occur multiple times during the iteration.

• Initial Planning Review
  Who: Management team/stakeholders, project manager

  Throughout the iteration, informal planning for the next iteration should be ongoing. The initial planning review is the first occasion where the management team meets to formalize plans for the next iteration. At this meeting, the project manager will collect inputs and rank upcoming features. He or she will also start to flesh out the feature requirements so that estimates can be made and dependencies can be addressed.

• Dependency Review
  Who: Project manager, technical management, lead developers

  After the first pass at defining the next iteration is completed, a dependency review is held. This is the opportunity for the various inter-dependent teams to ensure that everyone will be accommodating any upcoming changes. For example, if the web site team will be making a change or adding a feature that will affect an API used by the desktop application, then this meeting should be where the project managers for the two teams meet to discuss the implications and deployment strategies.

• Final Planning Review
  Who: Management team/stakeholders, project manager

  Leading up to the final planning review, the project manager should have placed initial estimates on all of the features and thoroughly fleshed out the requirements. The final planning review meeting gives the management team one last opportunity to change course on the next iteration before it is locked it. Timeline estimates are reviewed and everything is agreed upon.

Throughout the iteration, informal planning should be happening to lay out upcoming iterations. As the iteration progresses the management team will also be able to see previews of some of the new features as they are made available for testing on staging. This helps ensure that the capability is implemented as intended and satisfies the curiosity of the management team to see what is being built.

Conclusion

This post has given you an overview of Intridea Forge. In the final post in the series, we'll tell you about why it has worked well for our team and we'll share some of the challenges that we faced implementing it. We'll also give some tips and lessons learned about how to implement it on other projects.

• Read more about Redesigning Agile: Part I - Challenges

One of the best comments I've heard about Agile development is that "adopting Agile is itself an agile process". It sums up very nicely the fact that each team and each project will have their own needs when it comes to finding a process that works for everyone, including the stakeholders. It also means that no matter how much you like a specific methodology, it's virtually impossible to implement that process on your team following its strict definition.

As a Scrum Master I've always been partial to Scrum, but after spending six months trying to make it work on one of our large client projects, I reevaluated and started from the ground up to design a whole new Agile process. We'll be introducing our new process in a future blog post, but for now I wanted to talk about some of the challenges we faced implementing Scrum.

There was nothing specific that was wrong with Scrum for our project. We just ran into enough minor hiccups that we started to lose some of the benefits that we should have been able to reap by using it. Before starting, I needed to figure out exactly what wasn't working (and also what was working well) with Scrum so that those issues could be addressed by our new process.

Here are some highlights of our challenges:

• Missing or ambiguous roles

  Scrum works best when a strong Product Owner and a strong Scrum Master maintain a bit of tension between them. The PO needs to demand more and the SM needs to keep things reasonable. The result is an aggressive schedule with a great deal getting done, but development kept to a sustainable pace.

  This can break down if your Scrum Master needs to do a lot of the product planning and prioritization (or isn't involved enough in the planning process). Likewise you can also run into significant problems if you have trouble finding someone who is both empowered and knowledgable enough to be a strong Product Owner.

  Another potential issue is that as painful as it may be, priorities sometimes need to be set by a committee. If the client can’t abide by a single person who selects the features for the upcoming iteration, the process needs to be flexible enough to support that. This also means that if the person who is most responsible for setting priorities is unavailable, then one or more surrogates can easily step in.

• Self-organization doesn't work for every project

  Even when you have competent and self-motivated developers, not all teams work well when attempting to self-organize. For our team, there were a few factors that made it a challenge.

  First, we were regularly adding and removing people. As other projects spun up or down, we would have fairly regular shifts in team composition. While we always had the same number of developers, we frequently had new developers coming on who were new to the system. Since our product is highly complex, they simply could not figure out which of the backlog items from the current iteration were good ones to try to work on.

  Second, with a team spread out across the world, real-time collaboration is hard. In our case we have developers across the US and China and were working with client contacts in every US time zone as well as Spain and South Africa, making it impractical to rely on instant availability. The consequence is that your team needs to be more autonomous and focused on working independently. In my experience, self-organization works best when the whole development team is collocated and able to work with one another face-to-face.

  Last, the client will sometimes need more detailed insight into who is working on what. In traditional scrum, having a backlog story hanging out and unassigned at the beginning of a iteration can make some clients uneasy.

  To get around these issues, we found that the project manager for the development team needs to assign tasking to provide focus and avoid issues with non-real-time collaboration. This needs to be done in such a way that the clients are aware of the assignments, but the developers are still empowered with technical and design decisions without needing approval.

• Rigidly timed iterations can break velocity

  There are a number of situations where having a fixed-time iteration doesn’t work. In the web development world, you will often have short (e.g. 1 week) iterations. Many features easily fit into one iteration, but often a large feature needs to be spread across multiple iterations and simply cannot be broken down into concrete backlog items that can be deemed complete during a single iteration. Also, changes in client schedules, shifting personnel, and unexpected things like conferences, holidays, etc. can make iteration timing problematic.

  Scrum requires a stable team that has a consistent velocity. That velocity is used to determine how much to tackle in the next iteration. The problem is that a team’s velocity is based on the past performance of that team (i.e. how much they got done during the last few iterations). When you have a team that either changes frequently because of personnel or unstable iteration times and shifting customer requirements, then the velocity can become unstable.

  When these factors affect velocity, your most important planning metric becomes meaningless.

• Better transparency into release schedules

  With web development, you often end up in a cycle where major new features come out each iteration and marketing of those features can get lost in the noise. We need a way to say “the next major version of our site is online and it includes features foo, bar, and baz”. The client needs an opportunity to brag about progress and plan ahead for potential downtime.

• Incorporate a mechanism for QA and bug fix cycles

  There needs to be a clean way for the QA team to be fully integrated into the iteration cycle. The acceptance criteria and the planning of regression tests need to be a core part of the process while not holding up the development team. If you have separate QA and development teams then your developers can end up sitting around waiting for QA at the end of a sprint in Scrum.

  At the same time, there needs to be a real dedication to continuous code improvement and bug fixing that isn't dependent on the Product Owner deciding to prioritize those fixes.

While accommodating all of these changes, we still needed to harvest some of the benefits of Scrum and other Agile processes.

Even though we were looking for a process that would allow our clients more insight into who was working on what, we still wanted our developers to be able to focus on the task at hand and not be concerned with shifting priorities or popup crises. The client needs to remain hands-off and the process needs to be set up to make that comfortable for both the client and the development team.

Obviously one of the biggest advantages with Agile development is that you are able to respond to changing requirements (the idea of dropping “ready, aim, fire” in favor of “ready, fire, aim, aim, aim, aim, …”). The new process needs to support that as well or better than the process it is replacing.

These are some of the most important issues that we needed to incorporate into our new process. In the next post in this series, we'll actually describe our new process and how it works. After that, we'll include some tips for implementing it on your project and some of the challenges that might come up.