In the #BIWisdom discussion on Twitter hosted by Howard Dresner every Friday is a very informative channel, and last week the discussion turned to Agile BI.

Agile is a term that is much abused, and is often used as an excuse for lack of planning and/or lack of documentation, when in fact it should be neither, and in my view involves much more planning than a waterfall method, as you replan as items change:

I thought I’d take the time in this first newsletter of ours to talk about how we have implemented agile techniques within our BI practice.

Read more…

In the #BIWisdom discussion on Twitter hosted by Howard Dresner every Friday is a very informative channel, and last week the discussion turned to Agile BI.

Agile is a term that is much abused, and is often used as an excuse for lack of planning and/or lack of documentation, when in fact it should be neither, and in my view involves much more planning than a waterfall method, as you replan as items change:

I thought I’d take the time in this first newsletter of ours to talk about how we have implemented agile techniques within our BI practice.

But first, why agile? A traditional data warehouse project from the early days of BI was an all-encompassing project that tried to solve every possible data need that the business analyst may ask, and often turned out to take years. In the interim, the business needs or indeed the business itself may have changed. In one project I worked on, a technically completely successful integration of data to and from a partner company, the day before we went live our business announced the cancellation of that partnership. 6 months of work disappeared in a minute….

The intent of agile methodologies is to enable quick reactions to business changing, while still allowing control and planning to function – as an unplanned project is bound to fail. In the case I walk you through below, minor changes can be accommodated in a 2 week timeframe, and new prioriities will be included in at most 6 weeks (and often much much quicker) without disrupting the development flow and wasting effort. This approach also tests the product set built more often, and validates (and more importantly invalidates) assumptions much quicker

The other intents are to enhance a communication flow, both within the teams and with the external users – both to get better input from the business team, but also to communicate both successes and problems better. Your biggest challenge as you adopt agile is to get your team to openly admit failures and delays to the client, as they will struggle to lose face. Getting the failures and delays communicated often and early will help them break this block, and

We use a hosted tool called OnTimeNow to do our planning and backlog management: you can use any tool, and many agile teams use a whiteboard or cards to do the planning, but I find the flexibility of a software tool immeasurably valuable.

Let us first define some of the terms:

• A project is a project in the traditional sense. In OnTimeNow, we use Product as our mapping to Project.
• A backlog is the list of items outstanding to deliver on a project.
• A decision log is a list of all decisions made on the project.
• A defect log keeps track of all outstanding issues on the project.
• A release is a portion of work that will be put into production – although much of the BI consulting work is not product based, a release is still equivalent to a point release in software. A release consists of 1 or more sprints, though typically 2 to 4.
• A sprint is a defined period of time (in our case 2 weeks) in which work is done. A key difference between sprint and waterfall is that the sprint time is the defining facto, in waterfall it is the feature list
• The Sprint plan is the initial plan outlining which features will be delivered during the sprint and including the dates. Dependencies are not explicitly tracked visually in OnTimeNow, so the sprint planner needs to consciously build these into his or her plan. This is done in a joint sprint planning session with the client and including their business team.
• The weekly plan is a twist that we have added, more for internal and external communication reasons and for ensuring that the team are constantly replanning – this is done during a weekly session internally at a weekly planning session, starting first thing Monday morning, and is aimed at cross-sprint planning as well.
• In traditional Agile, the daily standup is a daily 15 minute meeting conducted standing – standing because it keeps people to the 15 minutes! We do this on each of our projects, but for many of them, it is done over Skype as we have remote teams. This is one of the most essential parts of Agile, as it helps keep communication lines open across the team.
• A user story is what is used in place of a typical specification in Agile – each user story describes, from the point of view of a user, a single requirement of the software. In terms of BI, the story writing aspect often only applies to the front-end or user facing parts of the project, as loading correct data into a warehouse isn’t a typical user story.

Now that we have the terms down, let’s talk about the agile process in general. One of the biggest misconceptions about agile is that it does not have documentation – instead, documentation is often a release of its own early in the project.

A typical project for us would start with a single sprint release to do upfront analysis. This analysis would result in the following outputs:

• A technical architecture document detailing the technologies to be used
• A hardware architecture document outlining the hardware required
• A project architecture documenting what team members are required for the project, and also listing what the releases are for this project. Barring dependencies that can’t be adjusted, this list is mutable and the order can be adjusted at any time, with the exception of the currently running release.
• A list of requirements at a very high level captured into OnTimeNow as a backlog, in an “In Estimation” workflow step

The backlog would then be reviewed by the delivery team and the client, and broad estimates applied. These estimates along with the team sizing would then allow for a release to be broken up into sprints. The items approved for delivery will be moved into the “Approved” workflow status that is the full level of planning done across the whole release plan, as more information is required to give more detail.

Each release starts with a sprint planning session that also doubles as a release review. Firstly, the breakup into the sprints is confirmed, and then the first sprint is planned. Into the backlog is captured the detailed level user stories and items that are required to deliver successfully, down to a 2 to 4 hour level. Both 1 hour and 8 hour estimates are also allowed, but no backlog item can go over a day. Also, values between these are not allowed, to build in some slack time. Each item is then given a due date, and moved to the “Current Sprint status”.

Each task is moved to “In Progress” as it is started, and into Completed as it is done. Any item held up for any reason is moved into a “Roadblock” status.

Then, daily, the lists of items outstanding (due in the past and not completed, due today, or in roadblock) is discussed in the daily standup. Each person on the team needs to give a quick overview of “What did I complete yesterday? What am I planning for today? What is holding me back?” with an email update (pulled straight) from OnTimeNow sent out to all stakeholders

Any new requests, or defects on work delivered, is captured into the backlog during the week and is not allowed to be started (barring crisis items, defined as items that stop work going forward) – for defects, they are allocated a priority in the weekly planning session, and for the new items, only In a new sprint. This back pressure on changes helps the delivery team keep a focus on the current task, and not be distracted.

Weekly planning sessions are then held across all of the teams: a similar approach to the dailies, but a much longer session as it is across all the teams, and it is for the week’s tasks. We have a hard limit at 2 hours, as past that point we don’t see it as productive. These weekly plans are then also sent out to clients so that they are intimately involved in all aspects of what work is being done, and indeed have a big say in what the priorities are. Part of this update is reviewing the burndown chart: this gives an indication of whether the project will complete on time, and needs to be the focus of the weekly planning meeting, along with strategies to make the deadline if it is slipping.

Every sprint ends with a retrospective: what went well, what could have gone better, what we need to allocate as improvements in the next sprint. This is a key part of the process, and I’s important that it not become a blame game if something went wrong, and instead is aimed at continual improvement. A big part of the retrospective is the allocation of incomplete tasks to future sprints, and a review of whether the release will need another sprint to achieve all the deliverables.

Every organisation works differently: Agile methodologies can be very effective for BI projects, but they take a lot of discipline to implement. Be careful not to fall in the trap of thinking Agile development means you’re not creating documentation or planning – Agile means planning more often!