Agile Scrum and BI

I’ve worked with Agile Scrum a few times, and a few times very unsuccessfully.  Here are the things I believe need to happen for Scrum to be successfully implemented with a BI Team.

#1.  Educate all participants of what Scrum is and is not.  It is important that the management, users, and developers are all on the same page as to how Scrum works and what the fundamental principles of Scrum are.

#2.  Structure the Sprints in a way that makes sense for BI.

#3.  Have the correct Product Owner.

What Scrum is not….

#1.  Scrum is not just stand up meetings and/or a board of tasks.  

#2.  It is also, not doing the same thing, just faster.

I feel there are a few key concepts, that get the point of Scrum across.  Here is a Scrum primer, or review depending who is reading.

Scrum is an iterative and incremental framework for managing product development.  A key concept is the MVP or Minimal Viable Product. The aim of Scrum is to start with the MVP and continuously add features in short time constrained chunks (Sprints).  At the end of each Sprint the product is reviewed with the users, and new features are prioritized. The idea being to limit over engineering and to make sure the solution fits the need, as well as, adjust for shifting market conditions.  Which makes it very popular in IT.

Most management techniques date back to the industrial revolution.  This is where processes were simple enough that one person could understand everything.  Generally this was the manager. This manager was the person that was groomed and trained, and they could help the people beneath them meet their individual objectives.   With the complicated projects of today, no one person can possess all the skills necessary to complete the project. So, you need a team of individuals with all skills necessary to complete the task at hand.  The Scrum master is a servant leader who helps resolve logistical issues, and remove roadblocks facing the development team.

Another key role in scrum besides the Scrum Master and Development Team is the Product Owner.  In my opinion the hardest and most important role is Product Owner. This is the person that communicates with the users and stakeholders to prioritize the product backlog.  Or put another way, prioritizes the requirements.

Traditional waterfall project management may be described as having fixed scope and resources where time is most often adjusted.  Scrum has fixed resources and time where most often scope is adjusted.

So what are the Biggest Mistakes I’ve Seen with Scrum in a BI project?

  1.  A backlog is created and the expectation is set to have it completed in a certain total amount of time.

This is where management has laid out a bunch of features (user stories) and expects them to be completed in say 4 months.  There is one of two things happening here. Either, management has not switched over from traditional waterfall thinking to Scrum, or this project should had used a waterfall project management method to start with.

  1.  Manager decides to control the scrum Development team.

In a lot of cases the current manager or PM may feel threatened by the scrum process.  It is important at the outset to make the current managers role clear. If necessary, give assignments such as higher level reporting/communication to give that managerial role purpose.

  1.  Resources not dedicated fully to project or Development team not having all skills necessary to complete project.

The team should be committed as much as possible to the task at hand.  The development periods are short and intense, and full time participation is needed to meet the window.  The team should have all the skills and resources necessary to complete the project. If this is not the case, rectifying this would be the job of the Scrum Master.

  1.  Backlog adjusted or features added mid-sprint.

The idea of the sprint is that it is time blocked short enough that at the end new features can be added or current features can be modified.  The time block is short enough as is, you should allow the development team to focus.

  1.  Sprint structure – trying to do too much in one Sprint.

This is the biggest mistake with the most catastrophic results.  Each sprint is supposed to end with a tested shippable production ready product.  In BI we generally need complete steps that all depend on each other. For instance a typical BI process may look like this.  

Source data-> load or persist this data -> validate data -> analyze the data -> create some models or structure ->transform the data -> validate transformed data -> create metadata layer for BI tool -> create the presentation to the users ->validate metadata and final result.

It is a mistake to try to do all of this in 3 weeks.  Even on a small scale.  This just leads to sloppy incomplete work, burnout, and developers up to 4am the night before the Sprint review meeting.

I have two sprint structures that I favor.  

#1 – Use a waterfall plan to create and populate a base model.  You basically do all the steps above from source data to creating the metadata layer.  Then you start your Sprints when it comes to what the users actually see (ie, dashboard, API, self-service development).  Based on user feedback you will inevitably have to update the model, ETL, etc.; but hopefully now it is in bite sized chunks.

#2 – Re-Establish the MVP (Minimal Viable Product).  In most BI projects, the output, or MVP is simply a set of data.  In a lot of cases you could produce a spreadsheet and the end-user community could survive, if they had to.  Finance users would probably prefer it 🙂 The following is an example of how a series of Sprints might look, using this approach.

Sprint # Sprint 1 Sprint 2 Sprint 3 Sprint 4 Sprint 5 Sprint 6 Sprint 7
Product Delivered
Dims Added
More Dims added
Metadata created
Dashboard 1
Dashboard 2

These would not be planned ahead of time, this is just an example of the flow.  You start with a data set.  Maybe, just a table loaded with transactional data and some logic to create certain metrics.  Your users can look at it and verify it contains most of what they need.  In Sprint 2, maybe you need to adjust this data set.  

It’s important in these early sprints the entire Development team is involved.  The analyst should help make sure the data is meaningful and accurate. The architect should be planning the model.  Dashboard developers working on Mock-ups.  BI tool experts and architects looking at the grain and how it will roll up.  ETL folks will obviously be working on development to stage and clean the data. Even though we are just delivering a ‘data set’ there should still be plenty for everyone to do.

Sprint three you start dimensionalizing the dataset.  Sprint 4 you add more dimensions. Sprint 5 create the metadata, and show the users how certain things will roll-up, or what the self-service layer may look like.  Then finally Sprint 6, you start iterating on dashboards.

One thought on “Agile Scrum and BI

  1. This is a great post and it digs up so many emotions about sprints and waterfalls past. I am a fan of the MVP concept with the caveat that it has to have a solid foundation just in case you never get the luxury to go back and rebuild from scratch in these days of short management attention spans 😉 What I have learned is that the methodology doesn’t matter as much as the team. If you have the right team of people that work well together, they will get the job done quickly and efficiently. I have seen things built in 2 weeks by one team that would take months for another team to create. This is a management problem. Good managers/product owner need the courage and authority to pull the right team together and allow time to focus.; even if it is just in 2 week increments.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s