I was recently asked to talk about the importance of creating and following a Test Strategy for a Business Intelligence project. Naturally, I thought this would be a good, quick topic to cover in this blog.
Like most developers, I have participated in some basic testing tasks in the past. However, it wasn’t until more recently in my career that testing has become more of a focus, with being the primary testing resource for 3 out of 5 of my last projects. Below are some of the questions I found myself asking when I was first tasked with being a BI QA resource. My answers are based on my experience in those 3 projects, but primarily knowledge gathered for my most recent project where my role was the BI QA Lead.
What is BI/Data Warehouse Testing?
It is the process of tracing the accuracy and validity of the data throughout a business intelligence system. This is the key to ensuring the quality of the data and reporting output, which will help with user adoption. BI testing has a slightly different goal from traditional software testing. Software testing focuses on ensuring correct functionality of the application code, while BI testing can include code functionality testing as well, but primarily focuses on checking the business value of the information derived by the delivered data and reports.
What Are Some of the Challenges?
Time
Creating tests can take time and can sometimes become a secondary thought when planning development cycles, with of course, focuses primarily on the development itself. Time has to be allotted to allow the testers gather enough information and understanding about the data being processed to properly develop tests. This can be especially true for smaller BI teams or projects.
Resources
With the fast pace nature of BI projects, developers tend to have a pretty full work load and having to wear both the developer and tester hats in the project can be a challenge.
Multiple Areas of the BI System to Consider
Testing can become very in-depth. You could test every data touch-point in a BI system such as the ETL databases, OLAP cubes or tabular models, as well as Reports. It might also be sufficient to test only the data warehouse tables and end reports. Regardless of the depth you decide, chances are you are going to touching multiple systems with potentially have to account for different data outputs.
How Does a Test Strategy Help?
Creating a testing strategy helps by addressing these challenges and accounting for them during the beginning planning phases of a project. This can be achieved by providing structure and process around handling these challenges. For example, the Test Strategy might state that due to resource availability, BI Developers will also be responsible for BI Testing. However, if this is the case, it is a good practice to not have the developer also be the tester for a unit of work to ensure unbiased, proper testing is completed (it can be hard to find/test for errors when you are close to the code).
Are There Other Benefits to Having a Test Strategy?
A Testing Strategy is also important because it ensures development matches the established project requirements and should outline acceptance criteria for the BI solution. Creating a solid testing strategy also establishes consistency in testing across the team, which can help to a decrease post-deployment defects. This also helps to organize testing by aiding in the creation of test design patterns for each area of the BI system. This has helped me on engagements by providing direction to the testers to ensure both consistency and efficiency in the test development.
So, What Typically is Included in a Test Strategy?
Generally you would want to include:
- The test objectives and scope – making sure to include what is considered out of scope as well
- Any assumptions about the test environment, business requirements, etc.
- The roles and responsibilities – not just the testers but the developer or business resources that are available to the testers
- The Testing methodology & plan – including the testing strategies that will be used, as well as the types of tests in those strategies that will be developed.
for example, Data Quality of the Data Warehouse tables might be a strategy listed, and a test might be source to target testing of all dimensions and facts - Defect Tracking
- Test Acceptance Criteria and deliverables
Data Dev-On-The-Go! 