The Quality Baseline: Why we Stopped Delegating Product Quality Management to the QA Team

We follow a stringent quality standard for our products, and our QA team take no prisoners when apprehending bugs thanks to our continuous deployment pipeline.

But "quality" in the broader sense is hard to pin down. We were missing an official document or process to cement a group agreement on what defnes a well-rounded web application or website.

This is where our newly developed Product Quality Baseline document comes into play. And provides extra assurance that everyone involved will get a successful project outcome.

Our Product Quality Baseline defines the agreed characteristics (functional/non-functional requirements) of a well-rounded web product, and the quality metrics by which each characteristic can be measured.

The current standard baseline contains 16 different quality indicators in 6 different categories. It also specifies the tools that must be used for measuring, and a column where the developer can request tool training. Because continuous improvement isn’t just for the product!

We use this standard baseline as a starting point when we do planning with our customers. Together we alter it to accommodate changes to satisfy the project expectations, budget and time.

Below is a sneak peak...

• It is a living document that is constantly improved according to new best practices, research and tools
   

• It can be adapted according to customer needs or requirements

• The contents must be agreed by the customer and project team

• Devs can use their own performance tools to measure and test but the final pass result must be achieved using the stipulated tools

• It can only be used on completely new products where we have full control and overview of the code

• Although development and QA do the testing, the Project Manager is ultimately accountable for implementing the baseline in the project.

Now we have a baseline template to define what quality is and the metrics we will use to measure it, we have adopted it into our project and quality management process:

1. Sales presents our standard quality baseline to the customer
2. Customer gives input on how it should be adjusted for their project. For example, does the website need to be fully audible to make it accessible for visually impaired?
3. QA and Project Manager determine how the Quality Product Baseline can be best constructed and agree which tools and tolerances should be followed. For the above example, this would include agreeing on the accessibility testing tool and benchmark
4. Project Manager adjusts the document and communicates it back to the other stakeholders - the customer and Sales - so it can be signed off.In line with Agile project management processes further adjustments can be made after the kick off stage, subject to agreement from all stakeholders.
5. Dev moves forward with their work, using the Baseline document as a reference.
6. We benchmark and test each time there is a sprint release to make the product is on track for final delivery. Sprint releases are an optimal time for feedback and learning!

It agrees and sets expectations between all parties upfront

Before we get swept up in about the possibilities of a digital blank slate, it clarifies the fundamentals of the well-rounded digital product we are about to build. It presents the quality standard we offer, and gives all internal and external stakeholders the opportunity to give input on how they foresee the product performing. This avoids any misalignment due to different definitions of quality or the danger that requirements taken by some as red are not discussed. At the beginning. Not on product delivery.
 

It improves the development quality of the entire product

In agile, developers develop at the ticket level, meaning that they don’t always have a holistic view of the entire website or app. Our QA Manager Visay recently wrote a blog post on the "definition of done" at Web Essentials, which is a list of quality acceptance criteria that each ticket must pass before we can declare it complete. This is quality assurance at the ticket level. But what about when all product components have been assembled? Our quality baseline defines quality acceptance at the product level. And as part of the iteration cycle, our development team, QA developers and Technical Lead will continuously review the product against the baseline document and review the document itself to make sure that expectations are satisfied.
 

It functions as a communication tool

The document communicates what we will deliver as part of the product and which metrics are used to determine success criteria between our Sales team and the customer. It is also a communication tool between the dev and QA teams, outlining what the devs can and need to deliver in terms of overall product quality.

So far we have used it in a couple of projects and different teams within Web Essentials are working to improve it further. Coming up:

Learning by doing

The baseline is in the early stages and the more we use it in projects, the more hands-on experience we can gather to improve how it is communicated and implemented.

Improved user experience for everyone

The baseline is a shared document with multiple stakeholders and needs to be crystal clear for all. The Sales team are pros at communicating concepts to customers in a way that is easy for them to understand. This makes them perfect candidates to improve the language and presentation of the baseline to make it less technical and more user-friendly.

Automation

Manual testing is still a big part of the quality baseline. The QA team is currently working on using automation using their favorite tool - Jenkins - for all tests (because if a score is possible then we can automate it!)

Modularization

Some technologies we use offer features such as accessibility already, but these often need further optimization before we can add them to the final product. In future we want to improve and modularize these features so we can offer a flexible pre-built package of optimized baseline items. By reusing modular, tried and tested elements, we can significantly cut down individual manufacturing time and cost so we can allocate the customer's budget more efficiently. We aim to use these packages to improve the quality of our t3kit websites even further.

Education

It is very important to ensure that every developer knows what the baseline is and how to fulfill it in product development. We are continually providing training, practices and documentation to support our developers in the process, as every developer should be able to work on the tasks.