The Quality Baseline: Why we Stopped Delegating Product Quality Management to the QA Team
A conversation no one wants to have at project closure:
Customer: You promised accessibility - so why doesn't the audible website work on Internet Explorer 9??
Sales: I can feel the glowing testimonial slipping through my fingers...
Project Manager: But I made sure the functional requirements for accessibility were fulfilled!
Development / Technical Lead: We researched and implemented the latest best practice solution
QA: Why were we not told about this edge case?!
What defines a quality web product? And does everyone invested in the product - users, sales, customer, developers, QA - agree what this is?
It's unlikely, given each party inevitably has a focus on their own priorities. If you ask all 5 stakeholders that question, you should expect to receive at least 5 different answers.
On the customer side it might be that web pages should be "fast loading". But what does “fast” mean? Are we talking about 2 seconds? 5 seconds? Or 10 seconds? What is your “fast”? What is their “fast”?
Or the customer might have no quality expectations and expect it all to come from the expert, which can also lead to misalignment come launch day (see above).
Project closure should be time to exchange high fives, not blows! So it's crucial to expose these differences at the start, when there is still time and budget to add, remove or rework features.
Aligning expectations aside, we as supplier need to make sure we are thinking how to deliver more than what the customer is asking for. The customer's design may be beautiful on desktop but what if it falls to pieces on mobile? What about SEO? UX?
Getting team buy-in on product quality metrics
We follow a stringent quality standard for our products, and our QA team take no prisoners when apprehending bugs thanks to our continuous deployment pipeline.
But "quality" in the broader sense is hard to pin down. We were missing an official document or process to cement a group agreement on what defnes a well-rounded web application or website.
This is where our newly developed Product Quality Baseline document comes into play. And provides extra assurance that everyone involved will get a successful project outcome.
A 16 indicator quality baseline example
Our Product Quality Baseline defines the agreed characteristics (functional/non-functional requirements) of a well-rounded web product, and the quality metrics by which each characteristic can be measured.
The current standard baseline contains 16 different quality indicators in 6 different categories. It also specifies the tools that must be used for measuring, and a column where the developer can request tool training. Because continuous improvement isn’t just for the product!
We use this standard baseline as a starting point when we do planning with our customers. Together we alter it to accommodate changes to satisfy the project expectations, budget and time.
Below is a sneak peak...
|Category||Value Added||Tool||Best Fit||Tolerance|
|Standards||Follow best HTML standards||W3 HTML Validator||0 errors||0|
|Integrity Check||All internal links are working or redirected to the new url. This will protect existing SEO rankings and improve SEO health.||Screaming Frog SEO Spider||0 broken link||0|
|UI/UX||Fast page loading||Google page speed insights|
≥ 90 score for Desktop
≥80 score for Mobile
|Agreed with client|
Rules for the quality baseline document
- It is a living document that is constantly improved according to new best practices, research and tools
It can be adapted according to customer needs or requirements
The contents must be agreed by the customer and project team
Devs can use their own performance tools to measure and test but the final pass result must be achieved using the stipulated tools
It can only be used on completely new products where we have full control and overview of the code
- Although development and QA do the testing, the Project Manager is ultimately accountable for implementing the baseline in the project.
Integrating the quality baseline into our project management process
Now we have a baseline template to define what quality is and the metrics we will use to measure it, we have adopted it into our project and quality management process:
|Sales presents our standard quality baseline to the customer|
Customer gives input on how it should be adjusted for their project. For example, does the website need to be fully audible to make it accessible for visually impaired?
QA and Project Manager determine how the Quality Product Baseline can be best constructed and agree which tools and tolerances should be followed. For the above example, this would include agreeing on the accessibility testing tool and benchmark
Project Manager adjusts the document and communicates it back to the other stakeholders - the customer and Sales - so it can be signed off.
In line with Agile project management processes further adjustments can be made after the kick off stage, subject to agreement from all stakeholders.
Dev moves forward with their work, using the Baseline document as a reference.
We benchmark and test each time there is a sprint release to make the product is on track for final delivery. Sprint releases are an optimal time for feedback and learning!
It agrees and sets expectations between all parties upfront
Before we get swept up in about the possibilities of a digital blank slate, it clarifies the fundamentals of the well-rounded digital product we are about to build. It presents the quality standard we offer, and gives all internal and external stakeholders the opportunity to give input on how they foresee the product performing. This avoids any misalignment due to different definitions of quality or the danger that requirements taken by some as red are not discussed. At the beginning. Not on product delivery.
It improves the development quality of the entire product
In agile, developers develop at the ticket level, meaning that they don’t always have a holistic view of the entire website or app. Our QA Manager Visay recently wrote a blog post on the "definition of done" at Web Essentials, which is a list of quality acceptance criteria that each ticket must pass before we can declare it complete. This is quality assurance at the ticket level. But what about when all product components have been assembled? Our quality baseline defines quality acceptance at the product level. And as part of the iteration cycle, our development team, QA developers and Technical Lead will continuously review the product against the baseline document and review the document itself to make sure that expectations are satisfied.
It functions as a communication tool
The document communicates what we will deliver as part of the product and which metrics are used to determine success criteria between our Sales team and the customer. It is also a communication tool between the dev and QA teams, outlining what the devs can and need to deliver in terms of overall product quality.
What's next for this quality management initiative?
So far we have used it in a couple of projects and different teams within Web Essentials are working to improve it further. Coming up:
Learning by doing
The baseline is in the early stages and the more we use it in projects, the more hands-on experience we can gather to improve how it is communicated and implemented.
Improved user experience for everyone
The baseline is a shared document with multiple stakeholders and needs to be crystal clear for all. The Sales team are pros at communicating concepts to customers in a way that is easy for them to understand. This makes them perfect candidates to improve the language and presentation of the baseline to make it less technical and more user-friendly.
Manual testing is still a big part of the quality baseline. The QA team is currently working on using automation using their favorite tool - Jenkins - for all tests (because if a score is possible then we can automate it!)
Some technologies we use offer features such as accessibility already, but these often need further optimization before we can add them to the final product. In future we want to improve and modularize these features so we can offer a flexible pre-built package of optimized baseline items. By reusing modular, tried and tested elements, we can significantly cut down individual manufacturing time and cost so we can allocate the customer's budget more efficiently. We aim to use these packages to improve the quality of our t3kit websites even further.
It is very important to ensure that every developer knows what the baseline is and how to fulfill it in product development. We are continually providing training, practices and documentation to support our developers in the process, as every developer should be able to work on the tasks.
To sum up...
Quality issues can often be fixed and in fact, the usual case is that they are. But to deliver quality in budget and in time is not so easy and that's where this new approach makes a real difference.
The Quality Baseline in its current form may be a fairly fresh concept but already plays a very important role in our software development projects. It gives every stakeholder a voice on quality and invokes collaboration from the beginning to the end of the project.
Looking forward to sharing more insights along the way!
Want to cut out project quality worries so you can focus on creating an innovative product? Talk to us today!