Very often organizations get too focused on one approach or the other, especially when it comes to having the right quality engineering practices. The paradox of missing the forest for the trees is quite common.

Our biases, organizational priorities, individual competencies, and culturally accepted norms significantly influence how we approach challenges. Over the years, our experience in creating better software has led us to adapt the “Four Quadrants of Knowledge” framework into the realm of software quality and reliability at QualityKiosk. Engineering for exceptional user experiences demands a multidimensional approach that integrates validation, customer insights, and the discovery of outliers while maintaining robust security.

In quality engineering, identifying and addressing potential issues isn’t just a technical task—it’s a strategic imperative. Quality is not confined to functional behavior outlined in requirements; it’s an inherent driver for shaping requirements and delivering reliable, engaging, and innovative digital products. The Quadrants of Quality framework empowers engineering leaders, product managers, and business owners to transform their approach to quality, transcending the focus from features to a holistic view of the customer experience.

The Four Quadrants of Quality

Let’s begin by revisiting the “Four Quadrants of Knowledge,” popularized by Donald Rumsfeld, within the context of app quality:

1. Known-knowns: These are app quality issues that we’re aware of and understand. We design tests and can define specific expected results.
2. Known-unknowns: These are app quality issues we know exist, but don’t fully understand. We can design tests around these but cannot define what results to expect.
3. Unknown-knowns: These are app quality issues that we don’t identify beforehand but instead come to realize based on insights from production data, user feedback etc.
4. Unknown-unknowns: These are app quality issues that we’re not aware of and for which we can’t predict the outcomes.

Each quadrant provides valuable insights into software quality strategies, enabling teams to proactively enhance software resilience and performance.

1. Known-Knowns: The Foundation of Predictability

This quadrant focuses on the well-understood and clearly defined quality parameters. These typically encompass documented requirements, functionality, and expected behaviors. This quality quadrant emphasizes on ensuring that the core functionality and expected behaviors work as intended, through comprehensive testing automation.

Key QE Strategies to Navigate the Quadrant

• Case-Driven Testing: Writing test cases for well-documented functionalities ensures predictable validation. Approaches like behavior-driven development (BDD) and test-driven development (TDD) enable this effectively.
• Automation Testing: Automating repetitive test cases for APIs, regression, sanity, and contract testing improves efficiency and reduces toil. The quadrant’s clear definition of inputs, outputs, and user behavior makes it ideal for automation.
• SAST & DAST: Tools like Static Application Security Testing (SAST) and Dynamic Application Security Testing (DAST) validate known security vulnerabilities based on predefined inputs (e.g., source code for SAST or application runtime for DAST).

While Known-Knowns form the foundation of quality engineering, focusing solely on this area can result in missed opportunities to elevate the product and user experience.

2. Known-Unknowns: Navigating Uncertainty

This quadrant of the quality framework emphasizes the quality parameters where gaps in understanding exist but can be identified and addressed. These often pertain to performance, scalability, or specific environmental behaviors.

Key QE Strategies to Navigate the Quadrant

• Performance Engineering: Testing with known inputs such as load and duration while analyzing how the system responds provides actionable insights to improve scalability and reliability.
• Observability: Setting up logs, metrics, and traces helps monitor system behaviour in real-time. Tools like New Relic, Dynatrace, and AppDynamics provide valuable real customer insights. Using the said approach in lower environment enables teams to conduct A/B tests and iterate safely in controlled environments and release better software in the first place.
• End-User Monitoring (EUM): Understanding how users interact with the software allows product teams to prioritize meaningful enhancements based on actual usage patterns.

Optimizing this quality quadrant ensures that the application performs optimally across diverse scenarios and fosters a continuous feedback loop for improvement.

3. Unknown-Knowns: Uncovering Hidden Insights

The Unknown-Known quadrant refers to quality insights or knowledge that exist within the organization but are not recognized or effectively utilized. These often overlap with the activities of Known-Unknowns but emphasize uncovering hidden assumptions.

Key QE Strategies to Navigate the Quadrant

• Analyzing Historical Data: Revisiting past project data and lessons learned can reveal overlooked patterns or issues.
• Investigating Blind Spots: Observability tools can uncover performance bottlenecks in neglected areas of the codebase.
• Accessibility and Demographics Analysis: Considering factors like user accessibility and demographic impact can uncover areas for improvement, especially in highly interactive software like gaming platforms.

Effectively addressing unknown-knowns refines testing efficiencies, plugs bottlenecks, and prevents repeated mistakes.

4. Unknown-Unknowns: Embracing Exploratory Innovation

The Unknown-Unknowns quadrant encompasses the true quality engineering wild cards, engineering encapsulating the risks and issues that teams are entirely unaware of. Addressing these effectively require innovative and exploratory approaches to identify and resolve the blind spots. Examples of Unknown-Unknowns include discovering unpredictable behaviour in third-party APIs during downtime or identifying rare concurrency issues.

Key QE Strategies to Navigate the Quadrant

• Chaos Engineering: By intentionally introducing failures (e.g., simulating network latency or database crashes), you can identify and fix gaps in system resilience
• Monkey Testing: Through randomized testing, you can uncover unexpected behaviors, often using tools that simulate erratic user inputs.
• Penetration Testing: Conducting both white-box and black-box penetration testing to identify and address unknown security vulnerabilities, such as misconfigurations, unpatched systems, and zero-day exploits.
• Concurrent QE: By embedding quality engineering practices earlier in the development lifecycle through Concurrent Quality Engineering (CQE), you can manage complex interdependencies and interactions, preventing issues from cascading downstream.

By adopting these strategies, business can build the adaptability to tackle unforeseen challenges early in the SDLC.

Bridging the Quadrants: A Holistic Approach

The key to robust software engineering is not perfecting one quadrant but creating a balanced, interconnected strategy:

• • Automate Known-Knowns for a solid foundation
  • Monitor and Refine Known-Unknowns continuously
  • Investigate Unknown-Knowns to unlock hidden potential
  • Experiment with Unknown-Unknowns to build resilience

AI’s Role in Optimizing Quality Across Quadrants

As artificial intelligence continues to evolve, it presents remarkable opportunities across the quality engineering spectrum. However, it’s crucial to understand AI’s limitations and strengths in each quadrant.

In the Known-Knowns quadrant, AI excels at optimizing repetitive testing processes through intelligent automation, enhancing test case generation and providing predictive insights. For Known-Unknowns, AI-powered observability tools can uncover performance bottlenecks and provide real-time system insights.

In Unknown-Knowns, AI can analyze historical data to identify overlooked blind spots. While AI has limitations in the highly exploratory Unknown-Unknowns quadrant, it can complement human efforts through AI-generated test scenarios and exploratory suggestions.

Engineering Reliable, Sustainable and Evolving software with Quality at the Core

Like a meticulously designed high-performance vehicle, software systems require a delicate balance of power, adaptability, and user-centric design. The Quadrants of Quality framework provides decision-makers with a strategic lens to distribute focus and resources effectively.

By embracing this multidimensional approach, organizations can transform quality engineering from a reactive process to a proactive, innovative driver of digital excellence.