Gross Error Checks

"If it doesn't look right it probably isn't." Except what about when it really does 'look right?' 🤔

When learning to fly there are a lot of checks to learn, with some to memorise, at each stage of a flight. Acronyms are used to bring important checks and decision-making during a flight or emergency to the fore. This approach to decision-making within complex systems can help us manage digital risks and particular those of working with AI...

When early-stage trainee pilots are first learning to navigate, typically with less than 20 hours in a cockpit, they are taught about two things: the acronym ANC and gross error checks.

Aviate, Navigate, Communicate

Firstly ANC - keeping the main thing...the main thing.

Aviate, Navigate, Communicate. Aviate, the first building block - always fly the aircraft first, attitude, speed, altitude, engine, look out the window. This will avoid any immediate mishaps.

Then navigate - Where are you? Where are you going? Start by looking widely at the horizon, what are the big features in the distance - hills, coast, cities. Then more closely roads, railways, towns and on to the map. How does what you see outside tally to the chart? This order is important and discipline here can save time...and a certain amount of embarrassment of being 'uncertain of position'.

The mental model here is working from 'territory', the reality of what is outside, to the chart, our 'map'. Working the other round inevitably results in errors - if you are in the wrong place, reality will tell you before the map ever could.

Communicate - only then will pilots talk to air traffic control, or other aircraft. Once pilots have 'A' and 'N' under control they have the situational awareness to provide useful communications.

As cognitive load increases e.g. in an emergency, the pilots are taught the discipline to prioritise and Aviate first, then work back up to Navigate and Communicate. Aviation accident reports are littered with incidents where this discipline has broken down...due to degraded practice or distraction with non-priority actions.

The following summarises this challenge well:

"Effective prioritisation can be a balance between speed and accuracy – there will often be a trade-off between the two.
When speed (or immediacy) is essential then failure to prioritise effectively can lead to an increase in risk by delaying essential tasks beyond a point of usefulness (or recovery).
When accuracy is essential, failure to prioritise effectively can lead to latent errors based on false analyses or assumptions."
- Skybrary

Whilst the time frames might be different for pilots there are useful principles that transfer to knowledge work with automation and AI in technical domains.

Aviate → Technical Task: We need to think technical task first - what is it we are trying to achieve, the output and outcomes? What is the main thing? And how do we keep it the main thing? Especially in environments where distraction is all around.

Navigate → Direction: What is the method by which we have set out to achieve the thing? What are the potential distractions - some useful, most not so much.

Communicate → Contact: Who needs to know what we are doing? Who can we collaborate with to get things done?

So for technical knowledge work how about Task - Direction - Contact - TDC. Try it...let us know what you think.

Gross Error Checks:

Looking a little more closely at navigation and gross error checks...

Flying inherently requires a regular change in direction. Turning this needs action on all three parts of ANC. Aviate - make a balanced and safe turn. Navigate changing from one heading to another, correct, heading. Finally, Communicate - tells others you have made a change.

For an early career pilot this is a point where cognitive work-load goes up, taking risk of something going wrong up with it. One common error is turning on to the wrong heading, this error compounds the longer that heading is held. The sooner the error is noticed the better.

This is where the gross error check comes in.

After a turn on to a new heading the first checks are the gross error checks. The first of which is the question - 'Are we heading in the right direction, roughly?' This could take the form of checking a landmark on the horizon or one closer to the aircraft on the ground e.g. a specific shaped lake or town.

If the feature is in the wrong place....time to think again. One classic error here is making the map fit your ideas of the territory - confirmation bias. Working from ground to map... features in the right place? Yep!...chances of red faces are reduced and on goes the flight.

Gross error checks create pilot discipline to check the basic things, avoiding potentially big and costly errors...

Gross error checks have made their way into other parts of aviation as a quick sense check of potentially critical decisions e.g. aircraft load checks.

Applied to working with AI

How is this relevant to working with automation and AI?

Outputs from AI tools, particularly those with Deep Research capability 'look right'.

Come forward Gross Error Checks...

The International AI safety report highlighted the following risks from AI:

Risks from Malicious Use, Risks from Malfunctions, Systemic Risks. Under risks from Malfunctions it highlights:
A - Reliability: An AI system’s ability to consistently perform its intended function. With three different types:
1. Confabulations or hallucinations - "Inaccurate or misleading information generated by an AI system, for instance false facts or citations."
2. Common-sense reasoning failures.
3. Contextual Knowledge Failures.

B - Bias : Systematic errors in algorithmic systems that favour certain groups or worldviews and often create unfair outcomes for some people.

As an example from recent experience, I was working with an AI model, with Deep Research features to explore academics in a specific field. The product returned a fantastic 2,500 word reports. Great, I thought, lets get into it!

Before I did though, I thought I'd just do a quick sense check...a gross error check.
So I looked to cross-check the output with what I could manually find on the web...

Academic number 1 - right, name, wrong University, a different domain to the one I was looking at resulting in distraction. - Common sense reasoning failure?

Academic number 2 - despite searching for 15 minutes or so...no record of said academic from the University suggested - Hallucination?

Academic number 3 - no record of research in the target domain - Context knowledge Failure?

So What?

These quick findings instantly raise concerns about the validity of the output and shows the risk of taking current AI output at face value.

However, the output still has value in rapidly surfacing sources for closer inspection, identifying real academics for manual research - saving time and broadening the search.

The reasons for these failures will be many. I will hold my hand up to my own early stage prompting skills. This demonstrated to me the value of 'TDC' and Gross Error Checks for working with automation and AI.

As described by some "if it doesn't look right it probably isn't." The challenge with AI is that it does a great job of making it look right so we need to be more vigilant with prompting, sources and challenging output.

Start:

Try Task - Direction - Contact when working with AI and use Gross Error Check. Let us know how you get on.