Skip to content
Hazen + AI | Troy Redding

Proposed approach

How leadership should know whether AI is working

Six dimensions with operational definitions, proposed accountable roles, measurement cadence and decision gates. These are proposed examples, not achieved results or live KPIs. Numeric improvement targets, other than the requirement for zero unauthorized disclosures in permission tests, would be illustrative until agreed during discovery.

  1. 01

    Are we saving useful time?

    Compare paired median task completion time with the baseline, including review and rework, for the same task type and comparable complexity.

    Proposed measures
    Median search time including source verification; Median proposal preparation time including editing; Median review time including corrections
    Accountable roles
    Practice sponsor with the Director of AI Strategy & Adoption
    Cadence
    Baseline before pilot, monthly during pilot, quarterly after release
    Expansion and stop gate
    Expand when time improves without weakening quality. Stop if repeated measurement shows no useful benefit after specific workflow fixes.

  2. 02

    Is project work improving?

    Measure review cycles, rework and first pass acceptance for comparable work, with qualified reviewers judging whether quality changed.

    Proposed measures
    Review cycles per deliverable; Rework hours by task type; Reviewer confirmed first pass acceptance
    Accountable roles
    Practice leader with Quality and licensed engineering reviewers
    Cadence
    At project milestones and in the quarterly portfolio review
    Expansion and stop gate
    Expand only when engineering reviewers confirm maintained or improved quality. Stop use in the affected workflow if quality declines or review is bypassed.

  3. 03

    Are people using the tools again?

    Measure repeat use for eligible tasks among the defined pilot cohort rather than counting logins, prompts or training completion alone.

    Proposed measures
    Repeat task use among the pilot cohort; Eligible workflows using the approved method; Reasons users return to the prior method
    Accountable roles
    AI Adoption & Governance Manager as IT partner, with the practice sponsor
    Cadence
    Monthly during pilots and quarterly for supported services
    Expansion and stop gate
    Expand when repeat use is sustained and support needs are understood. Stop or redesign when repeat use stays low after workflow fit and training issues are addressed.

  4. 04

    Are the systems reliable enough?

    Use a fixed subject matter reviewed evaluation set to measure source supported correct answers, correction rates and traceability for the intended task.

    Proposed measures
    Source supported correct answer rate; Unsupported answer rate; Correction rate and source opening rate
    Accountable roles
    AI and machine learning engineering with independent subject matter experts
    Cadence
    Before release, after every material change and monthly during pilots
    Expansion and stop gate
    Expand only after meeting the evaluation target agreed during discovery. Stop release when results fall below that target or unsupported answers cannot be controlled.

  5. 05

    Are we controlling risk?

    Test permissions, required human review, data handling, incidents and vendor findings against the approved use case and access model.

    Proposed measures
    Permission tests with zero unauthorized disclosures; Required review completion; Open security, confidentiality and vendor findings
    Accountable roles
    Cybersecurity with IT, Legal, Risk and the data owner
    Cadence
    Before release, after access or model changes, and monthly for controlled systems
    Expansion and stop gate
    Do not expand unless permission tests show zero unauthorized disclosures and required review is evidenced. Stop affected access immediately after an uncontrolled disclosure.

  6. 06

    Are we making good investment choices?

    Compare total license, build, integration, support and human review cost with validated time, quality or risk benefits for the same workflow.

    Proposed measures
    Full run cost per supported workflow; Validated benefit compared with total cost; Pilot conversion and time to supported production
    Accountable roles
    Director of AI Strategy & Adoption with Finance and the practice sponsor
    Cadence
    Monthly during pilots and at quarterly investment reviews
    Expansion and stop gate
    Expand when validated benefit justifies full run cost and support capacity. Stop when cost exceeds validated benefit without an explicit strategic reason agreed by leadership.

Executive review

I would recommend a quarterly review where each effort gets one of four short decisions. The evidence should include benefits, review and rework, full cost, control results and remaining dependencies.

Continue
Results match the measures set at the start.
Expand
Clear value and controls hold up. Add users or workflows.
Fix
Value is likely but something specific needs to change.
Stop
The measures are not met or the risk is too high.