The minimum decisions, evidence, evaluations, and fallback paths required before AI influences operating work.
Governance is an operating design problem
Governed AI is sometimes reduced to a policy document or vendor questionnaire. Those artifacts matter, but neither tells an operating team what a particular system may do with a particular record for a particular person. The control map must connect authority, data, model behavior, human review, monitoring, and response.
NIST’s AI Risk Management Framework makes governance a cross-cutting function and pairs it with mapping context, measuring performance and risk, and managing the resulting priorities. The sequence is useful because it starts with intended use and consequence, not model novelty.
1. Authority and consequence
Start by naming the system owner, affected people, intended users, permitted actions, prohibited actions, and the person accountable for accepting residual risk. Then classify what happens if the system is wrong, incomplete, unavailable, or misused.
A system that drafts an internal project summary can tolerate a different error profile from one that recommends staffing, changes a customer commitment, or touches financial records. Controls should follow consequence rather than applying the same ceremony to every experiment.
- Who may initiate, review, approve, and override the output?
- Which decisions must remain human, and what evidence must the reviewer see?
- Which users, records, jurisdictions, or actions are outside scope?
- Who can suspend the system when a threshold is breached?
2. Evidence and data quality
Document source systems, record-level permissions, retention, transformations, lineage, refresh timing, and known gaps. Retrieval does not make weak evidence trustworthy; it only makes weak evidence easier to present.
ISO/IEC 5259-5 frames data quality governance through policies, roles, responsibilities, and processes. For an operating system, that means every consequential input needs an owner and an exception path—not merely a quality score.
- Minimum necessary access and purpose limitation
- Authoritative sources and precedence when records conflict
- Required freshness, completeness, and reconciliation thresholds
- Traceability from output to the records and transformations used
- A review queue for missing, ambiguous, or contradictory evidence
3. Evaluation before release
Evaluate the named task on representative known cases, including expected behavior, ambiguous inputs, missing evidence, conflicting sources, adversarial instructions, and requests outside scope. Define the threshold and failure action before seeing the score.
NIST’s Generative AI Profile identifies risks particular to generative systems and emphasizes measurement in context. A useful evaluation therefore tests the complete workflow—including retrieval, prompts, permissions, citations, interface, and human review—not only the base model.
- Task success and material-error rate
- Citation or evidence-grounding accuracy
- Abstention when evidence is absent or outside scope
- Permission isolation and sensitive-data handling
- Reviewer agreement, override rate, and time required
- Deterministic fallback when the AI component is unavailable
4. Monitoring and change control
A passing evaluation describes one version under one set of conditions. Models, prompts, retrieval indexes, source schemas, permissions, and user behavior change. Record those dependencies and identify which changes require retesting or approval.
Monitor a small set of operational signals: volume, failures, abstentions, evidence gaps, overrides, incidents, latency, cost, and drift in input patterns. Preserve enough logs to investigate while respecting retention and access limits.
5. Capability, training, and ownership
OECD research on small and medium-sized enterprises identifies skills, knowledge, data, and regulatory uncertainty among the constraints on AI adoption. A control map should therefore include the people operating the system: what they must understand, when they should distrust an output, and how they report a failure.
The end state is not autonomous intelligence. It is a controlled operating capability whose owners can explain its purpose, evidence, limits, tests, and fallback—and can turn it off without losing the underlying decision process.