Certified: The ISACA AAIR Audio Course

In this episode, we’re going to take the idea of testing and turn it into something continuous, because AI risk is not a one-time decision, it is an ongoing relationship between a system, its environment, and the people who rely on its outputs. Beginners often assume that if an AI system performed well during evaluation, it will keep performing well in production, but real-world conditions do not hold still. Data distributions shift, user behavior changes, vendors update features, and new misuse patterns emerge as people discover new shortcuts. Ongoing monitoring is the discipline that keeps the organization from being surprised by those changes, because it watches for drift, performance degradation, incidents, and emerging threats before harm becomes severe. Monitoring is also the mechanism that turns risk tolerance into reality, because tolerance boundaries are only meaningful if you can detect when they are being approached or crossed. Today’s goal is to make monitoring feel understandable and practical, even for beginners, by clarifying what to monitor, why it matters, how monitoring connects to governance, and how to avoid common monitoring failures like dashboards nobody reads. By the end, you should be able to explain ongoing monitoring as a control system, describe the main monitoring domains, and understand how monitoring feeds escalation and continuous improvement.

Before we continue, a quick note: this audio course is a companion to our course companion books. The first book is about the exam and provides detailed information on how to pass it best. The second book is a Kindle-only eBook that contains 1,000 flashcards that can be used on your mobile device or Kindle. Check them both out at Cyber Author dot me, in the Bare Metal Study Guides Series.

A strong way to begin is to treat monitoring as an early warning system rather than as a measurement hobby. Monitoring exists to detect rising risk early enough that the organization can intervene, not simply to collect numbers. That means monitoring must be tied to action, which includes thresholds, decision rights, and response paths. It also means monitoring must be designed around the harms the organization cares about, such as money loss, safety harm, trust damage, and legal exposure. In AI, monitoring is especially important because failure can be subtle, like small increases in error rates or slow changes in who is affected by mistakes. AI can also create convincing outputs that lead humans to rely too heavily, which means monitoring must consider human behavior signals as well as model behavior signals. A beginner misunderstanding is thinking monitoring is only about accuracy, when in reality monitoring must include fairness signals, misuse signals, and control health signals, because those are often the earliest indicators that harm is coming. When monitoring is designed well, it creates calm governance because leaders can see trends and adjust controls gradually. When monitoring is weak, the first signal is often a crisis, and crisis decisions are rarely optimal.

Drift is the first monitoring domain to understand, because drift is one of the most common and least visible ways AI risk increases over time. Drift is the phenomenon where the patterns the AI learned from no longer match the current environment, which can cause performance to degrade or error patterns to change. Drift can occur because customer behavior changes, products change, policies change, language changes, or adversaries adapt, especially in fraud and security contexts. Drift can also occur because the input data pipeline changes, such as when a new field is introduced, a data source is replaced, or missing values increase due to operational issues. Monitoring for drift often begins with input monitoring, meaning you watch whether the input characteristics are changing, such as whether certain categories appear more often, whether certain values become more common, or whether missing data rates rise. You also monitor output distributions, meaning you watch whether the system’s outputs shift, such as fewer cases being flagged as high-risk or more cases being classified into a certain category. These shifts do not automatically mean harm, but they are warning signals that the system’s relationship to reality may be changing. Beginners should see drift monitoring as the practice of watching for change in the conditions that shape model behavior. If you do not watch for drift, you may continue trusting a system that has silently become less reliable.

Performance monitoring is the second domain, and it includes watching how well the system is achieving its intended purpose in the real world, not just in test conditions. Performance monitoring often involves tracking error rates, but it is more useful to track error types and error impact, because some errors matter more than others. For example, missing a rare high-severity case may be far more damaging than misclassifying a low-severity case, even if both count as errors. Performance monitoring also includes watching for changes in confidence and uncertainty signals when available, because sudden changes can indicate instability. In many contexts, direct ground truth may not be immediately available, meaning you may not know right away whether a model output was correct, so performance monitoring can also use proxy signals, like increases in rework, increases in manual corrections, or increases in customer complaints. For high-impact use cases, performance monitoring should also consider whether humans are overriding outputs frequently, because frequent overrides can indicate that the model is less reliable or that the model is being used outside its appropriate context. Beginners sometimes assume performance is purely a model property, but performance in production is a system property that includes data quality, workflow design, and human reliance. Monitoring must therefore look at the broader system, not only the model output, to detect rising risk early.

Incident monitoring is the third domain, and it is about detecting when AI has contributed to harm, near misses, or unexpected outcomes that require investigation and corrective action. Incidents can be obvious, like a customer complaint about an unfair decision, but they can also be subtle, like repeated cases where employees must correct AI-generated summaries. Incident monitoring includes tracking how many AI-related issues are reported, what types of issues they are, and whether certain patterns are emerging. It also includes tracking time to detect and time to resolve, because slow response increases harm and reduces trust. A strong program also monitors near misses, meaning cases where harm was narrowly avoided due to human intervention, because near misses are early warnings that the control system is being strained. For example, if employees frequently catch the AI making a serious mistake before it reaches a customer, that is a near miss pattern that should trigger deeper review. Incident monitoring also includes ensuring that incidents are linked back to the risk register and to control improvements, because otherwise incidents become isolated stories rather than learning opportunities. Beginners should see incident monitoring as part of continuous improvement, not as blame tracking. If people fear punishment for reporting, incidents become hidden, and monitoring fails. A healthy culture encourages reporting so the program can learn and strengthen controls.

Emerging threats monitoring is the fourth domain, and it is especially relevant for AI because AI changes quickly and external threat landscapes evolve. Emerging threats can include new misuse patterns, such as employees finding new ways to bypass policy boundaries, or adversaries learning how to manipulate inputs to cause harmful outputs. Emerging threats can also include changes in vendor behavior, such as new features that alter how data is handled or new integrations that increase exposure. Regulatory changes and public expectations can also be considered emerging risk factors, because they can change what is defensible even if the system’s behavior has not changed. Monitoring emerging threats therefore involves scanning for signals that the environment is shifting, such as industry warnings, internal patterns of policy exceptions, or increased interest in certain AI features. It also includes watching for changes in the organization’s own usage patterns, because internal adoption can spread quickly, creating new risk surfaces. Beginners sometimes think emerging threats monitoring is only for cybersecurity teams, but in AI risk it is broader because threats include social, operational, and compliance dimensions. An emerging threat could be a sudden increase in deepfake fraud attempts, but it could also be a trend of employees relying on AI to draft sensitive communications without appropriate review. Monitoring emerging threats is about staying ahead of new risk pathways rather than reacting after harm spreads.

A strong monitoring program also includes fairness and harm-proxy monitoring, because fairness issues and trust harm often emerge as patterns over time. Fairness monitoring may include tracking whether outcomes differ across groups, whether error patterns are concentrated in certain populations, and whether decision thresholds are affecting groups unevenly. Harm-proxy monitoring includes tracking complaint trends, appeal rates, customer churn signals, and employee feedback related to automated decisions. These signals often appear before a formal incident is declared, making them valuable early warnings. Fairness and harm-proxy monitoring are especially important in high-impact decisions because small disparities can grow and become legally and reputationally significant. Beginners should understand that fairness cannot be verified once and forgotten, because populations change and data changes, and fairness can drift even when overall performance seems stable. Monitoring must therefore include periodic reviews of fairness-related signals, with clear thresholds for escalation when disparities exceed tolerance. This is where monitoring connects to governance because leaders need to decide what level of disparity is acceptable and what response is required. When fairness monitoring is absent, organizations often discover issues only after public scrutiny or legal complaints, which is the worst time to be learning.

Monitoring is only as good as its operational design, and a common failure is building dashboards that nobody reads. To avoid that, monitoring must be tied to cadence and ownership, meaning someone is responsible for reviewing monitoring outputs on a schedule and documenting what was found. Monitoring should also be tiered, meaning routine monitoring happens at an operational level while summarized monitoring and Key Risk Indicator trends are reported to governance groups and leadership. Another critical design element is alert fatigue, where too many signals trigger alerts and people start ignoring them. A good monitoring program uses thresholds that are meaningful and proportionate to impact, and it uses trend analysis to avoid reacting to random noise. It also defines what happens when a threshold is crossed, such as increased sampling, temporary scope restrictions, or escalation to leadership. Monitoring must also be resilient to change, meaning when models are updated or data sources change, monitoring must be adjusted to remain relevant. Beginners sometimes assume monitoring is set up once, but monitoring itself must be maintained, because what matters and what is measurable can change. The operating model and testing plans you learned earlier support this, because they define who maintains monitoring and how monitoring health is verified.

Ongoing monitoring also depends on connecting monitoring to the living risk register, because monitoring signals should update risk posture rather than living in separate dashboards. If drift indicators show increasing instability, the residual risk in the register should reflect that, and treatment plans may need to change. If incident patterns show repeated errors in a specific category, the register should capture that risk driver and the controls being added. If emerging threats signals show increasing misuse of external tools, the register should capture that as a governance and data exposure risk, and the organization may need to strengthen policy enforcement or provide safer alternatives. This connection makes monitoring actionable because it changes management decisions and resource priorities. It also supports executive reporting, because leaders want to know whether risk is trending up or down and what is being done about it. Monitoring without integration becomes noise, while monitoring with integration becomes a feedback loop that improves controls. For beginners, the key is to see that monitoring is not separate from governance; it is the sensory system of governance. Without it, leaders are making decisions blind, and that is not defensible.

To make this concrete, imagine a high-impact AI system that helps prioritize which customer complaints are escalated. Drift monitoring would watch whether complaint types and language patterns are shifting over time, and whether the distribution of urgency scores changes unexpectedly. Performance monitoring would watch whether high-severity complaints are being misrouted, using both direct outcome checks and proxy signals like rework and late escalations. Incident monitoring would track customer complaints about slow handling and internal reports of misclassification, including near misses where humans corrected the AI before harm occurred. Emerging threats monitoring would watch whether adversaries are manipulating complaint text to bypass detection or whether new product changes are creating new complaint categories that the model has not seen. Fairness monitoring would examine whether certain customer segments are consistently deprioritized or misclassified, and harm-proxy monitoring would track whether appeal rates or dissatisfaction signals are rising in those segments. Operational ownership would ensure these signals are reviewed on a defined cadence, and thresholds would trigger actions like increasing human review for certain categories or temporarily restricting AI automation. The risk register would be updated with these monitoring findings, and leadership would receive summarized trend reports. This example shows how ongoing monitoring creates early warning and continuous improvement rather than waiting for a major incident.

To close, building ongoing monitoring is about creating a living control system that watches for change and warns before harm becomes severe. Drift monitoring watches for shifts in inputs and outputs that signal the model’s environment is changing, while performance monitoring watches whether the system is still achieving its intended purpose with acceptable error patterns. Incident monitoring captures harm events and near misses, turning them into learning and control improvement rather than hidden failures. Emerging threats monitoring watches for new misuse patterns, adversarial manipulation, vendor changes, and shifting external expectations that can alter risk posture. Fairness and harm-proxy monitoring ensure that subtle disparities and trust erosion are detected early, especially in high-impact decisions. Monitoring must be operationally real, with ownership, cadence, thresholds, and escalation paths that prevent dashboards from becoming ignored artifacts. It must also integrate with the living risk register and executive reporting so monitoring changes decisions and resource priorities. When monitoring is built this way, the organization becomes proactive, resilient, and defensible, because it can demonstrate that it is watching risk continuously and responding before harm spreads. This sets us up for the next episode, where we define escalation triggers, because the value of monitoring depends on knowing exactly when risk must rise to leadership attention.