Certified: The ISACA AAIR Audio Course

In this episode, we’re going to take the idea of controls out of the abstract and turn it into a practical question you can apply to any AI use case: what exactly should we verify, and how often should we verify it. Beginners sometimes hear control and think of a single safeguard, like a permission setting or a policy rule, but in risk management, controls are a system of checks and boundaries that keep outcomes within tolerance. AI makes control design especially important because AI systems can look reliable until the environment changes, and because the human tendency to trust confident outputs can amplify harm. A testing plan is what keeps controls from becoming assumptions, because it defines how the organization will confirm that the controls exist, operate as intended, and remain effective over time. Without a testing plan, controls can decay quietly, documentation can become stale, and high-impact systems can drift into risky behavior while everyone assumes oversight still exists. The goal today is to show how to define AI controls and create testing plans that are proportional to impact, realistic for operations, and defensible under scrutiny. By the end, you should be able to describe the main control categories for AI, understand the difference between verifying design and verifying operation, and explain how testing frequency should scale with impact and with risk signals.

Before we continue, a quick note: this audio course is a companion to our course companion books. The first book is about the exam and provides detailed information on how to pass it best. The second book is a Kindle-only eBook that contains 1,000 flashcards that can be used on your mobile device or Kindle. Check them both out at Cyber Author dot me, in the Bare Metal Study Guides Series.

A good starting point is to define what we mean by control in the AI risk context, because controls include both technical and non-technical safeguards. A control is a measure that reduces risk by preventing harm, detecting harmful conditions early, or enabling corrective action before harm spreads. Controls can be preventative, like restricting AI use to approved tools and limiting what data can be input. Controls can be detective, like monitoring Key Risk Indicators (K R I s) for drift, bias signals, or policy violations. Controls can be corrective, like having clear escalation triggers and authority to pause an AI feature when tolerance is exceeded. Controls can also be governance-based, like requiring approvals, documenting intended use, and defining decision rights and accountability. A strong AI risk program uses all of these because AI risk is not only about technical failure; it is about how systems are used, how humans rely on outputs, and how the organization responds when issues appear. The reason testing plans matter is that controls can exist in name but not in practice, especially when teams are busy and systems change frequently. Testing is how you verify that the control system is real, not just described.

Now let’s talk about what to verify, because verification should be tied to the risk pathways that matter most. One major area to verify is use case boundaries, meaning the AI is being used only for its intended purpose and within approved scope. This includes verifying that high-impact decisions still have required human review, that the AI output remains advisory where mandated, and that automation has not quietly expanded. Another major area is data controls, meaning only approved data types are used as inputs, sensitive data is handled according to policy, and data flows to vendors are understood and controlled. Another area is model and output reliability, meaning the AI continues to perform within acceptable limits and does not show drift or harmful changes in output patterns. Another area is fairness and impact concerns, meaning outcomes do not develop unacceptable disparities and monitoring detects early warning signs. Another area is security and access, meaning only authorized users can access AI features and that system integrity is protected. Another area is documentation and governance evidence, meaning approvals, risk assessments, exceptions, and monitoring records are current and traceable. Another area is incident readiness, meaning escalation paths work, response roles are clear, and issues are handled consistently. Beginners should see that verification is not one thing; it is a set of checks aligned to where harm can enter and spread.

To design controls effectively, it helps to separate control design from control operation, because testing needs to evaluate both. Control design is the plan on paper, meaning what the control is supposed to do, how it is supposed to be applied, and what evidence should exist. Control operation is what actually happens in day-to-day practice, meaning whether people follow the process, whether monitoring is performed on schedule, and whether escalation happens when thresholds are crossed. Many control failures occur because design exists but operation is weak, such as when documentation requirements exist but are not enforced, or when monitoring dashboards exist but are rarely reviewed. A testing plan therefore includes both design verification and operating effectiveness verification. Design verification asks whether the control is appropriate and defined clearly for the use case, and operating verification asks whether the control is performed consistently and produces the expected effect. Beginners often focus on design because it is visible, but risk programs succeed or fail on operation. If a human review requirement exists but people routinely skip it, the control is not operating. If a threshold exists but nobody escalates when it is crossed, the control is not operating. Testing plans are what reveal these gaps early enough to correct them.

Preventative controls are often the first line of defense, and beginners should understand what preventative control verification looks like. Preventative controls include policy-based restrictions on tool usage, restrictions on data inputs, requirements for approvals before deployment, and controls that prevent high-impact automation without review. To verify these, the testing plan might check whether use cases have documented intended use boundaries, whether approvals were recorded before activation, and whether data input rules are being followed in practice. It might verify that only approved tools are being used in environments where sensitive data is involved and that unapproved tools are not accessible or are not being used. It might also verify that user access rights are appropriate, meaning only trained and authorized users can access restricted AI features. Preventative control testing is valuable because it catches problems before they become incidents, but it must be realistic because you cannot prevent every misuse through rules alone. That is why preventative controls should be paired with detective controls that find what slips through. For beginners, the key is to see preventative control verification as checking whether boundaries are enforced, not just written. When boundaries are enforced, risk declines because fewer harmful actions occur in the first place.

Detective controls are the next layer, and they are crucial for AI because AI behavior can change and because humans can make mistakes in how they use tools. Detective controls include monitoring for drift, monitoring for performance degradation, monitoring for unusual outputs, monitoring for fairness signals, monitoring for policy violations, and monitoring for incident patterns like increased complaints or rework. The testing plan for detective controls verifies that monitoring exists, that it uses meaningful Key Risk Indicators, that thresholds are defined, and that the monitoring is actually reviewed on the expected cadence. It also verifies that monitoring is targeted to the right systems, especially high-impact systems, and that monitoring does not focus only on general accuracy while ignoring harm proxies and control health signals. Another detective control is sampling and review, where a subset of AI decisions or outputs is examined periodically to detect patterns that metrics might miss. This can be especially important for generative outputs, where fluency can hide subtle errors or inappropriate content. Detective control verification also checks whether monitoring is connected to escalation, because detection without action does not reduce risk. For beginners, the important point is that detective controls are about early warning, and testing plans should confirm that warning signals are visible, reviewed, and acted upon.

Corrective controls are what allow the organization to respond quickly when risk increases, and they matter because no control system prevents all issues. Corrective controls include escalation triggers, authority lines to pause or restrict AI use, incident response procedures, and mechanisms to update documentation and controls after lessons are learned. To test corrective controls, the plan might verify that escalation pathways are known, that responsibilities are assigned, and that there is evidence of previous escalations being handled appropriately. It might verify that the organization can actually disable or restrict an AI feature quickly when needed, which is sometimes harder than people assume, especially for vendor features embedded in platforms. It might also verify that incidents are recorded, investigated, and linked back to risk register updates and control improvements. Corrective control testing often includes tabletop style verification of readiness, meaning checking that people know what they would do and that the process is documented and accessible. Even without running a full simulation, the organization can test whether contact lists are current, whether decision rights are clear, and whether monitoring thresholds are linked to action steps. For beginners, the key is to see corrective controls as a promise of responsiveness, and testing is how you prove that promise is real. If corrective controls are weak, the organization may detect a problem but respond too slowly, which allows harm to spread.

Fairness and transparency controls deserve special attention because they are often misunderstood as one-time evaluations rather than ongoing controls. Fairness controls can include pre-deployment evaluation across groups, limits on using AI for determinative decisions, requirements for human review and appeal, and monitoring for disparities over time. Testing these controls includes verifying that fairness evaluation occurred, that results and limitations were documented, and that ongoing monitoring continues to track relevant signals. It also includes verifying that the use case has appropriate transparency measures, such as documentation of limitations and, where appropriate, communication to stakeholders about how AI is used. Transparency controls also include traceability, meaning the organization can reconstruct who approved a system, what evidence was required, and what changes occurred. Testing transparency therefore includes checking documentation completeness, version history, and decision records. Beginners sometimes assume transparency means explaining model internals, but transparency in risk programs often means being able to explain purpose, limitations, and oversight in a way that supports trust and defensibility. When transparency controls are tested, the organization can show that it is not hiding behind complexity. This is especially important when decisions affect individuals, because challenges and complaints often focus on explainability and fairness. Testing ensures the organization can respond with evidence rather than with vague assurances.

Data controls are another major area because data is both the fuel for AI and a major risk source. Data controls include limiting what data can be used, ensuring data is collected and used lawfully, documenting data flows to vendors, and enforcing retention and access constraints. Testing data controls includes verifying that data classifications are applied correctly, that prohibited data types are not being input into unapproved tools, and that vendor data handling commitments are being followed. It also includes verifying that changes to data sources are captured and reviewed, because adding a new data input can change risk materially. For systems that involve external services, testing may include verifying that the organization understands whether data is stored, reused, or shared beyond the immediate use case, because those behaviors drive privacy exposure. A strong testing plan treats data controls as continuous, because employees may change behavior, vendors may update policies, and integrations may expand over time. Beginners should see that data control testing is not an optional compliance exercise; it is core risk management because data misuse can create immediate trust and legal harm. When data controls are tested regularly, the organization reduces the likelihood of accidental disclosure and improves its ability to defend data practices under scrutiny.

Now we need to address how often to verify controls, because testing frequency is a decision that must be proportional and realistic. High-impact systems require more frequent testing and monitoring review because harm potential is higher and because leaders have lower tolerance for surprise. Systems with high drift potential also require more frequent verification because conditions change quickly, and controls can become ineffective if not revisited. Systems that use sensitive data or that are customer-facing may require more frequent checks because privacy and trust harms can escalate rapidly. Lower-impact systems can be tested less frequently, but they still need baseline verification, especially for data handling and policy compliance, because even low-impact tools can create serious harm if misused with sensitive data. Testing frequency should also respond to signals, meaning if Key Risk Indicators trend upward or if incidents occur, the organization should increase testing or review temporarily until stability is restored. This adaptive approach prevents the program from being either complacent or overly burdensome. Beginners sometimes think of testing frequency as a fixed calendar rule, but a mature program treats it as a function of impact and signal strength. The goal is to test often enough to maintain confidence that controls are operating, without creating an unsustainable workload that causes testing to be skipped.

A strong testing plan also defines who performs the tests and what evidence the tests produce, because testing without accountability is unreliable. Some tests can be performed by operational owners as part of routine monitoring, such as reviewing K R I dashboards and documenting review outcomes. Other tests may be performed by independent reviewers for assurance, such as periodic checks that approvals exist, that documentation is current, and that monitoring reports are produced on schedule. Vendor-related controls may involve procurement or vendor management functions verifying contract commitments and reviewing vendor changes. Data controls may involve privacy or security functions verifying compliance with data restrictions and monitoring for policy violations. The testing plan should also define what happens when a test fails, meaning when a control is missing or not operating effectively, because failing tests should trigger corrective action, updates to the risk register, and potentially escalation if tolerance is exceeded. For beginners, the key is to see testing as part of governance, not as an afterthought. Testing is the mechanism that keeps leadership confidence justified, because it provides evidence that controls are not just designed but operating.

To make this feel concrete, imagine a high-impact AI system used to prioritize which customer disputes are escalated. Controls might include requiring human review for specific categories, restricting sensitive data inputs, monitoring misrouting rates and complaint trends, and documenting approvals and limitations. A testing plan would verify design by confirming that intended use boundaries and human review requirements are documented and approved. It would verify operation by checking that reviewers are actually performing required human review and that misrouting metrics are monitored and reviewed on schedule. It would verify data controls by sampling whether sensitive data is being used appropriately and whether data flows to vendors are within approved boundaries. It would verify corrective controls by confirming that escalation triggers exist and that the organization can restrict or pause the AI feature quickly if thresholds are crossed. It would include periodic checks of documentation completeness and change history, ensuring that updates are recorded and that monitoring thresholds remain appropriate. If K R I trends show rising misrouting, the testing frequency might increase temporarily, and the use case might be restricted until stability returns. This example shows how testing is not separate from operation; it is part of keeping the system within tolerance. Beginners should see that the testing plan is what makes control claims defensible.

To close, defining AI controls and testing plans is about deciding what safeguards must exist, how to verify they operate, and how frequently verification must occur to maintain justified trust. Controls include preventative boundaries, detective monitoring, and corrective response mechanisms, along with governance evidence like approvals, documentation, and clear ownership. Testing plans verify both control design and operating effectiveness, because controls can fail either by being poorly designed or by not being followed in practice. What to verify includes intended use boundaries, data controls, reliability and drift signals, fairness and transparency expectations, access and security protections, and incident readiness and escalation paths. How often to verify depends on impact, drift potential, data sensitivity, and observed risk signals, with higher-frequency testing for high-impact and rapidly changing contexts. A defensible testing plan assigns responsibility for tests, defines what evidence is produced, and defines what happens when a control fails, connecting testing results back into the risk register and governance decisions. When controls are tested consistently, the organization reduces surprise, improves response speed, and strengthens its ability to defend AI use under scrutiny. This sets the stage for the next topic, where we build ongoing monitoring as a living discipline that watches drift, performance, incidents, and emerging threats over time.