Artificial intelligence is moving into sensing, target recognition, sensor fusion, autonomy, and decision support across defense programs. Most of these systems are tested the way software has always been tested: does it meet its requirements in expected conditions? That question is necessary, but it isn't enough. An AI model that passes every requirement can still be fooled by an input its developers never imagined, or by one an adversary designed on purpose.
This guide is for program managers, chief engineers, T&E leads, and cybersecurity professionals responsible for fielding AI-enabled capability. It covers the policy and standards you'll be measured against, how adversaries attack AI, what a TEVV program looks like across the lifecycle, and the questions to ask before your next milestone. For a field-level view of the problem, see our analysis of adversarial AI risk in tactical weapon systems.
Why AI needs its own assurance discipline
Traditional software assurance rests on an assumption: behavior is specified in code, and if you verify the code, you've verified the behavior. Machine learning breaks that assumption in three ways.
Behavior is learned, not written. A model's decision logic comes from its training data. Two models with identical code but different data behave differently, so assurance has to cover the data pipeline, not only the software.
Performance is statistical and bounded by the operating envelope. A model that is 97 percent accurate on its test set says nothing about the 3 percent, or about conditions the test set didn't include: a different season, a new sensor, a camouflage pattern, a degraded signal. Assurance has to characterize where the model can be trusted, not just how often it's right.
The model is itself an attack surface. Adversaries don't need to breach a network to defeat an AI-enabled system. They can manipulate what the sensor sees, poison data the model learns from, or study a captured system to craft inputs that reliably fool it. These attacks work even against software with no conventional vulnerabilities.
AI assurance is the discipline of producing evidence, under realistic and adversarial conditions, that an AI-enabled system will behave acceptably across its intended operating envelope, and of knowing what it does when it can't.
The policy and standards landscape
Several documents define what "adequately tested" means for DoW AI. Program offices don't need to memorize them, but they should know which ones their reviewers will use.
DoD Directive 3000.09, Autonomy in Weapon Systems (updated January 25, 2023). For autonomous and semi-autonomous weapon systems, including those using AI, it requires rigorous hardware and software verification and validation, plus realistic developmental and operational test and evaluation. Some systems also require senior-level review before formal development and again before fielding.
DoD AI Ethical Principles (adopted February 2020). These commit the Department to AI that is responsible, equitable, traceable, reliable, and governable. "Reliable" and "governable" translate directly into test requirements: defined uses, tested safety and effectiveness across the system's lifecycle, and the ability to detect and disengage unintended behavior.
CDAO T&E frameworks and the DT&E of AI-Enabled Systems Guidebook. The Chief Digital and AI Office published frameworks in April 2024 for model-level, system-integration, human-systems integration, and operational T&E of AI. The Department's developmental T&E community published the Developmental Test and Evaluation of AI-Enabled Systems Guidebook in February 2025. Together they describe how AI testing fits into a program's T&E strategy.
NIST AI Risk Management Framework (AI RMF 1.0) (January 2023). A voluntary framework organized around four functions (Govern, Map, Measure, Manage) that gives programs a shared vocabulary for identifying AI risks and tracking how they're treated.
NIST AI 100-2, Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations (March 2025 edition). The common reference for classifying attacks on AI by the attacker's goals, knowledge, capabilities, and the lifecycle stage targeted. NIST plans regular updates as the field changes.
MITRE ATLAS. A public knowledge base of adversary tactics and techniques against AI-enabled systems, modeled on MITRE ATT&CK and built from real-world incidents and red-team demonstrations. ATLAS turns abstract attack categories into concrete techniques a red team can emulate.
How adversaries attack AI-enabled systems
NIST's taxonomy groups attacks on predictive AI into three classes. For tactical systems, each has a physical-world counterpart that matters more than its lab version.
Evasion attacks manipulate inputs at the moment of use so the model gets them wrong. In the lab, that might be imperceptible pixel changes. In the field, it's adversarial patterns painted on vehicles, decoys and camouflage designed against a specific model, or signals shaped to defeat an RF classifier. Evasion is the most direct threat to recognition and classification functions.
Poisoning attacks corrupt what the model learns. An adversary who can influence training data, through compromised data sources, mislabeled samples, or tampered pretrained models from the software supply chain, can degrade performance overall or plant a "backdoor" that triggers only on a specific cue. Poisoning is hard to detect after the fact, which is why data provenance is an assurance requirement and not just a data management practice.
Privacy and extraction attacks pull information out of the model: details of its training data, or enough of its behavior to build a functional copy. For defense systems, the concern is less personal privacy and more that a captured system, or a model exposed through an interface, lets an adversary develop evasion attacks offline.
Systems built on generative AI and AI agents add more risks: prompt injection (direct and indirect), manipulation of retrieved content, and misuse of the tools an agent is allowed to call. Any program using large language models in planning, analysis, or decision support should include these in its threat model.
Two features of the defense environment raise the stakes. First, adversaries are motivated, resourced, and adaptive: they'll study fielded systems and iterate. Second, AI attacks converge with electronic warfare and cyber operations. Spoofing a sensor, jamming a data link, and fooling a classifier can all be parts of one operation. Testing AI in isolation from the electromagnetic and cyber environment misses how it will actually be attacked.
A TEVV program across the lifecycle
Test, Evaluation, Verification, and Validation (TEVV) for AI isn't one test event. It's a thread that runs from requirements through sustainment.
1. Define the mission context and operating envelope. Before any testing, specify where the model sits in the mission thread, which decisions depend on its output, and the full range of conditions it must handle: environments, sensors, targets, weather, degradation. The operating envelope is the yardstick for every later test. If it isn't written down, there's no way to say whether testing was adequate.
2. Establish data provenance and quality. Document where training, validation, and test data came from, how it was labeled, and who could have influenced it. Keep test data independent of training data. Many "high-performing" models have simply been tested on data too similar to what they learned from.
3. Measure baseline performance across the envelope. Report performance by condition, not as a single headline number, so it's visible where the model is strong and where it degrades. Include metrics that matter to the mission (for example, the operational cost of a false negative versus a false positive), not only standard machine-learning scores.
4. Test adversarially, informed by the threat. Use the threat model to choose attacks. Test with varying attacker knowledge: white-box (full knowledge of the model), gray-box (partial knowledge), and black-box (only observing outputs). That reflects the range of access a real adversary might have. Keep the access your analysts have for testing separate from the access you assume the adversary has, so a lab attack that needs privileged access isn't reported as feasible through an operational interface. Prefer attacks that are physically realizable in the operational environment over purely digital ones.
5. Evaluate at the system level. The model is one component. Test what the system does when the model is wrong. Do downstream checks, sensor fusion, or the operator catch the error? A model weakness that the system architecture contains is a different risk from one that propagates straight to an engagement decision.
6. Include the human. Operators need to calibrate their trust: rely on the system when it's reliable, and recognize when it isn't. Human-systems integration testing checks that interfaces convey confidence and uncertainty in ways operators actually use under workload.
7. Mitigate, then re-test independently. Mitigations include adversarial training, input preprocessing and anomaly detection, model ensembles, architectural safeguards, and operational procedures. Each mitigation should be regression-tested so that fixing one weakness doesn't open another, ideally by someone other than the engineer who implemented it, and documented with evidence.
8. Monitor in operation. Models drift as the world changes, and adversaries adapt. Plan for performance monitoring, retraining triggers, and re-testing after every model update. A model update is a software release and needs the same discipline.
9. Package the evidence. Results should feed the program's T&E reporting, risk acceptance decisions, and the Program Protection Plan. Report verified findings, supported inferences, and unknowns separately, so decision-makers can see how much weight each result can bear. Evidence that exists only in a data scientist's notebook doesn't help a milestone decision authority.
Counter-AI: the other side of the same knowledge
Adversaries are fielding AI-enabled sensors, targeting aids, and autonomous systems too. Counter-AI applies the same understanding of AI weaknesses offensively: developing tactics, techniques, and procedures to deceive, degrade, or defeat adversary AI. Counter-AI informs electronic warfare, cyber, deception, and information operations planning.
Defensive assurance and counter-AI reinforce each other. Teams that understand how to attack AI are better at testing their own, and teams that test their own systems rigorously learn which attacks work in practice.
Blue and red in one workflow: E3DICT
In November 2026, AMPERSAND began executing E3DICT (Efficient Engineering Evaluation and Development of Counter-AI in Tactical Missile and Aviation Systems) under DEVCOM AvMC's ARISE (Artificial Intelligence Risk, Integrity, Security, and Evaluation) effort. E3DICT builds a repeatable capability that uses the same workflow to assess U.S. and partner AI ("blue") and to characterize adversary AI ("red") in aviation, missile, and integrated air and missile defense contexts.
Running blue and red through one workflow is deliberate. The principles behind it apply to any program building an AI assurance capability:
- Provenance and custody from intake. Every model, dataset, and runtime is recorded and versioned from the moment it arrives, and untrusted artifacts run only in isolated environments.
- Threat review as a standing function. Subject-matter experts review new threat intelligence, published research, and attack catalogs on a regular cadence, and turn the relevant methods into a versioned, reusable test library.
- Realistic adversary access. Each scenario states what the adversary could plausibly access, separately from what the evaluation team can access.
- Honest reporting. Findings distinguish verified facts, supported inferences, and unknowns.
- Independent retest. The people who implement a mitigation are not the ones who confirm it works.
- Fixed and mobile capability. Evaluation runs in the lab and can deploy to where systems are tested and fielded.
See E3DICT and our other programs on our past performance page.
Ten questions to ask before your next milestone
- Is the AI component's operating envelope documented, and does the test plan cover it?
- Can we trace the provenance of all training data and pretrained models?
- Is our test data independent of our training data?
- Do we report performance by operating condition, not just as an aggregate?
- Do we have a written threat model for attacks on the AI itself?
- Have we tested against physically realizable adversarial attacks, at varying attacker knowledge?
- Do we know what the system does, end to end, when the model is wrong?
- Have operators been tested on whether they can tell when to trust the system?
- Is there a defined process for re-testing after model updates?
- Do our AI assurance results appear in our T&E reporting and Program Protection Plan?
If several answers are "no" or "not yet", that's normal for programs early in AI adoption. Treat the gaps as the starting point for an assurance plan.
How AMPERSAND helps
AMPERSAND's AI & Counter-AI team combines threat systems, cybersecurity, and software engineering experience to deliver threat-informed AI resiliency assessments, adversarial ML red teaming, TEVV pipeline development, and counter-AI TTP development for aviation, missile, and tactical programs. Our evaluation pipelines are built for traceability and repeatability, and our deployable assessment approach brings testing to ranges and unit locations. Where independence matters, our software IV&V practice applies the same separation between developer and evaluator.