Start with the action after the detection
A camera sees something. Then what? If nobody knows who should act, how quickly they must respond, and what evidence they need, the project has no operational definition. A detector can look convincing in a demonstration and still have no useful place in a factory, warehouse, shop, laboratory, or public space.
Begin with the decision that the image should support. State the event in operational terms, name the person or system that owns the response, and describe the allowed actions. An inspection system might hold an item for review. A stockroom system might create a check task. A vehicle-observation system might add a candidate event to a validation queue. These are different workflows, even when the underlying models use similar methods.
The timing requirement follows from the action. A result needed before a moving item leaves a station has a different budget from a result reviewed at the end of a shift. Measure the full interval from image capture to the point where a person or system can act. Model inference is only one part of that interval. Image transfer, decoding, queueing, post-processing, application logic, notification, display, and human review may take longer.
The NIST AI Risk Management Framework treats risk management as work carried through design, development, use, and evaluation. Its Map guidance also asks organizations to define expected benefits, performance claims, human roles, and oversight before choosing a design. For a Computer Vision project, that means writing the operating decision before comparing models or buying cameras.
Missed events and false events create different work
Feasibility depends on the consequence of being wrong. A missed event can allow a defect, unsafe condition, unavailable item, or unrecorded handover to pass unnoticed. A false event can stop a line, send staff to inspect normal conditions, fill a queue with noise, or erode trust in the system. The project team needs to understand both consequences because one threshold rarely minimizes both.
Avoid reducing the discussion to a single accuracy figure. Ask how often each error can occur before the workflow becomes uneconomic or unsafe. Then ask who reviews uncertain cases, what evidence appears in the review screen, and how much review work the operation can absorb. The answer may differ by event type, location, shift, product family, weather, or camera.
Acceptance criteria should reflect these consequences. Useful criteria may cover missed events, false-event volume, end-to-end timing, evidence quality, review effort, availability, and behavior when the input is unusable. Each criterion needs a test condition and an owner who can accept the result. A model score produced on a convenient test set does not answer those operating questions.
The scene is part of the system
Computer Vision performance is tied to the physical path that creates the image. Lighting can vary across a shift or season. Distance and lens choice determine how many useful pixels describe the object. Camera angle can hide a surface or distort a plate, label, gauge, or package. Movement creates blur. Other people, vehicles, tools, packaging, dust, rain, reflections, or machine parts can block the view.
Compression and transport also matter. A clear local feed may become less useful after a recorder, remote desktop, video-management system, or low-bandwidth link changes the image. Installation access can rule out the best theoretical viewpoint. Vibration, cleaning, maintenance, privacy zones, cable routes, power, and protective housings can constrain the final position.
Walk the real site before defining the dataset. Record candidate camera positions and collect samples through the conditions that matter to the operation. Include poor conditions instead of treating them as outliers. If the environment can be changed at reasonable cost, test that option too. A fixed light, controlled background, guide rail, trigger signal, slower inspection point, or better camera position may solve more of the problem than a more complex model.
This is also the point to decide whether the system should reject an image. An unreadable frame should not quietly become a confident business event. Define quality states such as usable, uncertain, obstructed, overexposed, stale, or missing, then decide how each state enters the workflow.
Representative data must include the awkward cases
A dataset is representative when it reflects the conditions in which the system will be judged. It should cover the actual cameras, locations, products, backgrounds, operating states, and image paths. It also needs enough examples of the events that matter. A long recording period can still produce almost no evidence for a rare failure.
Rare events require a collection plan. Existing archives may help, provided their image path and conditions match the intended deployment closely enough. Staged events can expose the system to controlled variations, but they may omit the disorder of normal work. Simulation or generated imagery can help with development questions, yet it should not replace a final test on real acquisition conditions.
Separate development data from the evidence used for acceptance. Otherwise repeated tuning can turn the test set into part of the development process. Keep source images, collection context, labeling rules, label history, and dataset versions. This makes disagreements and later changes traceable.
Ground truth also needs an operational definition. Two reviewers can look at the same frame and disagree because the event boundary is unclear, the decisive detail is hidden, or the label depends on information outside the image. Record ambiguous states instead of forcing a guess. When truth comes from another system, such as a machine signal, transaction, manual inspection, or physical measurement, verify the time alignment and identity link.
Test the full path under ordinary failures
A feasibility test should use the intended acquisition route, transport, preprocessing, model, application, review screen, notification path, and evidence store. It should include the people who will receive the result. Their response often exposes issues that a laboratory model test cannot show, such as unclear images, duplicate alerts, missing context, inaccessible screens, or an action that conflicts with existing work.
Test expected failures on purpose. Disconnect a camera or network link. Send stale, dark, blurred, compressed, obstructed, or malformed input. Restart a component. Change the upstream image size or frame rate. Fill a queue. Confirm whether the system reports the condition, buffers safely, recovers cleanly, and preserves enough information to explain what happened.
Human review should have a defined role. It may confirm uncertain events, correct a classification, decide whether an action is justified, or supply later ground truth. The review interface should show the relevant source frame or clip, time, location, quality state, model version, and any linked operating context. The NIST AI RMF Measure guidance recommends measuring and documenting human oversight and testing explanations with operators, users, and decision-makers.
Privacy controls belong in the same test. When personal data is processed, the GDPR requires principles including purpose limitation, data minimization, storage limitation, and data protection by design. The European Data Protection Board guidance on video devices gives more specific direction for video processing. The project should define what is captured, what is retained, who can see it, how long it remains available, and whether a less intrusive image path can meet the same operating need.
Traceable vehicle observations show what an event needs
One GAIAA case study involved an urban camera system that had to turn imperfect frames into structured vehicle observations and validated plate candidates. The engineering path detected and tracked vehicles, selected useful frames, localized and normalized plate regions, ran recognition on focused evidence, validated candidates, and retained visible quality states.
The useful output was not a bare character string. Each event kept the source frame, vehicle track, plate crop, candidate, and quality state. That evidence path supported review and made acquisition problems visible. Privacy controls were part of the event design. The case does not establish an enforcement use, and no such use is implied here.
The same principle applies elsewhere. An inspection event may need the item image, defect region, line and product context, model version, quality state, and reviewer decision. A shelf event may need the frame, mapped location, time window, and check outcome. The contents differ, but the event should retain enough evidence for the owner to act and later understand the decision.
Feasibility can lead to four sensible decisions
A good assessment does not have to end with approval. It can support four useful decisions: proceed with the proposed scope, change the physical or operating environment, narrow the question to a tractable event, or stop because the evidence and consequences do not support deployment.
Stop or redesign when there is no defined action, no owner, no lawful and practical acquisition path, no way to obtain representative evidence, or no credible ground truth. The same applies when the event is too rare to evaluate, the decisive information is not visible, false events would overwhelm the operation, missed events would carry unacceptable consequences, or the site cannot support the required installation and maintenance.
When the decision is to proceed, leave feasibility with a written test plan. It should name the operating event, response owner, image conditions, dataset boundary, ground-truth method, acceptance criteria, timing budget, privacy controls, review process, failure tests, and evidence record. The NIST TEVV-Athlon framework, published as an initial public draft in August 2026, follows a similar system-level idea: evaluation should be adapted to organizational goals and real-world outcomes. Because it remains a draft, it is useful as current direction, not as a settled standard.
If you are evaluating a Computer Vision opportunity, start with the operating decision and the conditions in which the system would have to work. GAIAA can take that assessment from site and data review through an end-to-end feasibility test, with the result framed as a decision about scope, evidence, risk, and next investment.