How ChatGPT with Codex and Computer Use Became My QA Agent

A feature can pass type checking, unit tests, and E2E tests, yet still reveal a problem when someone actually uses it.

Manual testing therefore remains necessary. You still need to open the application, follow a user flow, and observe what happens on screen.

Until recently, this validation was human in the most literal sense: I was the one doing it on my projects.

I now use ChatGPT's Computer Use feature with Codex mode. The AI can observe the screen, click, type, and navigate the interface. I no longer ask it only to build a feature and pass the automated tests. I also give it the manual test scenarios described in my specs.

Codex has become my QA agent.

Code moves faster, verification becomes the bottleneck

In my article about the shift from vibe coding to agentic engineering, I explained how AI compresses implementation time much more than the other stages of software development.

Code that used to take several days to write can now be produced in a few hours. We still need to verify that the product works as intended. The more we produce, the more the bottleneck moves toward verifying the result.

In my projects, verification already relies on several layers. Type checking validates type consistency. Unit tests validate business rules. Integration tests verify that several components work together. E2E tests protect critical user flows.

These automated tests are essential, but they mainly verify the behaviors we anticipated and turned into assertions. On their own, they do not guarantee that the final experience is clear and pleasant to use.

The code can be correct while the product is still broken.

Computer Use adds another layer. The AI runs the scenarios I previously completed by hand after type checking and automated tests had passed.

Computer Use lets Codex see the product it builds

OpenAI launched Computer Use in ChatGPT Codex in April 2026. The feature allows the agent to observe the screen, click, and type with its own cursor. OpenAI explicitly mentions frontend iteration and application testing among its uses for developers. The Computer Use announcement describes how Codex is moving beyond code.

The feature is available in the ChatGPT desktop application on macOS and Windows. It requires the Computer Use plugin and specific permissions for the applications it needs to control.

In practice, several surfaces can be used to test a product:

• the browser built into Codex;
• Chrome with the ChatGPT extension installed;
• Computer Use for a desktop application, a mobile simulator, a flow across several applications, or an interface that cannot be verified from the terminal.

Codex no longer looks only at the files that describe the application. It can observe the result presented to the user.

This closes the loop between intent, code, and actual behavior.

My Markdown spec already contained the QA plan

In my context engineering method, every significant feature starts with a Markdown spec.

It includes:

• the user problem;
• the expected outcome;
• what is in and out of scope;
• product and technical decisions;
• acceptance criteria;
• the automated and manual tests to run.

I was already using the manual scenarios as a final checklist after the automated checks. The difference is that I no longer have to run all of them myself.

The same spec now drives three stages:

1. an implementation agent builds the feature;
2. it runs type checking and the automated tests;
3. a QA agent reads the manual scenarios and runs them with Computer Use.

The official documentation also recommends pointing Codex to an existing test plan in the repository so that its QA pass follows the flows already defined. The official guide to QA with Computer Use also recommends specifying the environment, critical flows, and expected report format.

A simplified section from one of my specs might look like this:

## Manual tests

### Environment

- Use the local environment.
- Use the `qa@example.com` test account.
- Enable the `new-onboarding` feature flag.
- Do not perform any action on a real account.

### Scenario 1: complete onboarding

Preconditions:

- The account has just been created.
- No project exists yet.

Steps:

1. Open the dashboard.
2. Start onboarding.
3. Complete all three steps with the test data.
4. Submit the form.

Expected result:

- The user lands on their first project.
- A confirmation message appears.
- Onboarding does not appear again after reloading the page.

Things to check:

- Check loading states.
- Check the French and English copy.
- Report any visual misalignment or action without feedback.

The spec does not merely tell the agent where to click. It also tells it what should happen after each action.

This matters because an agent capable of navigating an interface does not automatically know how the product is supposed to behave.

A French CFTL study illustrates the current gap. Among 258 testing professionals surveyed in late 2025, 56.05% said they were implementing or planning to implement AI-assisted manual test case design, but only 19.75% said the same about test execution. We already use AI to write the plan. We are now starting to let it execute that plan. The full survey results are available from the CFTL.

My QA workflow with Codex

This pass starts once type checking and the automated tests are green. Computer Use is not a way to compensate for a failing test suite. It adds product verification through the interface.

1. I prepare a safe environment

I prefer a local or staging environment with an account and data reserved for testing.

Computer Use can act within an authenticated session. I do not want a misunderstanding to create a real order, change a client's data, or trigger a payment.

I specify the feature flags, the initial account state, and the data the agent may use. If the scenario requires a sensitive action, I stay present and retain final approval.

2. I give it the spec as its source of truth

The agent reads the feature description, acceptance criteria, and manual testing section.

I do not simply ask it to "test the application." Such a broad instruction would force it to invent the scope, expected behaviors, and severity levels.

I also tell it which problems to watch for:

• broken functionality;
• navigation errors;
• missing loading states;
• inconsistent copy;
• visual issues;
• actions without feedback;
• regressions in an existing user flow.

3. The first pass produces only a report

I keep testing and fixing separate whenever possible.

During the first pass, the agent does not modify the code. It runs the scenarios and documents any discrepancies. This prevents it from immediately fixing a symptom without preserving evidence of the original problem.

For each issue, I ask it to provide:

• the reproduction steps;
• the expected result;
• the actual result;
• an estimated severity;
• a screenshot when helpful;
• any scenarios it could not test.

That last category matters. I prefer an explicit "not tested" to a report that makes everything look successful.

4. I triage issues with my product judgment

Codex can notice that a button does not respond. It can also spot overflowing text or navigation that leads to the wrong screen.

It is less reliable when deciding whether friction is acceptable, whether it understood a business rule correctly, or whether a detail should block a release.

I therefore remain responsible for triage:

• is the problem real and reproducible?
• does it block the user?
• does it come from the feature or the environment?
• should it be fixed now?
• does it deserve an automated regression test?

5. Codex fixes the issue, then replays the exact same scenario

Once I confirm the issue, I assign the fix to another agent. I keep this task in a separate conversation to preserve the context of the QA pass. That agent investigates the cause, proposes a minimal fix, and runs the relevant automated tests.

The QA agent then replays the manual scenario that previously failed.

The loop becomes:

The same document preserves the original intent throughout the cycle.

Computer Use and automated tests look for different things

I am not opposing the two approaches. They complement each other.

When Computer Use discovers a critical bug that can be reproduced reliably, I turn the scenario into an automated test. The QA pass reveals the problem, and the lasting test prevents it from returning.

The agent sometimes wants to complete the flow instead of finding the bug

Navigation agents are generally trained to achieve a goal. When a button does not respond, they may try another route and finish the task without treating the obstacle as a defect.

This behavior is useful for an assistant. It is much less useful for a tester.

A study published at ACL 2026 describes two limitations of GUI agents:

• goal-oriented masking, when the agent bypasses an issue to complete its mission;
• execution-bias attribution, when it mistakes a genuine bug for an error in its own clicking or manipulation.

Even the specialized system evaluated in the study remains far from perfect defect detection. The GUITester paper describes these limitations of exploratory testing with GUI agents.

I therefore add explicit rules to my scenarios:

• do not silently bypass a blocker;
• record every difference from the expected result;
• retry an action once to distinguish a bad click from a reproducible bug;
• separate product issues from network or environment problems;
• never mark a scenario as successful if a step was not observed.

Once again, the quality of the result depends less on a clever prompt than on the context and validation criteria given to the agent.

What I do not delegate to it

Computer Use does not replace a complete testing strategy.

I do not let it handle the following on its own:

• the decision to release;
• the evaluation of business or legal risk;
• security testing;
• performance and load testing;
• the validation of payments or real data;
• comprehensive accessibility testing;
• the final judgment on the quality of the experience.

The agent can also produce false positives, miss an issue, or fail because of slow loading. It can see the contents of authorized applications, screenshots, and some displayed data. Permissions and test accounts are therefore not minor configuration details.

The Computer Use safety documentation recommends keeping each task within a clear scope, closing sensitive applications, and remaining present for flows involving accounts, payments, credentials, or security settings.

I treat it as a fast, persistent, and still imperfect collaborator. Not as an automatic "ready for production" stamp.

My role as a Product Engineer shifts again

When AI writes the code, my value no longer comes from producing every line myself.

When it handles some of the manual testing scenarios, my value no longer comes from personally clicking through every feature either.

My work increasingly consists of:

• defining the behaviors that matter;
• identifying the riskiest user flows;
• preparing a reliable testing environment;
• writing observable acceptance criteria;
• interpreting issues;
• deciding what can be released.

Computer Use automates the interactions. It does not decide what makes a good product.

The real change is not simply that Codex can click for me. It is that the same Markdown spec can now drive both the construction and verification of a feature.

I write the intent. The agent builds the feature and runs the automated tests. A separate pass verifies the manual scenarios with Computer Use. I retain final judgment.

This creates a much more complete Product Engineering loop.

📌 Do you have a web or desktop product to build, a feature to make more reliable, or a development workflow to structure around AI agents? Discover my Product Engineering services.

Sources

• OpenAI, "Codex for (almost) everything".
• OpenAI, Computer Use documentation.
• OpenAI, "QA your app with Computer Use".
• CFTL, "2025 CFTL AI survey results".
• Gao et al., "GUITester: Enabling GUI Agents for Exploratory Defect Discovery".
• Kong et al., "WebTestBench: Evaluating Computer-Use Agents towards End-to-End Automated Web Testing".