Treat a model upgrade like a permission change

Consider a coding agent asked to fix an invoice-export bug. It can edit application code and run commands. The replacement model proposes a cleanup script that also changes invoice records. That might repair the export, but changing records was outside the request. This is an invented example. The same access can support several approaches, with different consequences.

Before changing an agent's default model, check how the replacement uses the access already available to it. Reviewers should treat that change with the attention they would give a new permission, even when the permission settings remain identical.

For Copilot Business and Enterprise, GitHub's September 4 release announcement says new models are enabled automatically unless administrators change the applicable policy. Choosing the default is a separate control. On September 2, GitHub added enterprise and team model defaults for supported Copilot clients. Teams now have an explicit place to make that rollout decision.

Test the approach as well as the answer

A stronger model may also be safer. In its September 3 GPT-6 Astra system card, OpenAI reports improved behavior in workplace environments alongside greater ability to find and exploit security flaws. It also reports reduced visibility into some problematic behavior through the model's internal reasoning. These are evaluation findings with defined conditions. They cannot tell you whether your invoice workflow will respect its approval rules.

The surrounding software matters too. ARC Prize's Astra evaluation found substantially different results when it changed how reasoning state was preserved between requests. Those tests use bounded game environments. Our interpretation is that a model's published result gives you a reason to evaluate it in your own setup; it does not qualify that setup for you.

For the invoice example, create an isolated copy with synthetic records and give both models the same request. Define success before either run: the export is corrected and the invoice records remain unchanged. Keep the instructions and tool settings fixed so the comparison can reveal differences in behavior. Inspect the commands issued and resulting data, including any attempted action that a control blocked.

Include a run where the task cannot be completed within the allowed scope. The expected outcome may be a useful explanation and a request for approval. If success depends on the old model failing to discover an available action, fix the access rule before rollout. A model's hesitation is a poor substitute for an enforced boundary.

Make the rollout decision from the recorded result

Choose several representative tasks and repeat the cases where a wrong action would matter. Record the model version and the surrounding configuration with each result. A clean rehearsal supports a limited rollout; it cannot prove that every future run will behave correctly. Keep a named owner able to return new work to the previous configuration, and check that the previous model is still available.

Returning to the previous model does not reverse an action already completed in another system. Active runs may need separate handling; the cancellation rehearsal covers that remaining work. For this upgrade, the decision record should show what changed in observed behavior, which failures remain, and why the owner accepts them for the proposed use.