Workflow and AI Deployment: Whoever Signs the Output Should Sign the Model Swap #AIg

A model change can pass for a settings change: an engineer points the application at a newer or cheaper model, the demo looks fine, and the workflow keeps running. Basil C. Puglisi reads that habit as a governance gap, because the people who sign the workflow’s output never get to test what changed underneath it.

Puglisi’s position is direct: swapping the AI model inside a live workflow is a workflow change, and the acceptance test belongs to the named owner who answers for what the workflow produces. A new survey and a new deployment announcement show why.

Survey data shows model swaps often take weeks of engineering

The October 2026 VB Pulse Agentic Orchestration Report from VentureBeat Intelligence is one recent reading on this. The firm surveyed 106 people, and 100 qualified: people at organizations with 100 or more employees, plus consultants serving them. Of those respondents, 47% say their organization replaced the model in a production AI system in the past 12 months while keeping the surrounding application.

Among the 45 respondents whose organizations swapped, 69% spent a few weeks or more on the swap, and 36% spent more than a month, up from 12% in September. Only 13% called the swap close to a configuration change. The report explains why: the app around the model, meaning “its prompts, tools and tests,” often has to be reworked to match.

Performance fell for a large share as well: according to the same report, 44% of swappers saw performance drop and either reworked the application or went back to the original model. That figure was 58% in September, and the report says the change is not statistically significant.

These are survey results from a small panel, and the report itself cautions that 88% of qualified respondents came through a research panel and that it cannot separate changes in who answered from changes in what organizations do. Treat them as a pattern worth testing in your own shop, since they cannot serve as an industry benchmark.

Failures reach customers and reports, and 22% of respondents can’t say

The same report asks about failures. Of respondents, 36% say an AI agent or automated workflow failed in the past 12 months in a way that affected a customer, an employee or a business process, and another 22% don’t know whether one did. Among the 34 who reported a failure, 56% of the most significant failures reached customers or a financial, regulatory or board report, while 38% were caught internally.

That 22% matters for a model swap. A team that cannot tell whether a workflow failed cannot tell whether a new model made it worse. Control also looks thin at the platform level: of those expecting a hybrid of provider controls and their own, 21% have documented which workloads each side governs, and 42% of those who answered cannot point to anything that automatically stops their agents’ token spending.

Northern Trust’s analyst work shows where a test set comes from

On November 9, 2026, Microsoft and Northern Trust announced a client onboarding agent. These figures are company statements and carry no independent verification. The release says the agent cut data preparation time by 83%, from a baseline of approximately two and a half weeks to an average of three days across measured production cases, with a 96.5% F1 first-pass quality score. It describes F1 as a standard AI evaluation metric combining precision and recall.

The release also says the workflow has human oversight embedded throughout, so analysts can review outputs, evaluate reasoning and resolve exceptions, and that analysts can review, refine and approve the agent’s work. Northern Trust describes its firm-wide approach as “keeping judgment and accountability with people.”

A footnote in the release explains the score. It comes from controlled testing across seven representative test cases, where agent output was compared with analyst-prepared reference outputs withheld during execution, before analyst feedback or corrections. The release also says Microsoft tuned against F1 scoring during development and does not say whether the seven test cases sat outside that tuning. Puglisi reads that detail as the most useful line in the announcement. Those analyst-prepared references are the asset that makes a later model swap testable. Without them, a team has little beyond the vendor’s own benchmarks to test a swap against.

Machine checks screen a swap, and a named owner signs it

Puglisi’s HAIA-RECCLIN work describes Dispatch as sending different jobs to different AI platforms, chosen by what each has proven it does well. The same page argues that fabrication is a property of every current system, so the question is whether the workflow around it catches the mistake. It makes the case for “putting the checkpoint around the model rather than inside it.” It also observes that many organizations crossed the line into agents acting between reviews through vendor updates nobody read. A vendor-pushed update can be a model swap nobody requested.

Automated regression suites and LLM-as-judge scoring have a place. Puglisi files them under Responsible AI, where the machine checks the machine. They screen a swap quickly and cheaply, and the signature belongs to a named human. Under Checkpoint-Based Governance (CBG), AI Governance means a named human holds binding authority to accept, modify or reject, and the decision leaves a record of who decided, on what evidence and why.

Five controls and three KPIs for an operations team

Puglisi would advise a client operations team to start with these controls:

1. Keep a register per workflow. Record the model and version, the prompt and tool set, the named owner, and any change a vendor pushes.
2. Build a reference set. Collect human-prepared outputs for the workflow’s real cases and hold them back from prompt tuning, so the test stays honest.
3. Treat every model change as a change request. Rerun the reference set, then the workflow owner accepts, modifies or rejects the swap before it touches customers or reports. Write the rollback path before release.
4. Sample live output after the swap. Review a fixed share of production results for a set period, since 22% of the surveyed respondents don’t know whether an agent or workflow failed.
5. Document who governs what. Write down which workloads the vendor platform governs and which your own layer governs, and name who stops runaway spending.

Three KPIs make the controls measurable on an executive dashboard:

• The share of production workflows with a current reference set.
• The share of model changes, vendor updates included, that received a signed accept, modify or reject before release.
• The defect or exception rate in the first 30 days after a swap, compared with the prior model.

Limits, and the update nobody requested

Both anchors carry limits. The survey bases are small (45 swappers, 34 failures, 34 expecting a hybrid control plane) and the panel is not a random sample. The Northern Trust numbers are company-reported, and the F1 score rests on seven test cases. CBG and the reference-set practice are Puglisi’s proposals for others to test, challenge, improve or replace; no study here validates them.

The unresolved piece is the update that arrives with no change request. A register can only record what a team learns about, so the next question is contractual and operational: what notice a vendor gives before it changes a model, and who in the client’s organization receives it. Teams should find that answer before the next update lands.

Sources

• Microsoft Source. (2026, November 9). Northern Trust and Microsoft Frontier Company collaborate to transform client onboarding through AI. https://news.microsoft.com/source/2026/11/09/northern-trust-and-microsoft-frontier-company-collaborate-to-transform-client-onboarding-through-ai/
• Puglisi, B. C. (2026). Checkpoint-Based Governance (CBG). basilpuglisi.com. https://basilpuglisi.com/cbg/
• Puglisi, B. C. (2026). What is HAIA-RECCLIN? Reasoning and Dispatch. basilpuglisi.com. https://basilpuglisi.com/haia-recclin/
• VentureBeat Intelligence. (2026, October). VB Pulse Agentic Orchestration Report. VentureBeat. https://venturebeat.com/vb-pulse-agentic-orchestration-report

Questions readers ask

Is swapping the AI model in a workflow a configuration change?

Rarely. In the VentureBeat Intelligence October 2026 survey, only 13% of 45 swapping organizations called it close to a configuration change. Prompts, tools and tests built around the old model often need rework, so Basil C. Puglisi treats a swap as a workflow change that needs an owner’s sign-off.

What is a reference set for an AI workflow?

A reference set is a collection of human-prepared outputs for the workflow’s real cases. A team reruns the new model against it after any change and compares results. Puglisi advises holding the set back from prompt tuning, so the comparison measures the model and not the tuning.

Why isn’t an automated evaluation enough to approve a model swap?

Automated evaluations and LLM-as-judge suites are machine checking machine, which Puglisi classifies as Responsible AI. They screen a swap efficiently. Approval needs a named human with authority to accept, modify or reject, which Checkpoint-Based Governance calls AI Governance, plus a record of the decision.

Are vendor updates model changes?

Often. A vendor-pushed update can alter model behavior without a team choosing it. Puglisi advises logging vendor updates in the same register as deliberate swaps and routing each through the same accept, modify or reject review, because the workflow’s output reaches customers either way.

How reliable are the figures behind this advice?

They carry limits. The VentureBeat survey has small bases (45 swappers, 34 failures), and 88% of qualified respondents came through a research panel. The Northern Trust results are company-reported. Treat both as signals to test locally, and treat the controls as proposals.

#AIg