Proactive AI suggestions worked best after developers committed
When should a coding assistant interrupt? That question matters as much as whether its suggestion is correct. Useful advice delivered in the middle of a thought can still be a bad feature.
A five-day field study from JetBrains, Carnegie Mellon University, and Delft University of Technology gives the problem some unusually concrete numbers. The researchers built ProAIDE inside JetBrains Fleet and watched how professional developers used it during normal work. The system produced 229 interventions across 5,732 interaction points.
Timing separated the welcome prompts from the irritating ones. Developers engaged with 52% of suggestions shown after a commit. When the assistant followed a rejected AI edit with another offer of help, engagement fell to 31% and active dismissal rose to 62%.
Proactivity, in other words, is an interruption policy. It is not just another model capability.
Three triggers at three stages of work
ProAIDE intervened at three moments:
- An ambiguous prompt trigger noticed a short or unclear request and offered to refine it.
- A declined AI edit trigger followed a rejected edit with an offer to continue in chat.
- A post-commit review trigger appeared after “Commit Changes” and offered to inspect the finished unit of work.
Before the field study, the team ran contextual inquiries with seven professional developers and revised the prototype. The five-day study then involved 15 professionals completing self-directed tasks across 12 programming languages. Telemetry recorded whether they engaged, dismissed, or ignored each intervention. It did not collect source code, filenames, or complete model inputs and outputs.
The three triggers produced different behavior. Ambiguous-prompt interventions received 46% engagement, 23% dismissal, and 31% no response. Declined-edit follow-ups performed worst. Post-commit review performed best, and its engagement pattern remained steady even as active use dropped from 18 machines on day one to eight on day five.
Rejection does not mean “try harder”
A rejected edit can look like an invitation for the assistant to make another attempt. The study suggests that interpretation is usually wrong.
The developer may already know the next step. They may have rejected the edit precisely because the model misunderstood the task. Pushing the interaction into chat then adds a mode switch at the moment the developer is trying to recover control. Participants called this trigger intrusive and confusing.
Product teams need to separate two signals: “the model failed” and “the user wants more help.” Rejection is weak evidence for the first and poor evidence for the second. A better default is to lower the prompt's prominence, keep help available, and wait for an explicit request before escalating.
Coding agents need the same restraint. A failed tool call, reverted patch, or rejected diff should not automatically launch a larger plan. The interface should distinguish correction from refusal. Good agent tool-interface design follows the same principle: produce reviewable results and leave the next action with the operator.
Why post-commit review fit
A commit marks a clean change in the developer's job. Implementation pauses and evaluation begins. At that boundary, a lightweight code-quality check belongs to the work instead of cutting across it.
Developers spent a mean of 45.4 seconds interpreting proactively triggered suggestions and 101.4 seconds on reactive sessions. The authors' Wilcoxon signed-rank test found a significant difference (W = 109.00, p = 0.0016, r = 0.533). That does not isolate timing, though. Proactive sessions mostly delivered ready-to-apply patches, while reactive sessions contained more example snippets. The output format probably explains some of the gap.
Start with workflow boundaries before choosing prompts or models. A commit, completed test run, failed build, opened pull request, or finished deployment may create a useful opening. Test each one separately. “After a commit” and “after a failed test” represent different intentions, even if both happen between coding actions.
Engagement is not proof of quality
Participants gave ProAIDE an average System Usability Scale score of 72.8 and generally found it easy to learn. Only 27% rated its chat suggestions as reliable, however, and 47% felt comfortable accepting them. A prompt can be well timed and easy to use without being trustworthy enough to apply.
The 52% engagement rate has the same limit. It does not show that half of the post-commit suggestions improved the code. Privacy safeguards prevented the researchers from collecting source code and full model outputs, so they could not measure code quality directly. The strongest evidence is about receptivity and workflow fit, not fewer defects.
An evaluation should therefore record trigger type, engagement, dismissal, ignore rate, time to first action, and whether the resulting diff was accepted or reverted. Model and provider details belong in separate fields from the trigger policy. If calls run through a multi-model API such as api.ish.chat, that split helps diagnose whether a poor result came from timing, model behavior, or the tool chain. The approach in instrumenting AI agents is a practical starting point.
Build the interruption policy first
A cautious implementation would:
- Anchor review prompts to explicit task boundaries.
- Treat rejected edits as a request for control, not permission to escalate.
- Show a small cue first. Generate or apply a patch only after engagement.
- Preview diffs and explain why the prompt appeared.
- Let developers configure timing, frequency, and suggestion type.
- Evaluate interruption policy separately from model quality.
The evidence is useful, but narrow. The participant group was small, familiar with JetBrains tools, and highly experienced with AI assistants. The study lasted five active days, usage declined, and the intervention covered code-quality suggestions inside one IDE. The authors also note limited statistical power. Longer studies across other editors, teams, and kinds of work are needed before anyone treats 52% as a universal benchmark.
The product decision is still clear enough to test now. Do not ask only whether the assistant can notice something useful. Ask whether the developer has reached a moment when another suggestion is welcome. Timing is part of the feature.
