Don't Trust an AI Plan Until It Closes the Loop

Colleagues trust AI's proposed solution, but I don't dare to follow it directly.

This morning, I was sorting out a technical solution with a colleague when we ran into an awkward problem.

Handling it according to the current solution leads to another problem; if we prioritize solving the other problem, it breaks the current solution. I used AI to reason through several directions, and each direction seemed to have its own rationale, but when placed back into the whole system, none of them formed a closed loop. Some modifications even caused previously fixed bugs to reappear.

During the discussion, I felt the colleague trusted AI's conclusions quite a bit and leaned toward proceeding with its suggestions. I have concerns about this: if we assume a solution is fine simply because it came from AI, we might skip the most critical judgment - whether it holds up in the current system and whether it will bring back old bugs.

AI can quickly produce a well-organized answer, even generate code along with it. But an answer that looks complete does not mean the solution has been verified. Especially in this discussion, the directions I tried had not worked out; if we pick the most plausible answer and keep going, I think the risk is high.

What made me even more uncomfortable was that the discussion and the process seemed to be relying on AI generation. I had a feeling the colleague was too dependent on AI and might not have truly settled down to think through the problem. I cannot be sure how much thinking he actually invested; but when the boundaries, dependencies, and old issues of a suggestion are not clearly explained, I find it hard to treat it as an executable solution.

A solution cannot be judged only by whether each step makes sense

This problem reminded me of building blocks.

One block looks fine, another block looks fine, but if they don't interlock, and you simply stack them, the higher the pile, the more likely it collapses when a small change happens.

Technical solutions are the same. A local modification might solve the immediate problem but affect another workflow; a new branch might seem to handle exceptions well, but it could bring back old issues. Look at each step alone, and you can find a reason; connect them, and the input, processing, and output of the problem may not form a closed loop.

AI can help propose different directions and check some scenarios that are easy to overlook. But whether its suggestions are suitable for the current system still depends on whether we have clearly spelled out the service interfaces, historical issues, and various constraints, and whether we have ultimately done verification. For this discussion, the directions I reasoned through did not meet those conditions.

I would rather put the whole plan on the table first

At the time, I did not directly criticize my colleague's approach; instead, I kindly made a few suggestions: we should first clarify the requirements and the service interfaces that are depended on, and then write down the complete plan. I hoped the discussion would shift from "what answer did AI give" to "can this answer solve our problem, and how do we verify it."

This plan should at least answer a few questions: Where are the boundaries of the problem? How do we plan to handle it? After handling it, how does the entire workflow close the loop? Will previously fixed issues reappear? What happens in different scenarios?

In addition, we should consider in advance risks such as memory leaks, out-of-memory, and multi-threading races, based on the actual situation. These risks may not appear every time, but if the current change touches relevant parts, it is worth clearly explaining them in the plan, rather than relying on luck after the code is generated.

My idea is that once the plan is discussed clearly, then let AI help generate the code, and then review the plan and code together. During review, we should not only check whether new issues are handled, but also look back to see if old issues have reappeared. This time we have not yet found a direction that can close all issues, so these suggestions are only the working method I hope to adopt next, and they do not mean the solution has been resolved.

The division of labor between humans and AI can be more flexible than "who goes first, who goes later"

When I say "people clarify their thinking first," it does not mean that every time you must independently come up with a perfect plan before opening AI. Sometimes the problem itself is vague; asking AI to help list possible directions and point out scenarios that are missed can instead help us start thinking.

What really needs to stay in our hands is the judgment of the problem and the result: which constraints truly exist, which suggestions can be tried, what conflicts exist between options, and how to verify that the modification does not introduce new problems.

AI can help organize thoughts, add details, generate code, and participate in basic verification. But after it gives its conclusion, we still need to check it against the actual environment. Especially for bugs that have appeared before, without re-verification, merely looking reasonable is not enough to say the problem is solved.

The process of writing a solution is also a process of understanding the system

I increasingly feel that writing a solution is not just to deliver a document. When we sort out interfaces and dependencies, we re-understand the system; when we consider old issues and edge cases, we also discover places we previously overlooked. This process may take time, but it gives us more confidence in what we are going to modify.

I am willing to treat AI as an ability amplifier. It can help me explore and complete repetitive work faster, but after I type a sentence, the responsibility does not transfer.

This time I have not yet obtained a satisfactory answer. However, before continuing to have AI generate the next solution, I want to first clarify the connections between those building blocks with my colleague.