Chain-of-thought (CoT) prompting is a de-facto commonplace method to elicit reasoning-like responses from giant language fashions (LLMs), permitting them to spell out particular person steps earlier than giving a ultimate reply. Whereas the resemblance to human-like reasoning is simple, the driving forces underpinning the success of CoT reasoning nonetheless stay largely unclear. On this work, we carry out an in-depth evaluation of CoT traces originating from competition-level arithmetic questions, with the intention of higher understanding how, and which components of CoT really contribute to the ultimate reply. To this finish, we introduce the notion of a possible, quantifying how a lot a given a part of CoT will increase the probability of an accurate completion. Upon examination of reasoning traces by the lens of the potential, we establish stunning patterns together with (1) its typically robust non-monotonicity (because of reasoning tangents), (2) very sharp however generally powerful to interpret spikes (reasoning insights and jumps) in addition to (3) at occasions fortunate guesses, the place the mannequin arrives on the appropriate reply with out offering any related justifications earlier than. Whereas a number of the behaviours of the potential are readily interpretable and align with human instinct (akin to insights and tangents), others stay obscure from a human perspective. To additional quantify the reliance of LLMs on reasoning insights, we examine the notion of CoT transferability, the place we measure the potential of a weaker mannequin underneath the partial CoT from one other, stronger mannequin. Certainly aligning with our earlier outcomes, we discover that as little as 20% of partial CoT can “unlock” the efficiency of the weaker mannequin on issues that had been beforehand unsolvable for it, highlighting that a big a part of the mechanics underpinning CoT are transferable.
