Since the frontier model scored 10/10, it saturated the benchmark, so there is no meaningful way to detect if the harness provided any benefit. You cannot conclude that a calculator provides no benefit when doing multiplication just because you ask me ten easy questions like "what is six times seven?" and I get all ten correct, the same as the calculator.
Given that, all the study found was that harnesses do work when the benchmark is not saturated. It does not provide any evidence one way or the other about the effect harnesses have on frontier model performance. Therefore, the recommendations in the "what this means in practice" section are unsupported.
Since the frontier model scored 10/10, it saturated the benchmark, so there is no meaningful way to detect if the harness provided any benefit. You cannot conclude that a calculator provides no benefit when doing multiplication just because you ask me ten easy questions like "what is six times seven?" and I get all ten correct, the same as the calculator.
Given that, all the study found was that harnesses do work when the benchmark is not saturated. It does not provide any evidence one way or the other about the effect harnesses have on frontier model performance. Therefore, the recommendations in the "what this means in practice" section are unsupported.