The Statistic That Passed Every Test I Had And Still Isn't an Edge
Most of what I publish is about killing ideas. This one is strange and caught me off-guard: An idea that survived everything.
It cleared the control test that kills almost everything I try and that I still won’t trade. That’s because there’s a gap between those two things and it is the most useful thing I’ve learned building this system, so let me show you what actually happened.
The pattern
For this particular example, I am trading the market on the daily gaps. I noticed what looked like a pull toward round numbers. This means price approaching a round hundred from just below seemed to complete the move up to it more often than chance. A sort of “magnet,” in trader folklore.
Folklore is usually the thing that confirms noise, so I tested it the way I have learned to test everything: against a control. Testing against a control is a sure way to determine it.
The claim: on day one, when price is positioned just below a round-number level, it completes the move to that level more often than a random position would.
The measurement, N = 11,697:
Round-number completion (day 1): 67.4%
Random-control completion: 52.7%
Edge: +14.7 points
Fourteen and a half points over a random control, on nearly twelve thousand samples, that seems almost too good to be true. That is not a rounding error, and it is not multiple-comparisons luck, the control is the multiple-comparisons check. It has been proven that most of my ideas die right here, matching their random twin within a point or two. Surprisingly, this one didn’t. It survived.
So I have a real, replicable, controlled statistical effect. Here’s why it’s still not in my trading system.
Reason one: real ≠ tradeable
A control test answers exactly one question: is this pattern distinguishable from noise? Here, yes. What it does not answer is whether you can capture the pattern with an order.
Those are different questions, and the gap between them is where most “edges” die. To trade the magnet you have to specify an entry, a stop, and a target; and the moment you do, you’re no longer measuring “does price complete the move,” you’re measuring “does a specific trade structure make money after costs.” Every version I built to capture the 67.4% either:
filled too late to catch the completion, or
got stopped out on the paths that completed eventually but dipped first, or
raised the win rate and lost money anyway. This is the classic trap. One build lifted win rate from 62.1% to 62.7% and cost $1,160 at the end of the test. A higher win rate on smaller wins and fatter losses is not an improvement; it’s a worse strategy wearing a better statistic.
The completion stat was real the whole time. It just didn’t survive contact with an order.
Reason two: no mechanism
The survivors in my system all have a why. A reason the edge should exist that isn’t “the backtest liked it.” When I dug into the magnet, the honest finding was deflating: past day one, the effect is better explained as recurrence; price revisiting recent levels because it’s ranging, than as any genuine round-number pull. The “magnet” label was a story I’d painted onto a more mundane phenomenon.
A statistic with no mechanism is a coin that has landed heads 67% of the time. It might be a weighted coin. It might be a fair coin in a streak you’ll pay to discover is over. Without knowing why, you can’t tell which, and betting real money on the difference is how the market extracts tuition.
Reason three: passing one test is a floor, not a verdict
Beating a control means the pattern isn’t noise. That’s a floor; a necessary condition, not a sufficient one. Above that floor sit all the questions the control can’t answer: does it hold every year, does it survive realistic fills, is there a mechanism, can an order capture it. The magnet clears the floor and fails the ceiling.
What I actually did with it
Not trade it. Not delete it either. I put it on a forward ledger, a read-only log that records what the pattern would have done, live, going forward, without risking a dollar. If the 67.4% is real and capturable, the ledger will show it surviving out-of-sample in daylight, and then it earns a place. If it was a ghost, the ledger catches it costing imaginary money instead of real money. Unfortunately, this is a very time consuming use of it because I need to gather enough forward trading data before being able to validate it.
That’s the honest home for a statistic that’s too strong to dismiss and too unexplained to trade: neither the trash nor the portfolio, but a lit room where it has to prove itself forward, on data that didn’t exist when I found it.
The through-line, and it generalizes well past markets: a result that beats a control has earned your attention, not your capital. Distinguishable-from-noise is the beginning of the investigation, not the end. The most disciplined thing you can do with a genuinely surprising number is refuse to trade it until you know why it’s true, and build yourself a way to keep watching it that costs nothing while you find out.
Reproducible: the claim is round-number day-1 completion vs a random-position control, and the whole point is that you can run the control yourself and see the 67% hold, then try to turn it into an order and watch it evaporate. I write these up as I hit them. If they’re useful, my email’s janheger89@gmail.com.