Improve a Method Without Curve-Fitting

Captain · CP05: Journaling and Improvement · About 25–30 minutes

Safety rule: Preserve the original evidence and include every loss, rejected setup and cost in your analysis.

Your mission today

By the end you can design a one-change strategy improvement and evaluate it on data that did not select the rule.

Before you start

Use the frozen CP01 strategy card, a demo execution report, your risk charter and a calculator. All numerical examples below are imaginary.

1. Understand the evidence problem

Curve-fitting means adjusting a strategy to match past noise rather than a repeatable market effect. When a learner notices a weakness, they should identify its cause and propose one testable change. Operational corrections, such as properly reading quotes, are different from changing the strategy's market rules. A new filter must be versioned and evaluated fairly.

After enough clean records, identify where errors arise: missed triggers, quote deterioration, invalid H1 pullback, spread stress or stop slippage. Fix operational errors first. Any strategy change requires a frozen rule, separate development and holdout sample, and truthful reporting of rejected variants.

A backtest or small sample describes observed outcomes. It cannot establish future profitability unless the method survives out-of-sample testing and realistic trading costs.

2. Worked example: what can go wrong?

Imagine four losing demo trades each had $2 excess spread cost, totalling −$8 in additional costs and making their total trading loss $8 worse. A student proposes rejecting entries above a written spread threshold. The new rule could prevent some costs but might also skip profitable trades. You cannot simply add $8 to historic P/L and declare success. Recompute the full offered set with eligible, skipped and unfilled cases, then use untouched holdout data.

The earlier preserved source also illustrated: The learner notices four trades with unusually high spread cost. They propose a maximum-spread filter, test it on the original offered opportunities and then test forward on unseen data. They must count trades the filter skips as well as kept trades, and avoid only comparing winners.

Original Black Box Journey educational diagram. Fictional results are not a verified market edge.

3. Follow a reproducible review

1. Classify the observed weakness: rule, execution, costs or data quality.
2. Write a falsifiable one-change hypothesis and version it.
3. Choose success metrics and exclusions before testing.
4. Evaluate all opportunities with costs, not selected winners.
5. Test on untouched later data; keep or reject the variant transparently.

A complete record is more valuable than a flattering summary. A reviewer should be able to reproduce the decision using the same information.

4. Identify the failure mode

A filter that retains 5 winners and rejects 95 other events may look impressive but be statistically fragile. Never optimise on the holdout then continue calling it unseen. Multiple candidate filters increase the risk of finding a lucky historical pattern.

Unsafe alternative: The novice keeps adding filters until the historical win rate looks perfect. Reusing the same data for selection and validation produces exaggerated confidence.

5. Produce the independent research deliverable

The original lab requires: Submit one single-change hypothesis, a predeclared scoring metric, and an untouched holdout test plan.

Record the original input, full trade set, execution costs, skipped events and the exact comparison. If a result changes after a correction, keep both versions.

6. Keep a holdout sample untouched while improving a method

Synthetic strategy experiment: Imagine 40 chronological eligible setups. Freeze an initial rule card and use the first 20 as a development sample. You may compare clearly defined rule variants there, recording all costs and rejected trades. Reserve the remaining 20 as a holdout sample. Do not inspect their outcomes while deciding which variant to keep. Otherwise those outcomes are no longer an independent check.

For a separate imaginary trade, a 10-pip stop on a $1-per-pip contract exposes −$10 before costs. A gross +12-pip exit would be +$12, but $2 in hypothetical execution costs would reduce net to +$10. If the strategy only appears attractive before spread and commission, the correct conclusion is that its cost-adjusted edge is unproven.

Research drill: Write one question, fixed entry/exit rules, instrument assumptions, costs, test split and a rejection threshold before opening the holdout. Record losing months, gaps, skipped opportunities and changes tested. If holdout results fail, do not secretly retune on those same outcomes and call the retuned model 'validated'.

Common mistake

Reporting a favourable final P/L as if it proves every earlier decision was valid. Keep process compliance and statistical outcome in separate columns.

Plain-English glossary

• Curve-fitting: Rules selected mainly to fit past noise.
• Hypothesis: Specific idea that can be tested and disproved.
• Variant: Frozen changed version of a strategy.
• Holdout: Data excluded from development.
• Offered set: All candidate opportunities, including rejects.

Try it yourself

Submit one single-change hypothesis, a predeclared scoring metric, and an untouched holdout test plan. Include at least one losing and one rejected observation. Write the denominator and units beside every statistic.

What you can now do

• Design a one-change strategy improvement and evaluate it on data that did not select the rule.
• Show an adverse case using exact currency or R units.
• Explain the limits of the evidence without promising profit.

Next lesson

CP06.1 · Closed-Bar Recognition Examination.