Captain · Independent Research and Operations · About 30–40 minutes
Safety rule: Never repair missing historical prices by inventing candles that favour your strategy.
Your mission today
By the end you can audit chart completeness, timestamps, quote sides and contract changes before using data for research.
Before you start
Complete CP07.1. Bring a small fictional EURUSD OHLC file and a spreadsheet. Use no real trading order to test uncertain data.
1. A beautiful chart can still contain bad records
A weather chart is only as trustworthy as its thermometer and timestamps. Price charts have the same problem. Data quality describes whether a dataset is accurate, complete, consistent and suitable for a stated question. A candlestick often contains OHLC, the open, high, low and close prices for a time interval. Those four numbers do not reveal the exact route taken inside the bar. They also may represent Bid quotes, midpoint quotes or a particular trading venue's prices. A broker chart is not a universal EURUSD transaction tape. Record source name, collection date, timezone, instrument code and quote convention. If a strategy uses an Ask execution but the data only contains Bid candles, its cost assumptions need explicit support. Never hide that limitation beneath a smooth chart.
2. A missing bar can change the apparent trade
Imagine an H1 EURUSD series that should contain 100 consecutive bars across a selected period with no expected market closures. Only 97 are present. Completeness is 97 ÷ 100 = 97% under that assumption. Three bars are missing. If one missing bar contained an unseen low that would have reached a stop, a backtest using only the remaining candles may report a fictitious winner. Suppose a long at Ask 1.08520 has a planned Bid stop at 1.08420. Its 10-pip adverse distance risks −$10 at an invented $1 per pip before costs. If the missing interval prevents checking whether that stop was reached, label the outcome UNKNOWN. Do not interpolate a convenient candle and report a $15 profit. A missing-data flag is a research result, not a shameful gap to conceal.
3. Audit each candle's internal consistency
For a simple price bar, high should be at least as large as both open and close, while low should be no larger than either. A bar with open 1.08500, high 1.08480 and close 1.08520 fails that check because the recorded high is below its open and close. Duplicated timestamps, negative prices for instruments that cannot validly have them under the data model, sudden unit changes or zero-volume placeholders also deserve inspection. Not every surprising price is invalid, so the rule should flag candidates for review rather than deleting all large moves automatically. Preserve the original source file. Produce a separate cleaning log describing each detected issue, whether the record was retained, excluded or corrected from a verified source, and why. An analyst who silently edits a historical candle cannot honestly call the study independently reproducible.
4. Time zones and session boundaries are part of the data
An H4 candle beginning at one broker's midnight may contain different transactions from an H4 candle on a server using another timezone. A timezone offset states the relationship to a reference such as UTC. Daylight-saving changes can alter the offset in certain regions. A date such as 1 November does not identify a unique market session without an hour and timezone. If one file uses UTC and another uses Malaysia time, merging them by the displayed clock can misalign events by eight hours. That can move an economic release into the wrong bar. Always normalise timestamps for comparison, but retain the original source timezone and offset. Use separate checks for missing bars during a normal session and expected gaps during weekends or market closures. Otherwise you may count legitimate closures as data errors or accidentally treat nonexistent overnight candles as tradable.
5. Contracts and spreads can change under one chart name
A symbol called XAUUSD or EURUSD may keep the same display name while its broker specification, spread, contract unit or execution rules change. Record the contract multiplier and account currency for each period studied. Point value means the cash change associated with a specified price movement at a specified position size. It must be verified, not borrowed from a different broker. A backtest built from midpoint OHLC cannot simply call its close an executable Ask buy. If you model a separate spread, explain whether the entry and exit quotes already include it. Double-counting costs makes results too pessimistic; omitting them makes results too attractive. Both are analysis errors. Where genuine tick data is unavailable, use conservative sensitivity tests and disclose what cannot be established from candle data alone, particularly stop-versus-target ordering inside one bar.
6. Deliver a data-quality gate before any strategy test
Create a one-page quality report with source identity, time span, expected bar count, actual bar count, duplicate timestamps, missing blocks, suspicious OHLC rows, timezone consistency and quote-side assumptions. Add a severity decision for each issue: usable, usable with disclosure, or excluded from the relevant test. Do not use one vague 'clean' label. For the fictional 97-of-100 bar example, record completeness 97% and identify the precise three intervals. Then inspect whether any strategy decision depends on those missing times. If it does, mark its outcome UNKNOWN rather than allowing a hypothetical target to win by default. Keep raw and cleaned files separate with version numbers and change notes. A professional research report should say not only what its model found, but also what the underlying records were incapable of proving.
Common mistake
A tester sees three missing H1 candles and fills them with prices that make the chart continuous. The invented lows might conceal stopped trades. Fix this by flagging the missing intervals and withholding any outcome that depends on them.
Plain-English glossary
- OHLC: open, high, low and close data for one price bar.
- Data quality: accuracy, completeness and fitness for a specific analysis.
- Timezone offset: difference between a local clock and a reference time.
- Point value: cash movement per defined price change and position size.
- Cleaning log: record of every documented alteration to raw data.
Try it yourself
Audit 100 consecutive imaginary EURUSD H1 rows containing three missing bars, one duplicate timestamp and one invalid high. Produce a dated quality report, an unchanged raw copy and a change log. Mark any trade requiring missing price evidence UNKNOWN.
What you can now do
- Explain which evidence supports the written claim.
- Reproduce the loss or quality calculation with correct units.
- Deliver a dated, independently reviewable research record.
Next lesson
CP07.3 · Building Your Trading Manual. Turn research and risk constraints into an operating document.
