A practical example of how small data-quality checks can prevent misleading conclusions in financial market research.
Two market-data feeds can report different values for the same instrument without either feed necessarily being broken.
The difference might come from timestamps, trading sessions, missing observations, historical adjustments, or source-specific coverage.
Before comparing charts or drawing conclusions, we should establish whether the underlying observations are actually comparable.
In this tutorial, we'll build a small Python validation workflow to identify several common issues:
Missing or inconsistent timestamps
Duplicate market observations
Records outside a defined trading session
Invalid price values
Gaps in a one-minute data series
The goal is not to build a production market-data engine. It's to demonstrate how a few explicit validation rules can make research more reproducible.
- Start With the Observation, Not the Chart
Imagine receiving these observations for a hypothetical instrument called DEMO.
Timestamp (UTC)
Price
Source
13:29
101.20
A
13:30
101.25
A
13:30
101.25
A
13:31
-1.00
A
13:32
101.40
A
13:35
101.60
A
At first glance, we have six rows of market data.
But a quick review raises several questions.
Why does 13:30 appear twice? Is a negative price valid for this particular data type? Are all observations from the intended trading session? And what happened between 13:32 and 13:35?
A chart alone might not make these problems obvious.
Before visualization, we need context and validation.
- Define the Research Context
For this example, we'll use the regular trading-session hours commonly associated with US equity markets:
9:30 a.m. to 4:00 p.m. America/New_York time.
This is an illustrative session definition, not a universal market-data rule. It does not account for market holidays, early closures, auctions, or instrument-specific trading schedules.
Python's zoneinfo module helps us work with the appropriate time zone instead of assuming a permanent UTC offset.
from datetime import datetime, time, timezone
from zoneinfo import ZoneInfo
EASTERN = ZoneInfo("America/New_York")
def parse_utc(text):
dt = datetime.fromisoformat(
text.replace("Z", "+00:00")
)
if dt.tzinfo is None or dt.utcoffset() is None:
raise ValueError(
"Timestamp must include timezone"
)
return dt.astimezone(timezone.utc)
def session_label(dt):
local = dt.astimezone(EASTERN)
if local.weekday() >= 5:
return "outside_regular_session"
if time(9, 30) <= local.time() < time(16, 0):
return "regular_session"
return "outside_regular_session"
This gives us two useful properties.
First, timestamps must carry explicit timezone information.
Second, session classification takes the relevant local market time into account, including daylight-saving transitions.
For production systems, use an appropriate exchange calendar rather than relying on weekday and clock-time checks alone.
- Build a Small Sample Dataset
We'll use entirely hypothetical data.
records = [
{
"symbol": "DEMO",
"timestamp": "2026-10-22T13:29:00Z",
"price": 101.20,
"source": "A"
},
{
"symbol": "DEMO",
"timestamp": "2026-10-22T13:30:00Z",
"price": 101.25,
"source": "A"
},
{
"symbol": "DEMO",
"timestamp": "2026-10-22T13:30:00Z",
"price": 101.25,
"source": "A"
},
{
"symbol": "DEMO",
"timestamp": "2026-10-22T13:31:00Z",
"price": -1.00,
"source": "A"
},
{
"symbol": "DEMO",
"timestamp": "2026-10-22T13:32:00Z",
"price": 101.40,
"source": "A"
},
{
"symbol": "DEMO",
"timestamp": "2026-10-22T13:35:00Z",
"price": 101.60,
"source": "A"
}
]
Notice that every record includes a source identifier.
That matters because two providers can legitimately deliver observations at the same timestamp. A duplicate should be defined within the appropriate source, instrument, and data-series context rather than across all providers indiscriminately.
- Validate Each Observation
Let's check three conditions:
Whether the source and instrument already have an observation at the same timestamp.
Whether the price is positive, as required by our simplified sample data contract.
Whether the record belongs to the regular session we defined.
seen = set()
valid = []
for row in records:
dt = parse_utc(row["timestamp"])
key = (
row["source"],
row["symbol"],
dt
)
issues = []
if key in seen:
issues.append(
"duplicate timestamp for source/symbol"
)
seen.add(key)
if row["price"] <= 0:
issues.append("nonpositive price")
if session_label(dt) != "regular_session":
issues.append("outside regular session")
if issues:
print(
row["timestamp"],
"->",
"; ".join(issues)
)
else:
valid.append((dt, row["price"]))
Expected output:
2026-10-22T13:29:00Z -> outside regular session
2026-10-22T13:30:00Z -> duplicate timestamp for source/symbol
2026-10-22T13:31:00Z -> nonpositive price
These results don't necessarily mean every flagged observation should be deleted.
An extended-session observation may be perfectly valid for research that includes premarket activity.
Likewise, duplicate timestamps could indicate a data-ingestion problem, an update to an earlier observation, or a feed that allows multiple events at the same time.
Validation flags issues. The research context determines how to resolve them.
- Detect Missing Minute Observations
After excluding the flagged rows in this simplified example, we can inspect the remaining chronological sequence.
valid.sort()
for (prev_dt, _), (current_dt, _) in zip(
valid,
valid[1:]
):
gap = int(
(current_dt - prev_dt).total_seconds() // 60
)
if gap > 1:
print(
f"Gap: {gap - 1} missing minute(s) "
f"after {prev_dt.isoformat()}"
)
Expected output:
Gap: 1 missing minute(s) after 2026-10-22T13:30:00+00:00
Gap: 2 missing minute(s) after 2026-10-22T13:32:00+00:00
The first gap arises because the invalid 13:31 record was excluded.
The second reflects missing minute observations at 13:33 and 13:34.
However, a missing timestamp is not automatically proof of missing source data.
An event-driven trade feed may contain no trades during a quiet minute. A complete one-minute bar feed, by contrast, might be expected to provide a bar for every scheduled minute.
Before interpreting a gap, we need to know the data contract.
- Why Historical Data Needs More Than a Price Column
The example identifies structural problems, but several important questions remain.
Adjusted versus unadjusted prices
A historical series adjusted for stock splits or other corporate actions may differ significantly from an unadjusted series.
Timestamp semantics
Does the timestamp represent the beginning of an interval, the end of a bar, an individual market event, or the time the provider processed the record?
Trading-session coverage
Does the feed include only regular hours, or also premarket and after-hours observations?
Provider revisions
Can a data source update previously published observations? If so, how are versions tracked?
These details can affect comparisons even when all price values are technically valid.
A reproducible research workflow should preserve them alongside the numerical observations.
- Build a Reproducible Research Record
Instead of recording only a chart interpretation, save the context needed to investigate the result later.
For example:
research_record = {
"instrument": "DEMO",
"source": "A",
"data_type": "one_minute_bars",
"timezone": "UTC",
"session_timezone": "America/New_York",
"session_scope": "regular_session",
"adjustment_policy": "unadjusted",
"validation_rules": [
"timestamp_timezone_required",
"duplicate_timestamp_check",
"positive_price_check",
"session_classification",
"one_minute_gap_check"
],
"research_note": (
"Sample observations require validation "
"before chart-based comparison."
)
}
A production implementation should also preserve the instrument identifier, source version, retrieval time, expected market calendar, adjustment methodology, and the validation results themselves.
The essential principle is simple:
A research observation becomes more useful when another analyst can reconstruct the conditions under which it was made.
- Connecting Validation to Research Workflows
Market-data quality is not just an engineering concern. It directly affects analytical interpretation.
A practical sequence is:
Screen → Add context → Review the chart → Verify the source → Record the observation.
This approach also helps explain the research-oriented positioning described in public materials about Ositrustees.
Ositrustees is described as a financial data and analytical technology platform focused on market screeners, sector monitoring, chart-based research, and structured market-data workflows.
The Python example above is an independent educational demonstration of data-validation concepts. It is not Ositrustees production code, an API example, or evidence of the platform's internal implementation.
The wider lesson applies to any market-data research environment: validation and provenance should be treated as part of the research workflow, not as optional cleanup after the analysis.
Conclusion
Reliable market research starts before a chart is rendered.
A small validation routine can expose duplicate records, invalid prices, mismatched session boundaries, and missing observations.
But the most important step is deciding what each warning means in the context of the instrument, source, and research question.
When we preserve timestamps, source identity, session assumptions, adjustment rules, and validation results, the research becomes easier to review and reproduce.
Research. Compare. Verify.
Technical note: This tutorial uses hypothetical observations and the Python standard library. Its weekday/session logic is intentionally simplified; use an exchange calendar and instrument-specific data contracts for production applications.
Top comments (9)
The
seen.add(key)placement gives me one extra fixture: an invalid row arrives first, then a corrected row with the same source, symbol and timestamp. The correction would still be flagged as duplicate. Then keep validation failures and accepted observations distinct, and state whether a revision replaces an earlier value. This late-event counter exercise asks a related question that which facts must survive when arrival order changes?Good question. I'd keep the original observation immutable and treat the corrected row as a separate version rather than overwriting the earlier record.
For each revision, I'd preserve the source, symbol, original timestamp, normalized timestamp, ingestion time, and a revision identifier.
The correction should also retain a reference to the record it supersedes, so the system can reconstruct what was known at any given point.
That distinction matters for backtesting and historical analysis. The latest corrected dataset and the dataset available at the time of the original decision aren't necessarily the same.
I'd also separate the event time from the time the correction was received. Otherwise, late-arriving updates can quietly change historical results without leaving a clear audit trail.
Thanks for bringing this up - it's an important part of making market-data research reproducible.
The as-of distinction makes this much clearer. A small replay fixture could put a correction just after the simulated decision, then show what each dataset version returns. That would let the backtest prove which information was actually available.
That's a useful test case, Emily. Placing the correction just after the simulated decision would make the as-of boundary explicit. The key is to query both versions using the same decision timestamp and confirm that the earlier view excludes information received later. Keeping event time, ingestion time and revision history separate makes that behaviour reproducible.
Thanks for adding a practical way to validate the workflow.
Thanks, Tedor. I’d make the fixture assert both views: the original value at decision time, then the corrected value at a later cutoff. That way the test catches both future leakage and a correction that never becomes visible.
Section 6 lists timestamp semantics as an open question, but the session rule in section 2 already depends on it. With 9:30 <= t < 16:00, a bar stamped 16:00 is flagged as outside the session. That is correct for start-stamped bars, where the last regular bar is 15:59, but it would drop the last bar of every day for a source that stamps bars by their end, and a bar stamped 9:30 would then belong to premarket. I would store the stamping convention per source and add a completeness check that a full regular day has 390 one-minute bars, with early-close days (13:00 ET) taken from the exchange calendar rather than a fixed 16:00. In the gap check, do you treat a minute with no trades as a missing bar or as a zero-volume bar?
Good catch, especially on the bar timestamp convention.
You're right that the 9:30 <= t < 16:00 rule assumes start-stamped one-minute bars. With end-stamped bars, that boundary needs to change, or the timestamps should be normalized before session filtering.
I'd store the timestamp convention with each source's metadata and use an exchange calendar for session completeness checks, including early closes.
On missing minutes, I wouldn't automatically treat every gap as bad data. Some providers omit intervals with no trades, while others may include zero-volume bars. I'd check the provider's bar-generation rules and keep missing intervals separate from confirmed zero-volume observations.
Thanks for pointing this out. It's a useful clarification to make explicit in the example.
Keeping missing intervals separate from confirmed zero-volume bars sounds right. One cheap check for the timestamp convention: after normalizing, assert that a full regular day runs from 09:30 to 15:59 as start-stamped bars. If a source comes out as 09:31 to 16:00 instead, its stored convention is probably wrong, and you find out at ingestion rather than from a quietly shifted series. For the gaps, comparing the summed minute volume with the provider's daily volume can help tell sparse bars from lost ones, though the two can differ by feed, so I would use a tolerance and not expect an exact match.
That's a good point about keeping missing intervals separate from confirmed zero-volume bars.
I agree that normalizing timestamps before applying the session filter is essential. Otherwise, the completeness check can end up flagging valid data as missing simply because the provider uses a different bar-labeling convention.
Your suggestion to compare the summed minute volume with the provider's daily volume is useful as a secondary consistency check. I'd also allow for differences in session coverage, late corrections, and the provider's aggregation rules rather than expecting an exact match.
The key distinction is between a missing observation, a confirmed zero-volume interval, and an interval excluded by the session definition. Treating those as separate states makes the analysis much easier to audit.
Thanks for expanding on the example.