Data Quality and Bias

The Top 10 That Forgot the Other 30

The terrace was cooler than the living room, so that's where Rohan and his uncle Vikram ended up after dinner, phones out, chairs turned toward each other.

"Look at this," Rohan said, angling his screen. "Top 10 Funds of the Last 10 Years. This one turned one lakh into almost six lakh. Six lakh, Vikram uncle. In ten years."

Vikram leaned in, adjusted his glasses, and read the list slowly, the way he read everything since retiring from the bank. "Nice list," he said. "Very nice list."

"I'm thinking of putting my Diwali bonus into the top one," Rohan said. "Number one on the list, ten years straight, why would I look anywhere else?"

Vikram didn't answer right away. He was scrolling on his own phone now, squinting. "Tell me something," he finally said. "This category — small-cap, right? How many funds were there in this category ten years ago, when it started?"

Rohan shrugged. "This shows ten. So, ten?"

"This shows the ten that are still standing today," Vikram said. "It doesn't show you who was standing at the starting line."

Rohan laughed it off at first. "Uncle, it's a ranking, not a race report."

"Humour me," Vikram said, and passed his phone across. "Go to the fund house's own website. Not the ranking app. Look for 'schemes merged' or 'schemes wound up.'"

Rohan typed, half-annoyed, mostly to prove his uncle wrong. The page loaded slower than he wanted it to.

He went quiet.

"What," Vikram said, not really a question.

"There's a list here," Rohan said slowly. "Funds that got merged into other funds. Funds that just... closed." He scrolled. "This is a lot, uncle."

"How many?"

Rohan counted, mouthing numbers under his breath, then looked up. "If I add these to the ten on the ranking... there were way more than ten funds in this category to begin with."

Vikram smiled, but not the smug kind. The kind that comes from having watched this exact scene happen many times, over many decades, with many nephews. "So when you look at that shiny list tomorrow morning," he said, "who's on it?"

Rohan understood before he said it out loud. "Only the ones that survived."

"And the ones that shut down along the way," Vikram said, "the ones that lost money, disappointed people, got merged quietly into some other fund's numbers — where do they show up on that ranking?"

"...Nowhere," Rohan said.

He sat back in his chair, phone dimmed in his lap, the number six lakh suddenly looking less like a fact and more like a photograph someone had cropped very, very carefully.

"So the average return everyone's cheering for," he said, thinking out loud now, "isn't really the average of everyone who tried. It's the average of everyone who's left."

Vikram just nodded and reached for his chai, letting the boy sit with that for a moment on his own.

• Explain how survivorship bias makes a track record look better than the real historical average by quietly excluding failures and dropouts.
• Explain how lookahead bias makes a backtest look smarter than it honestly could have been by using information that wasn't available at the time.
• Apply a two-question check ('what failures might be missing?' and 'could this have used future information?') to decide whether to trust an impressive performance number.

Here's the problem with any "top performers" list: the moment you trust it without a second look, you're not judging a strategy — you're judging a photograph someone already cropped for you.

Two specific tricks do this cropping.

The first is survivorship bias. It happens when the failures — the mutual funds that shut down, the stocks that got delisted, the strategies that quietly stopped being talked about — are removed from the dataset before anyone calculates the "average return." So when you see "Top 10 Funds of the Decade," you're only looking at the funds that survived the decade. The ones that lost money and closed never make it onto that page. The average return looks fantastic because it was never really an average of everyone who tried — it's an average of everyone who's left standing.

The fix isn't a formula. It's a pause, and two specific questions.

The second is lookahead bias. It happens when a test secretly uses information that wasn't actually available at the time. Imagine a video claiming "this strategy would have beaten the Nifty 50 every year for a decade" — but the test picks today's list of Nifty 50 companies and applies it backward in time. The problem: several of today's top companies weren't even part of the index years ago. The test is quietly using knowledge from the future to look smart about the past.

Both tricks change the question being answered. Instead of "what would have honestly happened to someone who invested at that time?" they answer the much easier question: "what looks good now that we already know how the story ended?"

The fix isn't a formula. It's a pause, and two specific questions.

Before trusting any impressive backtest or track record, ask: "What failures might be missing from this?" and "Could this have secretly used information from the future?" If you can't answer either one clearly, treat the number as a flattering guess dressed up as fact — not proof of anything. This doesn't mean every backtest is worthless. It means every one has to earn your trust by passing this two-question check first.

That night, after Vikram had gone home, Rohan couldn't stop poking at the numbers. He went back to the fund house's disclosure page and actually counted: 40 small-cap funds had launched in that category ten years ago. Today, only 22 were still around. The other 18 had been merged into other schemes or shut down entirely after underperforming — and none of them appeared anywhere on the "Top 10" list he'd shown Vikram on the terrace. The list wasn't lying about the 10 funds it showed. It was just never going to show him the other 30.

Curious now, he opened a YouTube video he'd bookmarked earlier — a creator claiming a stock-picking method "would have beaten the Nifty 50 every single year for the last decade." The chart looked airtight. But reading the video's description more carefully, Rohan noticed a line he'd skipped past before: the backtest applied today's list of Nifty 50 companies to prices going back ten years. He knew enough to catch the problem — several companies on today's Nifty 50 hadn't even been part of the index a decade ago. They'd grown into it since. The test wasn't picking stocks the way a real investor could have in 2016; it was reaching back with 2026's answer key already in hand.

Two claims, two different tricks, same result: a history that looked far better than it honestly was. The fund list had hidden its failures. The video had secretly known the future. Rohan realized that "does this look legit" wasn't a real question — the real questions were "what's missing?" and "did this know something it shouldn't have?"

Point: Survivorship bias and lookahead bias can both hide inside the same kind of impressive claim, and spotting them requires asking two separate, specific questions rather than one vague 'is this legit?' feeling.

• Assuming historical data behind a chart or ranking is automatically complete and honest just because it's made of real numbers. — Numbers and charts feel objective, so it seems like they must have been collected without gaps or shortcuts. Fix: Remind yourself that data can be filtered before you ever see it. Ask specifically what was included and excluded, not just whether the source seems trustworthy.
• Treating survivorship bias and lookahead bias as rare problems that only professional quants need to worry about. — The terms sound technical and usually show up in conversations about formal backtesting or fund analysis. Fix: Watch for both in everyday claims too — a 'top funds' list, a stock screener applied to past years, or a friend's success story that skips the failed attempts along the way.
• Believing a strategy's impressive backtest return is close to what a real investor following it would actually have earned. — The backtest is built from real historical prices, so it feels like a faithful replay of the past. Fix: Check whether the data excluded failures or used future information. If either is true, the real historical experience would have been worse than the backtest shows.

• A track record that quietly drops the failures or secretly knows the future will always look better than the honest past ever did.
• Survivorship bias hides failures: closed funds, delisted stocks, and dropped strategies vanish from the dataset before anyone averages the results.
• Lookahead bias hides time-travel: a test uses information that wasn't actually available at the moment the decision would have been made.
• Before trusting any impressive number, ask two specific questions: what failures might be missing, and could this have secretly used future information?
• A backtest can still be useful — but only after it passes the full-picture check, not before.

Q1. What is survivorship bias?

• When failures or dropouts (like closed funds or delisted stocks) are quietly excluded from a dataset, making the remaining average look better than it really was
• When a strategy is tested using only the most recent five years of data
• When an investor only buys stocks recommended by friends
• When a fund manager charges lower fees than competitors Answer: When failures or dropouts (like closed funds or delisted stocks) are quietly excluded from a dataset, making the remaining average look better than it really was — Survivorship bias happens when the losers - shut down funds, delisted stocks - never make it into the dataset, so only the winners get counted and the average looks artificially strong.

Q2. What is lookahead bias?

• When a backtest or claim uses information that wasn't actually available at the time the decision would have been made
• When an investor waits too long before selling a losing stock
• When a fund's fees increase every year without notice
• When a dataset includes too many failed companies Answer: When a backtest or claim uses information that wasn't actually available at the time the decision would have been made — Lookahead bias means the test secretly 'knows' the future - like using today's Nifty 50 list to test a strategy on prices from ten years ago, even though the index membership was different back then.

Q3. A mutual fund ranking website shows a 'Top 10 Funds of the Decade' list with excellent returns, but never mentions that 18 other funds in the same category shut down or merged during that time. Which bias does this illustrate?

• Survivorship bias
• Lookahead bias
• Both equally
• Neither, this is normal reporting Answer: Survivorship bias — This is survivorship bias - the failed and closed funds were quietly excluded from the list, so the average return you see only reflects the funds that survived, not everyone who actually tried.

Q4. A backtest is honest and complete as long as it uses real historical stock prices, even if it doesn't check which companies existed in an index at that time. Answer: False — Real historical prices alone aren't enough. If the test applies today's index membership backward in time, it secretly uses future information - that's lookahead bias, and it makes the result look better than it honestly could have been.

Q5. You see a YouTube video claiming a stock-picking strategy 'would have beaten the Nifty 50 every year for a decade' by testing it on the Nifty 50's current 50 companies applied backward to past prices. Before trusting this claim, what two questions should you ask? Answer: What failures might be missing from this data, and could this have secretly used information (like future index membership) that wasn't available at the time? — These two specific questions - checking for missing failures and checking for secretly-used future information - are the habit that separates a careful decision from being simply impressed by a number.

Q6. A friend shows a stock-picking method that "never lost money" over five years, based on a curated list of 8 stocks. Someone accepts it immediately, saying: "Five years of proof, that's solid enough." Is five years of a perfect track record on this list solid proof? Reveal: Weak: yes, five years without a loss is a strong, solid track record. Strong: the two-question check applies — were any losing stocks quietly dropped from the list (survivorship), and was the list built with hindsight about which stocks would do well (lookahead)? A perfect record on a curated list can hide both.