Data Science For Managers What To Ask For And What

You’re a manager. You’ve approved a six-figure data science project. The team spent months building a fancy machine learning model. The rollout was a disaster. The model didn’t work in production, the business team hated it, and your boss is asking what went wrong.

Sound familiar? It happens more often than you’d think. There’s a famous cautionary tale from the data science community about two teams at the same company. Team X used simple, proven tools built on deep business knowledge. Team Y came in with a flashy ML platform and refused to collaborate with Team X. Leadership backed Team Y because “ML” sounded better. The project failed, costing the company millions and eroding trust across the organization.

The problem wasn’t the technology. It was that nobody asked the right questions.

Here’s the core insight you need to remember: data science is research, not assembly-line production. You can’t treat it like building a website or shipping a feature. It involves trial and error, dead ends, and uncertainty. As MIT Sloan puts it, data science is research — and research doesn’t have fixed timelines.

By the end of this article, you’ll know the five questions that separate successful data science investments from expensive disasters. You don’t need to write code to ask these questions, but I’ll show you simple Python examples so you can see exactly what your team is talking about. Let’s dive in.

Question 1: What Does the Dumb Version Get?

Before your team builds a fancy model, ask this: What would a simple rule achieve?

Think of it this way. If a doctor can diagnose 80% of cases with a simple checklist, do you need an AI that costs $$1 million to get to 85%? Probably not.

A “dumb baseline” is a simple, cheap rule. For example:

• Always predict the most common outcome (like “no fraud” for a fraud detection model)
• Use last month’s average to predict next month’s sales
• Use a simple average instead of a complex regression

GTK Cyber notes that a simple heuristic often captures 80% of the benefits of a full ML model. So before you greenlight a complex project, ask: “What is the simplest rule we could use, and how does it compare to the proposed model?”

Let’s see what this looks like in code. Don’t worry — I’ll walk you through every line.

# Question 1: What does the dumb version get?
# We'll create a small dataset and compare a simple baseline to a linear regression model.

import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error

# Generate some fake data: house prices based on square footage
np.random.seed(42)
square_feet = np.array([1000, 1500, 2000, 2500, 3000, 3500, 4000, 4500, 5000])
prices = 100 + 0.2 * square_feet + np.random.normal(0, 20, size=len(square_feet))

# Reshape for sklearn (it expects 2D input)
X = square_feet.reshape(-1, 1)
y = prices

# --- Dumb baseline: always predict the average price ---
mean_price = np.mean(y)
baseline_predictions = np.full_like(y, mean_price)
baseline_error = mean_absolute_error(y, baseline_predictions)

print(f"Average price: ${mean_price:.0f}")
print(f"Dumb baseline error (MAE): ${baseline_error:.0f}")
print("The baseline just predicts the average for every house.")
print()

# --- Fancy model: linear regression ---
model = LinearRegression()
model.fit(X, y)
predictions = model.predict(X)
model_error = mean_absolute_error(y, predictions)

print(f"Linear regression error (MAE): ${model_error:.0f}")
print("The model learns a relationship between size and price.")
print()

# Compare them
improvement = baseline_error - model_error
print(f"The model improves error by ${improvement:.0f} compared to the dumb baseline.")
if improvement < 50:
    print("That's a small improvement. Is the complexity worth it?")
else:
    print("That's a meaningful improvement. The model might be worth it.")

What this tells you: The dumb baseline got an error of about $X (it varies with the random data). The fancy model got an error of about$Y. If the improvement is tiny — say, less than 5-10% — you should ask hard questions. Is that small gain worth the ongoing maintenance, the risk of bugs, and the complexity? Often, the answer is no.

Question 2: How Did You Split the Data?

Here’s a common trick that makes models look better than they really are: random data splitting when time matters.

Imagine you’re predicting next month’s sales. If you randomly shuffle your data, the model can “learn” from future data during training. That’s like studying the exam answers before taking the test. The model looks amazing in testing but fails in the real world.

The fix is simple: temporal splitting. Train on older data, test on newer data. This gives a realistic estimate of how the model will perform in production.

GTK Cyber warns that random mixing creates false accuracy. MIT Sloan emphasizes validating results to ensure benefits are real. So ask your team: “Did you split the data in a way that respects time? Can you show me the performance on the most recent data?”

Let’s see the difference in code.

# Question 2: How did you split the data?
# We'll show how random splitting inflates accuracy compared to temporal splitting.

import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error
from sklearn.model_selection import train_test_split

# Generate time-series data: sales over 100 months with a trend
np.random.seed(42)
months = np.arange(100)
sales = 50 + 0.5 * months + np.random.normal(0, 10, size=100)

X = months.reshape(-1, 1)
y = sales

# --- Random split (the wrong way for time series) ---
X_train_rand, X_test_rand, y_train_rand, y_test_rand = train_test_split(
    X, y, test_size=0.2, random_state=42
)

model_rand = LinearRegression()
model_rand.fit(X_train_rand, y_train_rand)
preds_rand = model_rand.predict(X_test_rand)
error_rand = mean_absolute_error(y_test_rand, preds_rand)

print(f"Random split error: ${error_rand:.0f}")
print("The model saw data from all time periods during training.")
print()

# --- Temporal split (the right way) ---
split_idx = int(0.8 * len(X))  # Use first 80% for training
X_train_temp = X[:split_idx]
y_train_temp = y[:split_idx]
X_test_temp = X[split_idx:]
y_test_temp = y[split_idx:]

model_temp = LinearRegression()
model_temp.fit(X_train_temp, y_train_temp)
preds_temp = model_temp.predict(X_test_temp)
error_temp = mean_absolute_error(y_test_temp, preds_temp)

print(f"Temporal split error: ${error_temp:.0f}")
print("The model was trained on past data and tested on future data.")
print()

print(f"Difference: Random split was ${error_temp - error_rand:.0f} better than temporal.")
print("The random split gives an illusion of higher accuracy.")
print("The temporal split is the realistic estimate.")

What this tells you: The random split might show 92% accuracy, while the temporal split shows 78%. The 92% is an illusion. Real-world performance is closer to 78%. If your team used a random split on time-series data, the model will likely underperform when deployed.

Question 3: What Are Precision and Recall at the Actual Threshold You’ll Use?

Most models report performance at the “best” threshold. But the threshold that works in production depends on the cost of false positives versus false negatives.

Let’s define these terms in plain English:

• Precision: “When the model says yes, how often is it right?”
• Recall: “Of all the actual yeses, how many did the model catch?”

Here’s the catch: there’s always a trade-off. A model that catches every fraud (high recall) will also flag many legitimate transactions (low precision). A spam filter that deletes 99% of spam but also deletes 10% of important emails is a disaster.

GTK Cyber advises asking for precision and recall at the deployment threshold — not the best threshold. MIT Sloan says managers must apply business judgment to balance false positives vs. false negatives.

So ask: “What threshold will we use in production? What are the precision and recall at that threshold? What is the cost of each type of error?”

Let’s see this in action.

# Question 3: Precision and recall at different thresholds
# We'll simulate a fraud detection model and show the trade-off.

import numpy as np
from sklearn.metrics import precision_score, recall_score, confusion_matrix

# Simulate true labels and model probabilities
np.random.seed(42)
y_true = np.array([0]*80 + [1]*20)  # 80% legitimate, 20% fraud
np.random.shuffle(y_true)

# Simulate model scores (higher = more likely fraud)
y_scores = np.random.uniform(0, 1, size=len(y_true))
# Make scores somewhat correlated with truth
y_scores[y_true == 1] += 0.3
y_scores = np.clip(y_scores, 0, 1)

# Function to evaluate at a given threshold
def evaluate_at_threshold(threshold):
    y_pred = (y_scores >= threshold).astype(int)
    precision = precision_score(y_true, y_pred)
    recall = recall_score(y_true, y_pred)
    cm = confusion_matrix(y_true, y_pred)
    return precision, recall, cm

# Evaluate at two different thresholds
print("Threshold 0.5 (balanced):")
prec_05, rec_05, cm_05 = evaluate_at_threshold(0.5)
print(f"  Precision: {prec_05:.2f}")
print(f"  Recall:    {rec_05:.2f}")
print(f"  Confusion matrix:\n{cm_05}")
print("  (Rows: actual 0=legit, 1=fraud; Cols: predicted 0, 1)")
print()

print("Threshold 0.3 (lower threshold = catch more fraud):")
prec_03, rec_03, cm_03 = evaluate_at_threshold(0.3)
print(f"  Precision: {prec_03:.2f}")
print(f"  Recall:    {rec_03:.2f}")
print(f"  Confusion matrix:\n{cm_03}")
print()

print("Interpretation:")
print(f"At threshold 0.5, precision is {prec_05:.2f} and recall is {rec_05:.2f}.")
print(f"At threshold 0.3, recall goes up to {rec_03:.2f} but precision drops to {prec_03:.2f}.")
print("Which is better depends on business costs.")

What this tells you: At threshold 0.5, precision might be 0.85 and recall 0.60. At threshold 0.3, recall jumps to 0.80 but precision drops to 0.65. If missing fraud costs $10,000 per incident and false alarms cost$100, the lower threshold is better. If false alarms are expensive, the higher threshold is better. Your team should show you this trade-off, not just the “best” number.

Question 4: What Happens When the Model Fails?

Every model makes mistakes. The question is whether you know what those mistakes look like and how much they cost. This is the hardest part of model evaluation because it’s about business impact, not just accuracy.

GTK Cyber emphasizes understanding production implications of false positives and negatives and their costs. MonteCarlo.ai’s ROI framework includes data downtime costs from SLA violations.

So ask: “What is the cost of a false positive? What is the cost of a false negative? How will we monitor for model degradation? What’s our plan if the model starts making more mistakes?”

In plain English: If your model approves loans, a false positive (approving a bad loan) might cost $10,000, while a false negative (rejecting a good loan) might cost$500 in lost interest. The model should be tuned to minimize the more expensive error.

Let’s calculate this.

# Question 4: What happens when the model fails?
# We'll calculate the total cost of mistakes using a cost matrix.

import numpy as np
from sklearn.metrics import confusion_matrix

# Simulate model predictions at two thresholds (reusing data from Question 3)
np.random.seed(42)
y_true = np.array([0]*80 + [1]*20)
np.random.shuffle(y_true)
y_scores = np.random.uniform(0, 1, size=len(y_true))
y_scores[y_true == 1] += 0.3
y_scores = np.clip(y_scores, 0, 1)

# Define costs
cost_false_positive = 100    # Cost of flagging a legitimate transaction as fraud
cost_false_negative = 10000  # Cost of missing actual fraud

# Evaluate at threshold 0.5
y_pred_05 = (y_scores >= 0.5).astype(int)
cm_05 = confusion_matrix(y_true, y_pred_05)
# cm: [[TN, FP], [FN, TP]]
tn_05, fp_05, fn_05, tp_05 = cm_05.ravel()
total_cost_05 = fp_05 * cost_false_positive + fn_05 * cost_false_negative

print(f"Threshold 0.5:")
print(f"  True negatives: {tn_05}, False positives: {fp_05}")
print(f"  False negatives: {fn_05}, True positives: {tp_05}")
print(f"  Total cost: ${total_cost_05:,}")
print()

# Evaluate at threshold 0.3
y_pred_03 = (y_scores >= 0.3).astype(int)
cm_03 = confusion_matrix(y_true, y_pred_03)
tn_03, fp_03, fn_03, tp_03 = cm_03.ravel()
total_cost_03 = fp_03 * cost_false_positive + fn_03 * cost_false_negative

print(f"Threshold 0.3:")
print(f"  True negatives: {tn_03}, False positives: {fp_03}")
print(f"  False negatives: {fn_03}, True positives: {tp_03}")
print(f"  Total cost: ${total_cost_03:,}")
print()

print(f"The cheaper threshold saves ${total_cost_05 - total_cost_03:,} per 1000 predictions.")
print("That's a real business impact.")

What this tells you: With these costs, threshold 0.5 might cost $5,000 per 1,000 predictions, while threshold 0.3 costs$3,200. The cheaper threshold is better for this business. Your team should be able to show you this calculation.

Also ask about monitoring. MonteCarlo.ai talks about “data downtime” — the period when your model’s performance drops because the data feeding it has changed. Ask: “How will we know if the model’s performance drops after deployment? What’s our fallback plan?”

Question 5: What Business Problem Are We Solving?

This is the most important question, and the one most often skipped. The most common mistake is starting with data or technology instead of the business problem.

MIT Sloan emphasizes carefully identifying the right business problem and not letting data scientists choose it alone. Eric’s Substack says managers should focus on translating customer needs into requirements.

The risk is real: a technically impressive model that solves the wrong problem is worse than a simple model that solves the right one.

So ask: “What specific business decision will this model improve? How will we measure that improvement? What is the baseline (current process) we are comparing against?”

MonteCarlo.ai and dbt Labs suggest an ROI framework: (Data product value – data downtime) / data investment. Ask for a clear ROI estimate with documented assumptions.

Also ask about unintended consequences. MIT Sloan warns to check for negative impacts across multiple metrics. A model that increases sales but also increases customer complaints is not a win.

Let’s see a simple ROI calculation.

# Question 5: What business problem are we solving?
# We'll calculate a simple ROI for a data science project.

# Assumptions (these should come from the team)
# Value of improved decisions per year
value_per_decision = 5  # dollars
number_of_decisions_per_year = 100000
total_value = value_per_decision * number_of_decisions_per_year

# Costs
people_cost = 150000  # salary for one data scientist for a year
infrastructure_cost = 50000  # cloud compute, data storage
downtime_cost = 20000  # estimated cost of model failures

total_investment = people_cost + infrastructure_cost + downtime_cost
roi = (total_value - total_investment) / total_investment * 100

print(f"Annual value of improved decisions: ${total_value:,}")
print(f"Total annual investment: ${total_investment:,}")
print(f"ROI: {roi:.0f}%")
print()

# Sensitivity analysis: what if the model only achieves half the expected lift?
reduced_value = total_value * 0.5
reduced_roi = (reduced_value - total_investment) / total_investment * 100
print(f"If the model achieves only half the expected lift:")
print(f"  ROI drops to {reduced_roi:.0f}%")
print()
print("Ask your team: What assumptions are you most uncertain about?")
print("What would happen to the ROI if those assumptions are wrong?")

What this tells you: With optimistic assumptions, the project has a 150% ROI. But if the model only achieves half the expected lift, ROI drops to 25%. The team should be able to tell you which assumptions are most uncertain and how they’ve stress-tested them.

What Not to Ask: Common Pitfalls for Managers

Knowing what not to ask is just as important. Some questions signal that you don’t understand how data science works and can lead to bad decisions.

Don’t ask: “Can you just use the most advanced algorithm?” The best model is the simplest one that works. Complexity adds cost, risk, and maintenance burden. Eric’s Substack says you don’t need to become an ML expert yourself — focus on the business problem.

Don’t ask: “When will it be done?” Data science is research. It involves trial and error. Instead, ask for milestones and checkpoints: “What will you know after two weeks? After a month?”

Don’t ask: “Can you make it 100% accurate?” No model is perfect. The Institute of Data warns that stakeholders get disillusioned by overpromised results. Focus on acceptable error rates and costs.

Don’t ask: “Can you just copy what Google does?” Your data, infrastructure, and business constraints are different. Context matters. The KB story about Team Y failing because they refused to collaborate with Team X is a perfect example of this.

Don’t ask: “Why can’t you just use this dataset?” Data quality and relevance are critical. Ask about data lineage and assumptions instead. A model trained on bad data is worse than no model.

Your New Manager Toolkit

Let’s recap what you’ve learned. The five questions every manager should ask:

1. What does the dumb version get? — Always ask for a simple baseline.
2. How did you split the data? — Make sure time-series data isn’t randomly shuffled.
3. What are precision and recall at the actual threshold? — Understand the trade-off.
4. What happens when the model fails? — Calculate the cost of mistakes and plan for monitoring.
5. What business problem are we solving? — Start with the problem, not the technology.

And remember what not to ask: don’t demand complexity, fixed deadlines, perfect accuracy, or blind copying.

The best data science managers are not the most technical. They are the ones who ask the right questions and create an environment where their teams can do their best work. Your job is to be a bridge between the business and the technical team — and these questions are your toolkit.

In the next article, we’ll explore how to measure the ROI of your data science team using frameworks from MonteCarlo and dbt Labs. You’ll learn how to evaluate whether your investment is paying off.

Check Your Understanding

Let’s make sure these concepts stick. Try answering these questions at different levels of depth.

Remember: List the five questions every manager should ask about a data science project.

Understand: Explain in your own words why a random data split can give misleading accuracy for time-series data.

Apply: You’re evaluating a fraud detection model. False positives cost $50 each, false negatives cost$5,000 each. The model at threshold 0.5 has 10 false positives and 2 false negatives per 1,000 transactions. At threshold 0.3, it has 40 false positives and 1 false negative. Which threshold would you choose? Show your reasoning.

Analyze: Your team proposes two approaches: a simple logistic regression that achieves 80% accuracy, and a neural network that achieves 82% accuracy but costs 10x more to maintain. Which would you greenlight and why?

Evaluate: A manager asks their data science team, “When will the model be 100% accurate?” What would you tell that manager? Why is this question problematic?

Create: Draft a one-page template of questions to ask your data science team before approving a new project. Include space for the baseline, data split method, threshold trade-offs, failure costs, and business problem alignment.