Mathematics and Data Science

Mean, median, mode, variance and standard deviation

Five numbers that describe a column. Quoting the wrong one is the most common way a report misleads people.

• Mean is the average and is pulled badly by outliers.
• Median is the middle value – the honest one for response times and salaries.
• Mode is the most frequent value, useful for categories.
• Variance / standard deviation measure spread. A small average with a huge spread means an unreliable system.
• Percentiles beat all of them for performance work: p95 and p99 say what the unlucky user got.

import statistics as st

response_ms = [180, 190, 200, 210, 190, 4800]   # one slow call

print("mean  ", round(st.mean(response_ms), 1))     # 628.3  <- dragged by the outlier
print("median", st.median(response_ms))             # 195.0  <- what users mostly feel
print("mode  ", st.mode(response_ms))               # 190
print("stdev ", round(st.stdev(response_ms), 1))    # 1875.6 <- huge spread = unstable

Probability and statistical distributions

Models output probabilities. You need to read them without over-trusting them.

• A probability is between 0 and 1. “0.87 fraud” does not mean fraud – it means the model is fairly confident, given what it learned.
• The normal distribution (bell curve) describes many natural measurements; most values sit near the middle.
• Real software data is often skewed instead – response times and order values have a long tail to the right.
• Base rate matters. If 1 in 1000 transactions is fraud, a test that is “99% accurate” can still be wrong far more often than it is right.

Linear algebra basics: vectors and matrices

You will not do the maths by hand, but the words appear in every error message and every API response.

• A vector is a list of numbers describing one item. An embedding in Module 7 is exactly this, just with 1,536 numbers instead of 3.
• A matrix is a table of numbers – a batch of vectors stacked up.
• Shape is rows x columns. “Shape mismatch” is the most common error in ML code and it always means two things lined up wrongly.
• Cosine similarity measures the angle between two vectors: 1 is identical, 0 unrelated. This is how semantic search decides what is relevant.

import numpy as np

a = np.array([0.2, 0.9, 0.1])     # a vector - 3 numbers describing one thing
b = np.array([0.25, 0.85, 0.15])

m = np.array([[1, 2], [3, 4]])    # a matrix - rows and columns
print(a.shape, m.shape)           # (3,) (2, 2)

cos = a @ b / (np.linalg.norm(a) * np.linalg.norm(b))   # cosine similarity
print("similarity", round(float(cos), 4))               # ~0.998 -> nearly the same

Correlation and feature relationships

Which columns move together – and the trap that follows.

• Correlation runs from -1 to +1: +1 moves together, -1 moves oppositely, 0 no linear relationship.
• df.corr() gives you the whole grid in one line.
• Correlation is not causation. Ice-cream sales correlate with drownings; both follow summer.
• Two highly correlated inputs add cost without adding information – usually one of them is dropped.
• A feature that correlates suspiciously well with the answer is often leakage: it would not exist at prediction time.

Data cleaning, missing values and outliers

The least glamorous part of AI and the one that decides whether anything works.

• Decide per column: drop the row, fill with a sensible value, or keep the blank as information in itself.
• Filling with the mean hides outliers; the median is usually safer.
• Text columns need trimming and case-normalising before anything is grouped or joined.
• An outlier is not automatically wrong – a 20 lakh claim may be genuine. Flag, do not delete.

import pandas as pd, numpy as np

df = pd.read_csv("claims.csv")

print(df.isna().sum())                       # how many blanks per column
df = df.drop_duplicates(subset="claim_id")   # same claim twice
df["amount"] = pd.to_numeric(df["amount"], errors="coerce")   # "N/A" -> NaN
df["city"] = df["city"].str.strip().str.title()               # "  chennai " -> "Chennai"
df["age"] = df["age"].fillna(df["age"].median())              # fill, do not drop the row

q1, q3 = df["amount"].quantile([0.25, 0.75])
iqr = q3 - q1
outliers = df[(df["amount"] < q1 - 1.5 * iqr) | (df["amount"] > q3 + 1.5 * iqr)]
print(len(outliers), "amounts worth a human look")

Exploratory data analysis and visualisation

Look at the data before you model it. Ten minutes here saves a week later.

• Start with the shape, the column types and the number of blanks.
• One chart per question: a histogram for distribution, a bar chart for categories, a scatter for two numeric columns, a line for anything over time.
• Check the balance of your target column. 95% “not fraud” changes everything about how you train and measure.
• Plot the same chart split by month or by region – a national average often hides one broken region.

Training, validation and test datasets

The single most important rule in machine learning: never score a model on data it has seen.

• Training set teaches the model, validation set tunes it, test set is touched once, at the end.
• stratify=y keeps the rare class present in every split – essential for fraud and defect data.
• random_state makes the split reproducible, so your results can be re-run.
• For anything time-based, split by date, not randomly: training on next month to predict last month is leakage.

from sklearn.model_selection import train_test_split

X = df[["age", "cover", "city_code"]]
y = df["claimed"]

X_train, X_temp, y_train, y_temp = train_test_split(X, y, test_size=0.3, random_state=42, stratify=y)
X_val, X_test, y_val, y_test = train_test_split(X_temp, y_temp, test_size=0.5, random_state=42, stratify=y_temp)

print(len(X_train), len(X_val), len(X_test))   # roughly 70 / 15 / 15

Project: analyse sales data and identify trends

1. Take any transactional export – orders, claims, tickets – with a date column and an amount.
2. Clean it: duplicates, blanks, text case, numeric types. Write down every change you made.
3. Compute mean, median and p95 of the amount, then the same split by month.
4. Plot the monthly trend and write three sentences: what changed, when, and what you would ask the business.
5. Last step: list the columns that would not exist at prediction time. That list is your leakage check for Module 4.