Customer Segmentation & Anomaly Detection
Finding the groups, and the customers who belong to none of them
Statistics graduation project · Team of 7 · 2025–2026
The question
Given a retail customer dataset — demographics and spending behaviour — can you find natural groupings without deciding in advance how many exist, and can you separate genuine outliers from ordinary variation?
The second half is the part most segmentation work skips. K-Means assigns every point to a cluster, including the points that don't belong to any of them.
The data
Raw customer records with missing values, inconsistent entries and outliers. Cleaning came before anything else: handling the missing values, resolving the inconsistencies, and standardising the features so that distance-based clustering wasn't dominated by whichever variable happened to have the largest scale.
Why DBSCAN
Density-based clustering doesn't require choosing k in advance, finds clusters that aren't spherical, and — the reason it mattered here — labels points belonging to no cluster as noise rather than forcing them into the nearest one. For a question that is half anomaly detection, that property is the whole argument.
Choosing the parameters
MinPts came from the standard rule — at least 2d + 1, so at least 5 for two features. I used 9, deliberately conservative. For eps: each point's distance to its 9th nearest neighbour, sorted and plotted, with the elbow at ≈ 0.4 in standardised space. Then a 16-combination grid search around it.
Every combination scored on three things at once — Silhouette, cluster count, and noise ratio — because a parameter set can score beautifully on Silhouette while labelling most of the dataset as noise. That is a good number attached to a useless model.
| eps | Noise | Silhouette | Reading |
|---|---|---|---|
| 0.3 | 15.5% | 0.4892 | over-segmented |
| 0.4 | 22.5% | 0.5849 | chosen — maximum Silhouette |
| 0.5 | 1.5% | 0.5312 | merges distinct segments |
| 0.6 | 0.0% | 0.4788 | too permissive |
The relationship is not monotonic — Silhouette peaks at 0.4 and falls off on both sides. Loosening eps past 0.5 drops the noise almost to zero, which looks tidier and is worse: it merges segments that are genuinely distinct.
Benchmark
DBSCAN performing well proves nothing on its own. I ran K-Means and Hierarchical (Ward) clustering over the same prepared data and compared all three across three validity indices, with PCA projections into two dimensions for visual inspection.
Three indices rather than one, because they disagree by design. Silhouette rewards separation. Davies–Bouldin penalises within-cluster spread relative to between-cluster distance. Calinski–Harabasz favours dense, well-separated clusters. A result that holds across all three is a result. A result that holds on one is a coincidence.
Silhouette
↑ higher is better
Davies–Bouldin
↓ lower is better
Calinski–Harabasz
↑ higher is better
4
clusters found by DBSCAN
5
by K-Means and hierarchical
45
points labelled noise
77.5%
coverage
The three methods do not agree on the number of segments. K-Means and hierarchical both settle on five, because both are obliged to place every customer somewhere. DBSCAN finds four and sets 45 customers aside as belonging to none of them. Which of those is the right answer depends entirely on whether you need full coverage or honest outliers.
Why the winner didn't really win
DBSCAN takes two of the three indices. That result is misleading, and it is worth saying so.
DBSCAN computes its scores on the 155 points it clustered, having excluded 45 as noise. K-Means and hierarchical compute theirs on all 200. The excluded points are precisely the hardest customers to cluster — they would depress any algorithm's Silhouette. That is a structural advantage in the measurement, not evidence of better clusters.
The tell is Calinski–Harabasz. It is computed on all 200 points for every algorithm, with no exclusion available — and there K-Means wins. All three methods converge on the same underlying segment structure.
From noise points to something a business can use
The output that mattered was not the clustering diagram.
Every point DBSCAN labelled as noise was classified into a category someone could act on: high income with low spending, low income with high spending, or genuinely isolated behaviour. Each of those is a different commercial situation and a different response.
A statistical result nobody can act on isn't finished.
Limitations
The dataset is a standard mall-customer set, not proprietary data — the method is real, the business context is illustrative.
And the segmentation is descriptive. It says who the groups are. It doesn't say what happens next, or what to do about them.
My role
I owned the DBSCAN clustering and anomaly-detection module and the outlier report. I also reviewed the data cleaning, EDA, K-Means and hierarchical clustering modules written by other members of the team.