Machine learning: predicting employee turnover

What it is

A model predicting voluntary departures, built on a company’s HR data, from five sources that had to be reconciled.

What was hard

One employee in six leaves, and missing a leaver costs far more than a false alarm. The whole difficulty sits in that trade-off.

What it proves

A second pipeline locks every step so the test set never takes part in its own preparation, and both versions are compared over the same data.

A team project, four people, within the machine learning block of the engineering programme. The case is a fictional pharmaceutical company trying to understand why its employees leave.

One employee in six walks out

One employee in six leaves every year, and that 16.12%productionObserved attrition rate over the year: 16.12 % (production). departure rate is expensive: recruitment, training, projects delayed while a replacement is found. The business need is twofold, predict who is at risk of leaving, and say which levers to pull.

Those are two different questions, and one model does not satisfy both: a model that predicts well but cannot be read tells nobody what to do on Monday morning.

A rare class, asymmetric errors

Five sources to reconcile. General employee data, two satisfaction surveys, and clock-in and clock-out records over a full year, to be merged on the employee identifier.

A one-to-five imbalance. The class of interest is the rarer one. A model predicting “nobody leaves” would already look excellent and would be of no use whatsoever.

Errors do not cost the same. Missing a leaver means losing the person. Raising a false alarm means one interview too many. Evaluation therefore has to look at recall before accuracy.

Five sources, seven algorithms

From five sources to the evaluation protocol

The protocol matters as much as the model

1. Five sources

   general data, two surveys, clock-in and clock-out records

2. Merge and exploration

   join on the identifier, statistical test per variable

3. Seven temporal variables

   hours worked, schedule gaps, early departures

4. Twenty-eight training runs

   seven algorithms, with and without rebalancing, under two protocols

5. Protocol grouped by individual

   no duplicate crosses the boundary

   Naive protocol, by rows

   the dataset replicates each profile: the model recognises individuals it has already seen

Merging and exploration. The five sources are joined on the identifier, then each candidate variable is tested against the target, chi-squared for categorical variables, Student’s t-test for continuous ones. This is not decorative: it is what discards unrelated variables before anything is run.

Seven variables built from the clock records. Clock-in and clock-out times taken day after day over a full year are useless as they stand. They become hours worked, gaps against the nominal schedule, counts of early departures and absences. That is the part that takes the most work, and it produces the most discriminating variables.

Twenty-eight training runs compared. Seven algorithms, from logistic regression to forests and gradient boosting, each with and without rebalancing of the minority class, under two different splitting protocols. Then 216combinationsvalidation setHyperparameter combinations explored: 216 combinations (validation set). for the selected model, each scored by five-fold cross-validation.

constraint met

A leak-proof protocol held end to end: the split precedes every transformation, imputation and rebalancing are fitted on the training set alone, and the test set is left at its natural distribution.

What I settled, and at what cost

Rebalance, and pay the advertised price. Oversampling the minority class catches 31.6points moretest setOut of a hundred employees about to leave, rebalancing catches thirty-two more of them. In exchange, more false alarms. leavers per hundred on average across the seven algorithms, and costs 10.1points lesstest setThe price paid: among the employees flagged as at risk, ten more out of a hundred are flagged wrongly. of accuracy on the alerts. On this problem the trade is worth making, because a missed leaver costs more than a pointless interview, but it is measured rather than assumed.

Two splitting protocols kept side by side rather than one quietly corrected. The first fits transformations before splitting, which lets the test set influence imputation; the second splits first. Keeping both makes it possible to quantify what the precaution changes, instead of asserting that it matters.

The notebook and the documentation do not settle on the same model, and I flag it rather than picking whichever suits me. The notebook concludes in favour of gradient boosting, top of the scoreboard. The documentation recommends the random forest, because it yields feature importances an HR team can read.

The trade-off between them is real: a model that wins a point and that nobody can interpret answers only one of the two questions asked. But the two documents in the deliverable settle it differently without saying so, and nobody noticed before I reread them.

reproducible artefact

A complete replayable chain: five sources merged, exploratory analysis with statistical tests, temporal feature engineering from the clock-in records, then seven algorithms compared with and without rebalancing, under two splitting protocols.

What the protocol held

What the data says about departures

Exploratory analysis yields three clear factors, each confirmed by a statistical test. Employees who travel frequently leave three times as often as those who never travel. Single employees leave two and a half times as often as divorced ones. And leavers have on average a year and a half less with their current manager.

A fourth gap shows up in the clock records, leavers working longer each day, but that one was never put to a test: it is a difference of means, not an established result.

That is the directly actionable part, and it depends on no model at all.

The leak-proof protocol, held end to end

A data leak is information slipping from training into evaluation through a back door, making a model look better than it is. The project built a second pipeline that guards against it at every step, and ran it alongside the first.

The split between training and test precedes every transformation. Missing-value filling, variable encoding and scaling are all computed on the training set alone, then applied as-is to the test set: at no point does the test set take part in computing anything. Rebalancing the rare class is likewise confined to training, and the test set keeps its natural proportion of departures.

And above all, both versions were kept side by side. The first fits the transformations before splitting, the second after. Rather than quietly fixing it, the project ran both over the same data to measure what the precaution changes.

The verdict is that it changes almost nothing, and the gap runs both ways depending on the algorithm. That is a result in itself: on this dataset, imputing before splitting was not inflating the scores. The precaution remains good practice, but it is justified on principle rather than by a measured gain, and the project went and checked instead of assuming.

constraint met

Split before every transformation, imputation, encoding, scaling and rebalancing fitted on the training set alone, test set left at its natural distribution, and both protocols compared over the same data.

What these scores are not worth

The dataset is synthetic, derived from a widely reused public case. The relationships it carries are plausible, they are not those of a real company, and none of the above transfers as it stands to an HR department.

The comparison scores are overstated, for the reason set out just below: the models were graded on people they had already met. The relative ranking of the algorithms stays informative, the absolute values do not.

The trap we did not see

The public-dataset trap we did not see. The leak-proof protocol described above is correct on every point one usually checks. It lacked one check that nobody on the project thought to run: counting the genuinely distinct people.

The file advertises 4,410rowsproductionEmployees in the dataset: 4,410 rows (production). numbered in sequence. There are in fact only 1,573peopleproductionThe file advertises 4410 employees numbered 1 to 4410. In reality most appear in triplicate: there are only a third as many real people.: most appear in three copies, identical across every HR attribute and both surveys, target included, the identifier aside. That replication comes from the original public dataset. Splitting those rows at random therefore scatters one person’s copies across both sides of the boundary.

A control model trained on those same attributes goes from 0.9996AUCtest setAn AUC of 1 means "never wrong", 0.5 means "answering at random". A score this close to 1 on a human problem is not an achievement, it is a warning sign. to 0.7332 ± 0.03AUCtest setThis is the model's real level: clearly better than chance, and a long way from the perfect score the usual split advertised. once people are kept on a single side. It is neither the project’s model nor its variables: the clock records do differ from one copy to the next. That control measures what recognising a record alone is worth, and the answer is: nearly the whole gap.

The clue was sitting in the repository, and nobody picked it up. The project’s written report announces 0.76F1test setThe project's written report announces a far lower score than the executed notebook shows, and it is the report that looks right. for the model it recommends, where the executed notebook shows 0.986F1test setF1 of the best tree-based model: 0.986 F1 (test set, n = 882).. Two documents from the same piece of work, two answers twenty points apart, and neither of them points to the other. A gap that wide between what you write down and what the machine prints is exactly the kind of signal worth digging into.

The lesson fits in one line of code never written: count the distinct individuals before splitting, especially on a public dataset thousands of people reuse without checking.

reproducible artefact

The diagnosis replays with one command: the script groups rows by person, compares both splits over the same data, and writes the two values above.

I would use the logistic regression as a detector. It was right there, among the seven algorithms compared, and it was shouting: a linear model cannot retain an individual, so the huge gap between it and the tree-based models was the symptom, available in the very first results table. Nobody read it as a signal, starting with me.

I would raise the question of real costs at the framing stage. The whole project trades recall against precision in the name of a replacement cost that was never quantified with the business. The trade-off is defensible, its justification remains a hypothesis.

Three projects in this portfolio carry the same shape of blind spot, an aggregate hiding an untreated case: here an individual already seen, on Leyenda a class never learned, and on INSEI one pipeline path far less covered than the other.