Performance versus best-in-class gradient-boosting and classical baselines, across seven public datasets.
MathFi.ai is a hosted prediction service, not a model factory. In every domain below there is a specific, costly decision a client must make: approve this loan or not, investigate this claim or not, escalate this patient or not. This analysis benchmarks the MathFi.ai end-to-end platform, its Feature Refinery feature-engineering layer and its Model Crucible model builder, against five widely deployed methods (XGBoost, LightGBM, CatBoost, Random Forest and Logistic Regression) across seven public datasets, each under an identical 80/20 train and test split. Differences in results therefore reflect modelling capability, not differences in data or partitioning.
Results are reported on the minority class, because that is where the value sits: the defaulting borrower, the fraudulent claim, the stroke patient, the failing signal. Under class imbalance, headline accuracy is easily inflated. A model that predicts “no stroke” for everyone scores about 95% accuracy and saves nobody. Minority-class recall, precision and F1 are what determine whether a model earns its place in production, and they are the lens applied throughout.
Across the seven benchmarks, MathFi.ai delivers the highest minority-class F1 in six of seven cases, and the highest or joint-highest minority-class recall in all seven. Where it trails on a secondary metric, we say so plainly and explain the trade-off.
In every table the best value in each column is highlighted. Where more than one model ties for the top score, all of them are highlighted.
MathFi.ai leads on every headline measure: the highest accuracy (89.17%), the highest minority-class F1 (88.50%) and the highest precision (86.21%), while matching the best recall achieved by any method (90.91%). This is the ideal underwriting profile: it identifies just as many risky applicants as the most sensitive competitor, but with materially fewer false alarms, meaning fewer creditworthy customers are wrongly declined. In a lending business, that precision gain converts directly into approved revenue that competitors leave on the table.
| Method | F1 % | Recall % | Precision % | Accuracy % |
|---|---|---|---|---|
| MathFi.ai | 88.50 | 90.91 | 86.21 | 89.17 |
| XGBoost | 86.21 | 90.91 | 81.97 | 86.67 |
| LightGBM | 85.96 | 89.09 | 83.05 | 86.67 |
| Random Forest | 85.96 | 89.09 | 83.05 | 86.67 |
| Logistic Regression | 84.75 | 90.91 | 79.37 | 85.00 |
| CatBoost | 83.48 | 87.27 | 80.00 | 84.17 |
Dataset: UCI Credit Approval ↗. 690 anonymised credit applications, 15 mixed categorical and numerical attributes. A canonical benchmark for mixed-type, partially missing financial data.
MathFi.ai leads on accuracy (76.00%), minority-class F1 (63.60%) and recall (70.00%). The recall margin is the commercially decisive one: MathFi.ai catches 70% of genuine defaulters, against 66.67% for the closest competitor and only 50 to 57% for the rest; CatBoost, for instance, identifies just half. Its precision (58.33%) sits fractionally below CatBoost (58.82%) and LightGBM (58.62%), a gap of less than half a percentage point. This trade is strongly favourable, and it is worth being explicit about why: a false positive costs a lender the modest margin on one declined loan, whereas a false negative costs the outstanding principal. At the dataset's own 5:1 cost ratio, MathFi.ai's 13-point recall advantage over CatBoost dwarfs its half-point precision deficit. Ranking on the metric that carries the money, rather than the metric that looks tidiest, is the whole point.
| Method | F1 % | Recall % | Precision % | Accuracy % |
|---|---|---|---|---|
| MathFi.ai | 63.60 | 70.00 | 58.33 | 76.00 |
| XGBoost | 59.70 | 66.67 | 54.05 | 73.00 |
| LightGBM | 57.63 | 56.67 | 58.62 | 75.00 |
| Random Forest | 55.74 | 56.67 | 54.84 | 73.00 |
| CatBoost | 54.05 | 50.00 | 58.82 | 74.50 |
| Logistic Regression | 49.65 | 58.33 | 43.21 | 64.50 |
Dataset: UCI Statlog (German Credit Data) ↗. 1,000 loan records, 20 attributes covering financial history, employment and purpose. Ships with an official 5:1 cost matrix penalising missed defaulters.
This benchmark shows a different and instructive profile, which we present candidly because it is the most honest demonstration of what our platform optimises for. MathFi.ai achieves the highest minority-class recall of any method by a wide margin, 96.24%, against 81.95% for the gradient-boosting methods, surfacing very nearly every fraudulent claim in the test set. On this dataset, however, XGBoost, LightGBM and Random Forest lead on aggregate F1 (82.26% versus 75.74%), precision (82.58% versus 62.44%) and accuracy (97.18% versus 95.09%). MathFi.ai's configuration here is deliberately recall-dominant: it catches nearly all the fraud, and flags more legitimate claims in order to do so. Whether this is the right operating point depends on the relative cost of a missed fraud versus a false accusation, and for high-value transactions the prevented fraud losses far exceed the operational cost of investigating the additional flagged cases. In practice a recall-dominant model feeding a human review queue is the standard design for fraud operations. We also note a feature in development: clients will be able to set the objective function of model tuning directly, so that the operating point shown here becomes a dial the client controls rather than a fixed characteristic of the platform.
| Method | F1 % | Recall % | Precision % | Accuracy % |
|---|---|---|---|---|
| MathFi.ai | 75.74 | 96.24 | 62.44 | 95.09 |
| XGBoost | 82.26 | 81.95 | 82.58 | 97.18 |
| LightGBM | 82.26 | 81.95 | 82.58 | 97.18 |
| Random Forest | 82.26 | 81.95 | 82.58 | 97.18 |
| CatBoost | 81.45 | 84.21 | 78.87 | 96.94 |
| Logistic Regression | 36.69 | 93.23 | 22.84 | 74.36 |
Dataset: Warranty Claims (Kaggle) ↗. Warranty claims against manufactured goods with product, region, claim-value and service attributes. Fraud is a small minority of claims.
MathFi.ai leads on every metric that reflects the ability to find at-risk patients: the highest minority-class F1 (27.04%, a third higher than the next-best method at 20.31%), the highest precision (17.01%) and joint-highest recall (66.00%). The recall result is the one that matters. MathFi.ai identifies 66% of the people who go on to have a stroke; CatBoost finds 26%, XGBoost 30%, LightGBM and Random Forest 34%. In other words, the leading gradient-boosting methods miss roughly two thirds to three quarters of at-risk patients, while posting accuracy figures of 85 to 90% that look superficially excellent. CatBoost's 90.02% accuracy, the highest in the table, is achieved almost entirely by predicting "no stroke": it is an accurate model and a useless screening tool. MathFi.ai accepts a lower accuracy (82.58%) precisely because it is doing the job asked of it. Two caveats stated plainly: the absolute F1 scores here are low for every method, because the problem is genuinely hard, and Logistic Regression matches our 66% recall, but it does so with markedly worse precision (11.74% versus 17.01%) and F1, generating far more false alarms per patient found.
| Method | F1 % | Recall % | Precision % | Accuracy % |
|---|---|---|---|---|
| MathFi.ai | 27.04 | 66.00 | 17.01 | 82.58 |
| CatBoost | 20.31 | 26.00 | 16.67 | 90.02 |
| Random Forest | 20.24 | 34.00 | 14.41 | 86.89 |
| Logistic Regression | 19.94 | 66.00 | 11.74 | 74.07 |
| LightGBM | 18.78 | 34.00 | 12.98 | 85.62 |
| XGBoost | 18.75 | 30.00 | 13.64 | 87.28 |
Dataset: Stroke Prediction Dataset (IEEE DataPort) ↗. Population health records with demographic, lifestyle and comorbidity attributes. Minority class is about 1 in 20.
This is a long-established, clean and well-separated benchmark, and several strong methods can reach perfect minority-class precision on it; LightGBM and CatBoost both do. The perfect 100.00% precision recorded by MathFi.ai should therefore be read as a property of a well-behaved dataset, not as a claim of infallibility. What distinguishes MathFi.ai is that it is the only method to achieve perfect precision without paying for it in recall: LightGBM and CatBoost reach 100% precision only by dropping to 85.00% and 80.00% recall respectively, whereas MathFi.ai holds 95.00% recall. It leads on accuracy (98.33%) and F1 (97.44%), and no competing method matches it on any metric.
| Method | F1 % | Recall % | Precision % | Accuracy % |
|---|---|---|---|---|
| MathFi.ai | 97.44 | 95.00 | 100.00 | 98.33 |
| LightGBM | 91.89 | 85.00 | 100.00 | 95.00 |
| CatBoost | 88.89 | 80.00 | 100.00 | 93.33 |
| XGBoost | 87.18 | 85.00 | 89.47 | 91.67 |
| Random Forest | 86.49 | 80.00 | 94.12 | 91.67 |
| Logistic Regression | 77.78 | 70.00 | 87.50 | 86.67 |
Dataset: UCI Ionosphere ↗. 351 radar returns, 34 continuous attributes derived from antenna signal processing.
This dataset showcases a capability none of the competing libraries offers out of the box: MathFi.ai produces a family of models from the same data, each tuned to a different operating point on the precision-recall frontier, and lets the client choose the one that matches their cost structure. We present two. The precision-tuned model is the outright leader of the field on the metrics that reward balanced, confident diagnosis: it posts the highest accuracy (86.42%), the highest minority-class F1 (84.20%) and the highest precision (87.01%) of any method tested, the configuration for a resource-constrained setting where each positive prediction triggers an expensive downstream investigation and false alarms must be contained. The recall-tuned model is built for the opposite priority, catching as many true cases as possible: it achieves 91.46% recall, over seven points clear of the best competitor (XGBoost, 84.15%) and more than thirteen points above LightGBM, meaning it misses roughly half as many diseased patients as the leading gradient-boosting methods. To be candid, that comes at a cost: its F1 (81.96%) sits just below XGBoost (82.63%) and its precision (74.25%) is the lowest in the table. It is the right model when a missed diagnosis is the outcome you cannot accept, and the client can have it from the same dataset at the flick of a configuration. Competitors force you to take whatever balance their default objective happens to produce; MathFi.ai lets the clinical or commercial context choose.
| Method | F1 % | Recall % | Precision % | Accuracy % |
|---|---|---|---|---|
| MathFi.ai precision-tuned | 84.20 | 81.70 | 87.01 | 86.42 |
| MathFi.ai recall-tuned | 81.96 | 91.46 | 74.25 | 82.06 |
| XGBoost | 82.63 | 84.15 | 81.18 | 84.24 |
| Random Forest | 82.21 | 81.71 | 82.72 | 84.24 |
| CatBoost | 80.49 | 80.49 | 80.49 | 82.61 |
| LightGBM | 80.00 | 78.05 | 82.05 | 82.61 |
| Logistic Regression | 78.75 | 76.83 | 80.77 | 81.52 |
Dataset: Heart Failure Prediction (Kaggle) ↗. 918 patient records combining five older cardiac datasets, 11 clinical attributes.
MathFi.ai leads on the two measures that govern clinical usefulness: the highest minority-class F1 (71.11%) and, decisively, the highest recall (84.21%), nearly sixteen points above every competing method, all of which cluster at 68.42% or below. CatBoost identifies only 42.11% of patients who die: it misses more than half of them. Our accuracy (78.33%) sits marginally below Logistic Regression (80.00%) and our precision (61.53%) below several methods, and as with consumer default that trade-off is deliberate and correct: the cost of escalating a patient who would have survived is an additional review, while the cost of failing to escalate a patient who dies is not recoverable. On a decision this asymmetric, recall is the metric the clinician is actually buying.
| Method | F1 % | Recall % | Precision % | Accuracy % |
|---|---|---|---|---|
| MathFi.ai | 71.11 | 84.21 | 61.53 | 78.33 |
| Logistic Regression | 68.42 | 68.42 | 68.42 | 80.00 |
| Random Forest | 66.67 | 68.42 | 65.00 | 78.33 |
| LightGBM | 55.56 | 52.63 | 58.82 | 73.33 |
| XGBoost | 54.05 | 52.63 | 55.56 | 71.67 |
| CatBoost | 51.61 | 42.11 | 66.67 | 75.00 |
Dataset: Heart Failure Clinical Records (Chicco and Jurman) ↗. 299 heart-failure patients, 12 clinical attributes such as ejection fraction and serum creatinine. Peer-reviewed and published.
Across seven independent public benchmarks spanning credit, insurance, clinical care and signal integrity, MathFi.ai delivers the highest minority-class F1 in six of seven cases and the highest or joint-highest minority-class recall in all seven. Its consistent signature is recall: in every domain tested it surfaces more of the rare, high-value cases than any competing method, and in several (stroke, fatality risk, warranty fraud) it does so by margins that change what the model is capable of doing in the field. On two of them, credit approval and signal integrity, it pairs that recall leadership with best-in-class accuracy and precision as well. Where it does not lead, we have said so: the warranty claims benchmark is a recall and precision trade the client should make consciously, and the recall-tuned cardiac model is an operating point, not an all-round winner. Both point at the same underlying capability, which is the platform's real differentiator: MathFi.ai can generate multiple models from a single dataset, each occupying a different position on the precision-recall frontier, and, with the objective-function control now in development, hand that choice to the client. Competing libraries deliver one model and one implicit set of priorities. We deliver the decision the client actually needs.
Bring us the decision your current models get wrong too often, and we will build on your own data and show you the lift.
See it on your dataAll seven datasets are public, widely cited and independently downloadable, so any reader may inspect them and reproduce these benchmarks. Each was selected because it is an established reference set and mirrors a real commercial or clinical decision under class imbalance, the condition in which conventional models most often fail quietly.
690 anonymised credit applications, 15 mixed categorical and numerical attributes, target: approve or decline. Attribute names and values are redacted to protect confidentiality.
A canonical benchmark for mixed-type, partially missing financial data. Because the features are anonymised, no model can rely on domain shortcuts; it must learn genuine structure, which is why it remains a standard test of a platform's feature-engineering capability.
1,000 loan records, 20 attributes (financial history, employment, purpose), target: good or bad credit risk. Ships with an official 5:1 cost matrix penalising missed defaulters.
The reference dataset for cost-sensitive credit scoring. Its published asymmetric cost matrix makes it one of the few benchmarks that explicitly encodes the real economics of lending, and it is directly representative of consumer underwriting.
Warranty claims against manufactured goods with product, region, claim-value and service attributes, target: fraudulent or genuine claim. Fraud is a small minority of claims.
Representative of high-volume, low-prevalence fraud detection across insurance, warranty and payments. The strong class imbalance and asymmetric loss profile make it a realistic proxy for production fraud operations.
Population health records with demographic, lifestyle and comorbidity attributes, target: stroke event. Minority class is about 1 in 20, roughly 5%.
A severe-imbalance benchmark widely used to expose the weakness of accuracy as a metric. Directly representative of population-level preventative screening, where the cost of a missed case vastly exceeds the cost of a follow-up.
Inspect the data: Stroke Prediction Dataset (IEEE DataPort) ↗
351 radar returns, 34 continuous attributes derived from antenna signal processing, target: good or bad return.
A long-established, clean, well-separated benchmark used to sanity-check classifier behaviour. Representative of sensor-integrity and anomaly-screening problems in telecoms, defence and industrial IoT. Its separability is why several methods reach perfect precision on it.
918 patient records combining five older cardiac datasets, 11 clinical attributes, target: presence of heart disease.
One of the largest consolidated public cardiac datasets. Widely used to benchmark clinical classifiers because its error asymmetry, missed diagnosis versus unnecessary test, mirrors genuine diagnostic decision-making.
299 heart-failure patients, 12 clinical attributes (ejection fraction, serum creatinine and others), target: death during follow-up. Peer-reviewed and published.
A peer-reviewed clinical mortality benchmark with an unusually small sample, a realistic stress test, since most real clinical deployments must learn from limited data rather than millions of rows.
Inspect the data: Heart Failure Clinical Records (Chicco and Jurman) ↗
Every method was evaluated on an identical 80/20 train and test split of each dataset, with the same input data supplied to MathFi.ai and to each competing library. Competing-method benchmarks were produced under the direction of the MathFi.ai CTO. All figures are reported as measured; no result has been omitted or reweighted.
Ready to see what MathFi.ai does on the decision that costs you most?
Get started