Performance

Minority-class benchmarking analysis

Performance versus best-in-class gradient-boosting and classical baselines, across seven public datasets.

Revised 14 July 2026  ·  Version 2.1  ·  Authors Dr. Saied Abedi (Co-Founder and CEO), David Fernandez (Co-Founder and CTO) and Sean O'Neill (Co-Founder and Chairman of the Board)

MathFi.ai is an end-to-end prediction service, not a model factory. In every domain below there is a specific, costly decision a client must make: approve this loan or not, investigate this claim or not, escalate this patient or not. This analysis benchmarks the MathFi.ai end-to-end platform, its Feature Refinery feature-engineering layer and its Model Crucible model builder, against five widely deployed methods (XGBoost, LightGBM, CatBoost, Random Forest and Logistic Regression) across seven public datasets, each under an identical 80/20 train and test split. Differences in results therefore reflect modelling capability, not differences in data or partitioning.

Results are reported on the minority class, because that is where the value sits: the defaulting borrower, the fraudulent claim, the stroke patient, the failing signal. Under class imbalance, headline accuracy is easily inflated. A model that predicts “no stroke” for everyone scores about 95% accuracy and saves nobody. Minority-class recall, precision and F1 are what determine whether a model earns its place in production, and they are the lens applied throughout.

Across the seven benchmarks, MathFi.ai delivers the highest minority-class F1 in six of seven cases, and the highest or joint-highest minority-class recall in all seven. Where it trails on a secondary metric, we say so plainly and explain the trade-off.

How to read the tables

  • F1: one score that balances recall and precision, so a model cannot look good by being reckless or by being timid.
  • Recall: of all the real cases, how many the model caught. Higher means fewer missed frauds, defaults or failures.
  • Precision: of the cases the model flagged, how many were right. Higher means fewer false alarms.
  • Accuracy: how often the model is right across every case, common and rare alike. On imbalanced data this is the number to distrust, since a model can score high accuracy simply by assuming the rare case never happens, as the stroke results below show.

In every table the best value in each column is highlighted. Where more than one model ties for the top score, all of them are highlighted.

Credit Approval: Automated Lending Decisions

The predictive decision. Anonymised consumer credit applications, where the platform must decide whether an applicant will be a good or bad credit risk. The challenge is that the data is mixed-type and partially redacted, so the decision must be made from weak, heterogeneous signals, exactly the condition of a real underwriting desk.
  • Leads on every headline measure: F1 88.50%, precision 86.21%, accuracy 89.17%. No competing method wins a single column.
  • Matches the best recall in the field (90.91%), so none of that precision is bought by letting risky applicants through.
  • The precision gap is where the money is: 4.24 points above XGBoost at identical recall, which means fewer creditworthy customers wrongly declined.
MethodF1 %Recall %Precision %Accuracy %
MathFi.ai 88.50 90.91 86.21 89.17
XGBoost 86.21 90.91 81.97 86.67
LightGBM 85.96 89.09 83.05 86.67
Random Forest 85.96 89.09 83.05 86.67
Logistic Regression 84.75 90.91 79.37 85.00
CatBoost 83.48 87.27 80.00 84.17

Dataset: UCI Credit Approval ↗. 690 anonymised credit applications, 15 mixed categorical and numerical attributes. A canonical benchmark for mixed-type, partially missing financial data.

German Credit Risk: Consumer Default Prediction

The predictive decision. One thousand consumer loan records in which the platform must flag applicants likely to default. The dataset is deliberately imbalanced and ships with an asymmetric cost matrix: missing a bad borrower is officially rated five times more costly than wrongly rejecting a good one.
  • Leads F1 (63.60%), recall (70.00%) and accuracy (76.00%).
  • Recall is the commercially decisive one: 70% of genuine defaulters caught, against 66.67% for the next best and 50.00% for CatBoost.
  • Precision sits fractionally behind CatBoost, 58.33% against 58.82%, a gap of half a point.
  • The dataset ships a 5:1 cost matrix, which makes that trade clearly favourable: a false positive costs the margin on one declined loan, a false negative costs the principal.
MethodF1 %Recall %Precision %Accuracy %
MathFi.ai 63.60 70.00 58.33 76.00
XGBoost 59.70 66.67 54.05 73.00
LightGBM 57.63 56.67 58.62 75.00
Random Forest 55.74 56.67 54.84 73.00
CatBoost 54.05 50.00 58.82 74.50
Logistic Regression 49.65 58.33 43.21 64.50

Dataset: UCI Statlog (German Credit Data) ↗. 1,000 loan records, 20 attributes covering financial history, employment and purpose. Ships with an official 5:1 cost matrix penalising missed defaulters.

Warranty Claim Fraud: Insurance Fraud Detection

The predictive decision. Warranty claims filed against manufactured goods, where the platform must decide which claims to route to a fraud investigator. Fraud is rare, a small minority of claims, so the model must find a thin signal in a large volume of legitimate activity without drowning the investigation team.
  • Highest recall by a wide margin: 96.24%, against 81.95% for the gradient-boosting methods. Very nearly every fraudulent claim in the test set is surfaced.
  • We do not lead here. XGBoost, LightGBM and Random Forest take F1 (82.26% against 75.74%), precision (82.58% against 62.44%) and accuracy (97.18% against 95.09%).
  • The configuration is deliberately recall-dominant: it catches nearly all the fraud and flags more legitimate claims in order to do it.
  • Whether that is the right operating point depends on the cost of a missed fraud against a false accusation. Feeding a human review queue, recall-dominant is the standard design for fraud operations.
MethodF1 %Recall %Precision %Accuracy %
MathFi.ai 75.74 96.24 62.44 95.09
XGBoost 82.26 81.95 82.58 97.18
LightGBM 82.26 81.95 82.58 97.18
Random Forest 82.26 81.95 82.58 97.18
CatBoost 81.45 84.21 78.87 96.94
Logistic Regression 36.69 93.23 22.84 74.36

Dataset: Warranty Claims (Kaggle) ↗. Warranty claims against manufactured goods with product, region, claim-value and service attributes. Fraud is a small minority of claims.

Stroke Prediction: Population Health Screening

The predictive decision. Population health records in which the platform must identify individuals at risk of stroke so they can be prioritised for preventative care. Only about one person in twenty in the data has had a stroke, making this the hardest imbalance in the benchmark set, and the clearest illustration of why accuracy is a dangerous metric.
  • Leads F1 (27.04%, a third higher than the next best at 20.31%), precision (17.01%) and joint-highest recall (66.00%).
  • Finds 66% of the people who go on to have a stroke. CatBoost finds 26%, XGBoost 30%, LightGBM and Random Forest 34%.
  • CatBoost posts the table's highest accuracy, 90.02%, almost entirely by predicting "no stroke". It is an accurate model and a useless screening tool.
  • Two caveats stated plainly: absolute F1 is low for every method because the problem is genuinely hard, and Logistic Regression matches our 66% recall with far worse precision (11.74% against 17.01%).
MethodF1 %Recall %Precision %Accuracy %
MathFi.ai 27.04 66.00 17.01 82.58
CatBoost 20.31 26.00 16.67 90.02
Random Forest 20.24 34.00 14.41 86.89
Logistic Regression 19.94 66.00 11.74 74.07
LightGBM 18.78 34.00 12.98 85.62
XGBoost 18.75 30.00 13.64 87.28

Dataset: Stroke Prediction Dataset (IEEE DataPort) ↗. Population health records with demographic, lifestyle and comorbidity attributes. Minority class is about 1 in 20.

Ionosphere: Radar Signal Integrity

The predictive decision. Radar returns from the ionosphere, classified as good, meaning structured and usable signal, or bad, meaning no usable structure. The decision is whether to trust or discard a sensor reading, a direct analogue of signal-integrity and anomaly screening in telecoms, defence and industrial monitoring. This dataset is not natively tabular: its 34 features are the real and imaginary components of a complex autocorrelation function computed from raw high-frequency radar returns, making it a direct demonstration of MathFi.ai operating on signal data converted into numeric features.
  • Leads accuracy (98.33%) and F1 (97.44%). No competing method matches it on any metric.
  • The only method to reach 100% precision without paying for it in recall. LightGBM and CatBoost also reach 100% precision, but drop to 85.00% and 80.00% recall to get there. MathFi.ai holds 95.00%.
  • Read that perfect precision as a property of a clean, well-separated dataset, not as a claim of infallibility.
MethodF1 %Recall %Precision %Accuracy %
MathFi.ai 97.44 95.00 100.00 98.33
LightGBM 91.89 85.00 100.00 95.00
CatBoost 88.89 80.00 100.00 93.33
XGBoost 87.18 85.00 89.47 91.67
Random Forest 86.49 80.00 94.12 91.67
Logistic Regression 77.78 70.00 87.50 86.67

Dataset: UCI Ionosphere ↗. 351 radar returns, 34 continuous attributes derived from antenna signal processing.

Heart Failure: Cardiac Disease Detection (Two Operating Points)

The predictive decision. Clinical and demographic records in which the platform must predict whether a patient has heart disease. The challenge is that the two error types have profoundly unequal consequences, an unnecessary follow-up test is an inconvenience while a missed diagnosis can be fatal, and different clinical settings weigh them differently.
  • Two models from the same data, at different points on the precision-recall frontier. None of the competing libraries offers this out of the box.
  • The precision-tuned model leads the whole field: highest accuracy (86.42%), highest F1 (84.20%) and highest precision (87.01%) of any method tested. It suits a setting where each positive triggers an expensive investigation.
  • The recall-tuned model catches 91.46%, over seven points clear of XGBoost (84.15%) and more than thirteen above LightGBM. It misses roughly half as many diseased patients as the leading gradient-boosting methods.
  • That costs something, and we say so: its F1 (81.96%) sits just below XGBoost (82.63%) and its precision (74.25%) is the lowest in the table. It is the right model only when a missed diagnosis is the outcome you cannot accept.
MethodF1 %Recall %Precision %Accuracy %
MathFi.ai precision-tuned 84.20 81.70 87.01 86.42
MathFi.ai recall-tuned 81.96 91.46 74.25 82.06
XGBoost 82.63 84.15 81.18 84.24
Random Forest 82.21 81.71 82.72 84.24
CatBoost 80.49 80.49 80.49 82.61
LightGBM 80.00 78.05 82.05 82.61
Logistic Regression 78.75 76.83 80.77 81.52

Dataset: Heart Failure Prediction (Kaggle) ↗. 918 patient records combining five older cardiac datasets, 11 clinical attributes.

Heart Failure Fatality Risk: Clinical Mortality Prediction

The predictive decision. Clinical records of heart-failure patients, where the platform must predict which patients are at risk of death during the follow-up period, so that care can be escalated. With only 299 patients in total, the challenge is to learn a reliable mortality signal from a very small sample, the situation of most real clinical deployments.
  • Leads F1 (71.11%) and, decisively, recall (84.21%), nearly sixteen points above every competitor, which all cluster at 68.42% or below.
  • CatBoost identifies 42.11% of the patients who die. It misses more than half of them.
  • Accuracy (78.33%) sits marginally below Logistic Regression (80.00%), and precision (61.53%) below several methods.
  • That trade is deliberate: escalating a patient who would have survived costs one extra review, while failing to escalate a patient who dies is not recoverable.
MethodF1 %Recall %Precision %Accuracy %
MathFi.ai 71.11 84.21 61.53 78.33
Logistic Regression 68.42 68.42 68.42 80.00
Random Forest 66.67 68.42 65.00 78.33
LightGBM 55.56 52.63 58.82 73.33
XGBoost 54.05 52.63 55.56 71.67
CatBoost 51.61 42.11 66.67 75.00

Dataset: Heart Failure Clinical Records (Chicco and Jurman) ↗. 299 heart-failure patients, 12 clinical attributes such as ejection fraction and serum creatinine. Peer-reviewed and published.

Overall assessment: MathFi.ai outperforms the competition

  • Highest minority-class F1 in six of seven benchmarks, and the highest or joint-highest recall in all seven. On credit approval and signal integrity it leads on accuracy and precision too.
  • Recall is the consistent signature: in every domain tested it surfaces more of the rare, high-value cases than any competing method. On stroke, fatality risk and warranty fraud, by margins that change what the model can do in the field.
  • Where it does not lead, we say so. The warranty claims benchmark is a recall-for-precision trade the client should make consciously, and the recall-tuned cardiac model is an operating point, not an all-round winner.
  • Both point at the real differentiator: several models from one dataset, each at a different point on the precision-recall frontier. Competing libraries give you one model and one implicit set of priorities. Objective-function control is in development.

Bring us the decision your current models get wrong too often, and we will build on your own data and show you the lift.

See it on your data

Appendix: dataset provenance and suitability

All seven datasets are public, widely cited and independently downloadable, so any reader may inspect them and reproduce these benchmarks. Each was selected because it is an established reference set and mirrors a real commercial or clinical decision under class imbalance, the condition in which conventional models most often fail quietly.

Credit Approval: Automated Lending Decisions

690 anonymised credit applications, 15 mixed categorical and numerical attributes, target: approve or decline. Attribute names and values are redacted to protect confidentiality.

A canonical benchmark for mixed-type, partially missing financial data. Because the features are anonymised, no model can rely on domain shortcuts; it must learn genuine structure, which is why it remains a standard test of a platform's feature-engineering capability.

Inspect the data: UCI Credit Approval ↗

German Credit Risk: Consumer Default Prediction

1,000 loan records, 20 attributes (financial history, employment, purpose), target: good or bad credit risk. Ships with an official 5:1 cost matrix penalising missed defaulters.

The reference dataset for cost-sensitive credit scoring. Its published asymmetric cost matrix makes it one of the few benchmarks that explicitly encodes the real economics of lending, and it is directly representative of consumer underwriting.

Inspect the data: UCI Statlog (German Credit Data) ↗

Warranty Claim Fraud: Insurance Fraud Detection

Warranty claims against manufactured goods with product, region, claim-value and service attributes, target: fraudulent or genuine claim. Fraud is a small minority of claims.

Representative of high-volume, low-prevalence fraud detection across insurance, warranty and payments. The strong class imbalance and asymmetric loss profile make it a realistic proxy for production fraud operations.

Inspect the data: Warranty Claims (Kaggle) ↗

Stroke Prediction: Population Health Screening

Population health records with demographic, lifestyle and comorbidity attributes, target: stroke event. Minority class is about 1 in 20, roughly 5%.

A severe-imbalance benchmark widely used to expose the weakness of accuracy as a metric. Directly representative of population-level preventative screening, where the cost of a missed case vastly exceeds the cost of a follow-up.

Inspect the data: Stroke Prediction Dataset (IEEE DataPort) ↗

Ionosphere: Radar Signal Integrity

351 radar returns, 34 continuous attributes derived from antenna signal processing, target: good or bad return.

A long-established, clean, well-separated benchmark used to sanity-check classifier behaviour. Representative of sensor-integrity and anomaly-screening problems in telecoms, defence and industrial IoT. Its separability is why several methods reach perfect precision on it.

Inspect the data: UCI Ionosphere ↗

Heart Failure: Cardiac Disease Detection (Two Operating Points)

918 patient records combining five older cardiac datasets, 11 clinical attributes, target: presence of heart disease.

One of the largest consolidated public cardiac datasets. Widely used to benchmark clinical classifiers because its error asymmetry, missed diagnosis versus unnecessary test, mirrors genuine diagnostic decision-making.

Inspect the data: Heart Failure Prediction (Kaggle) ↗

Heart Failure Fatality Risk: Clinical Mortality Prediction

299 heart-failure patients, 12 clinical attributes (ejection fraction, serum creatinine and others), target: death during follow-up. Peer-reviewed and published.

A peer-reviewed clinical mortality benchmark with an unusually small sample, a realistic stress test, since most real clinical deployments must learn from limited data rather than millions of rows.

Inspect the data: Heart Failure Clinical Records (Chicco and Jurman) ↗

Every method was evaluated on an identical 80/20 train and test split of each dataset, with the same input data supplied to MathFi.ai and to each competing library. Competing-method benchmarks were produced under the direction of the MathFi.ai CTO. All figures are reported as measured; no result has been omitted or reweighted.

Ready to see what MathFi.ai does on the decision that costs you most?

Get started