Algorithmic Fairness Bias Metric Calculator
Algorithmic fairness outcomes this calculator compares
This algorithmic fairness calculator converts group-specific model outcomes into directly comparable rates. Enter the confusion-matrix counts for groups A and B, and it calculates demographic parity difference, the group B divided by group A positive prediction rate ratio, and equal opportunity difference. These figures do not answer every ethical, policy, or legal question, but they reveal whether positive decisions and successful identification of actual positives differ between the two groups.
That group-level view matters because aggregate model accuracy can mask unequal outcomes. A model may look acceptable overall while one group receives fewer positive predictions, or while positive cases in that group are missed more often. Separating the confusion matrix by group exposes those patterns. The calculation can support an audit, a decision-threshold review, a teaching exercise, or an initial check before deeper fairness analysis.
The outputs should be read as measurements of the counts entered, not as findings about intent, protected status, or compliance. This page does not classify people or make a legal determination. It provides repeatable arithmetic for comparing the present decision rule when its inputs, threshold, and evaluation context are held consistent.
Confusion-matrix inputs for each group
For this fairness comparison, enter four confusion-matrix counts for each group from the same prediction task. A true positive is a positive prediction that was correct; a false positive is an incorrect positive prediction. A false negative is a real positive that the model missed, and a true negative is a correctly predicted negative. Use raw counts rather than percentages, and ensure that both groups were evaluated over the same period and at the same decision threshold.
- True positives: correct positive decisions.
- False positives: incorrect positive decisions.
- False negatives: actual positives the model did not select.
- True negatives: correct negative decisions.
The fairness metrics use these entries as counts with no conversion or hidden unit. The comparison therefore depends on consistent source data. Combining monthly counts for one group with annual counts for another would not yield a meaningful comparison. A group with no actual positive cases also has no defined true positive rate, because there is no positive-case denominator for equal opportunity.
The form’s initial values illustrate the calculator’s arithmetic only. They are not target outcomes and do not define a fair model. Replace them with counts from the model slice you are examining to obtain a compact comparison of its group-specific outcomes.
Fairness metric formulas used by the calculator
This calculator first derives a positive prediction rate and a true positive rate for each group, then compares those rates. The positive prediction rate is the share of all cases in a group that received a positive prediction, regardless of whether that prediction was correct. It is the rate used for demographic parity.
For equal opportunity, the calculator uses the true positive rate: the share of actual positive cases that received a positive prediction. This rate is also commonly called recall or sensitivity for the positive class.
Demographic parity difference is the group A positive prediction rate minus the group B positive prediction rate, displayed in percentage points. A result close to zero means similar overall positive-decision rates. A positive result means group A has the higher positive prediction rate; a negative result means group B does.
The positive rate ratio expresses the same selection-rate comparison multiplicatively, using group B divided by group A. A ratio of 1.00 indicates equal rates. Values below 1 indicate that group B’s positive prediction rate is lower than group A’s, while values above 1 indicate the reverse.
Equal opportunity difference compares the two groups’ true positive rates. False positives and true negatives do not enter this rate directly; the metric asks whether actual positive cases are recognized at similar rates. As with demographic parity difference, a value near zero indicates closer alignment, and the sign identifies the group with the higher true positive rate.
These measures need not move together. Changing a threshold can leave the positive prediction rates close while widening the true positive rate gap, or do the opposite. That difference is expected because demographic parity and equal opportunity describe different aspects of model behavior.
Default confusion-matrix example and metric results
The default group A entries contain 50 true positives, 10 false positives, 20 false negatives, and 120 true negatives. That produces 200 total cases. Its positive prediction rate is 60 divided by 200, or 30.00%, and its true positive rate is 50 divided by 70, or approximately 71.43%.
The default group B entries contain 40 true positives, 15 false positives, 30 false negatives, and 100 true negatives. This group has 185 cases. Its positive prediction rate is 55 divided by 185, approximately 29.73%, while its true positive rate is 40 divided by 70, approximately 57.14%.
For these entered counts, demographic parity difference is approximately 0.27 percentage points, since 30.00% minus 29.73% is small. The group B-to-group A positive rate ratio is approximately 0.99. However, equal opportunity difference is approximately 14.29 percentage points because group A has the higher true positive rate. The example shows why similar selection rates do not necessarily mean that actual positive cases are recognized similarly.
| Group | Total cases | Positive prediction rate | True positive rate |
|---|---|---|---|
| A | 200 | 30.00% | 71.43% |
| B | 185 | 29.73% | 57.14% |
In practical terms, these default counts show nearly equal overall selection rates but a substantially higher rate of correctly identifying actual positives for group A. A review prompted by that pattern could examine score distributions, outcome labels, data quality, and the effects of alternative decision thresholds rather than relying on parity alone.
Reading the demographic parity and opportunity outputs
The result panel presents three algorithmic fairness comparisons. Demographic parity difference is shown in percentage points and compares the groups’ overall positive prediction rates. The positive rate ratio reports group B divided by group A on a multiplicative scale. Equal opportunity difference, also in percentage points, compares the groups’ true positive rates among actual positive cases.
No single numerical cutoff establishes fairness in every use case. The consequences of errors, the affected population, the decision process, and applicable requirements vary by domain. Treat the values as evidence about model behavior rather than as a complete verdict. Even a small observed gap can merit review when many people are affected or missed positives carry serious harm; a larger gap based on sparse data also requires careful investigation.
Direction is important when reading the results. Both difference metrics are calculated as group A minus group B, so a positive difference means group A has the higher relevant rate. The ratio intentionally uses group B divided by group A. Consequently, a positive difference and a ratio below 1 can describe the same directional pattern.
The copy button can help document repeated threshold or data scenarios. Change one input set at a time, copy the resulting summary, and preserve the comparison alongside the assumptions used. That makes it easier to trace how a model change affected parity and opportunity measures.
Limits and careful use of fairness comparisons
These algorithmic fairness calculations assume that every record belongs to exactly one confusion-matrix cell and that groups A and B were evaluated under the same model and decision policy. The page will calculate rates from inconsistent inputs, but rates drawn from different thresholds, periods, label definitions, or populations should not be interpreted as a sound group comparison.
Sample size is a central limitation of fairness-rate comparisons. A few changed records can substantially alter rates for a small group, especially when that group has few actual positive cases. Formal audits may supplement point estimates with confidence intervals or other uncertainty analysis. This calculator does not estimate uncertainty; it reports the direct arithmetic from the counts supplied.
Fairness objectives can also conflict. A threshold adjustment that narrows a demographic parity gap may widen a true positive rate gap or change false positive outcomes. Such tradeoffs are common when base rates and score distributions differ. A meaningful review identifies the relevant harms for the domain, documents tradeoffs, and evaluates more than one measure.
- Counts must be non-negative: negative confusion-matrix entries are not meaningful.
- Each group needs observations: a zero total prevents calculation of group rates.
- Equal opportunity needs actual positives: TP plus FN must be greater than zero in each group.
- Metrics are partial evidence: use these comparisons with broader model, data, and domain review.
Used carefully, this calculator makes a fairness discussion auditable: the group counts, formulas, and consequences of changed assumptions remain explicit and checkable.