Kullback–Leibler Divergence Calculator

Introduction to KL divergence between discrete distributions

This KL divergence calculator compares two discrete probability distributions, or probability vectors, P and Q over the same outcomes. Enter each list as comma-separated values such as 0.6, 0.4. The calculator normalizes each list before it measures the gap between them, so raw counts and weights are acceptable as long as every entry is non-negative and each position still refers to the same outcome.

• KL divergence (sometimes called “forward KL”)
• Reverse KL
• Cross-entropy
• Jensen–Shannon divergence (JSD), a symmetric comparison built from the midpoint of P and Q

Choose ln for results in nats, or choose base 2 when you want results in bits and prefer the binary-log view of the same divergence calculation.

That normalization step makes the comparison shape-based rather than size-based. If P and Q are meant to describe a classifier, a language model, a survey response mix, or any other discrete table, the values should be lined up first. A small KL result means the shapes are close; a larger result usually means one list gives much less probability to an outcome that the other list considers important.

Definitions and formulas for discrete KL comparisons

For this KL divergence calculator, P and Q are discrete distributions over outcomes , with , , and with both lists normalized to sum to 1 before the divergences are computed. In practice, that means the calculator can accept probabilities, counts, or weights, provided the two lists have the same length and describe the same ordered outcomes.

After normalization, the totals satisfy and , so the output depends on the distribution shape rather than the raw scale of the input numbers.

KL divergence

The Kullback–Leibler divergence from to measures how costly it is to use Q as the stand-in for data generated by P:

If you choose ln, then log is and the unit is nats. If you choose log2, the unit is bits.

Cross-entropy

For KL divergence and log-loss calculations, cross-entropy is the average cost of encoding outcomes from with probabilities supplied by :

Formula: H(P, Q) = − ∑ i = 1 n P_i · log(Q_i)

It relates to KL divergence via:

, where is the entropy of P.

Jensen–Shannon divergence (JSD)

In this calculator, Jensen–Shannon divergence is built from the midpoint distribution so both inputs contribute evenly:

JSD(P, Q) = 1/2 · D_KL(P‖M) + 1/2 · D_KL(Q‖M)

With base-2 logs, JSD is bounded between 0 and 1 bit for discrete distributions.

How to interpret KL divergence, cross-entropy, and JSD

• KL(P‖Q) = 0 only when the two probability lists match outcome by outcome after normalization.
• Asymmetry matters: KL(P‖Q) is generally not equal to KL(Q‖P), because each direction penalizes a different kind of mismatch between the two distributions.
• Units: nats (ln) or bits (log2). Bits are often easier to interpret as “extra bits per event” under an optimal code.
• Cross-entropy can be read as the expected coding cost if data truly follow P but you code using a model Q.
• JSD is often preferred when you want a finite, symmetric comparison and better behavior around zeros in one of the probability lists.

In practice, the most useful result is often not the single KL number but the pattern across the two directions. If forward KL is large while reverse KL is modest, Q may be missing an outcome that P treats as important. If both are small, the tables are broadly similar after scaling. That makes this calculator handy for checking model outputs, comparing empirical frequencies, or deciding whether two discrete summaries can reasonably be treated as the same shape.

Worked example: comparing P = 0.6, 0.4 to Q = 0.5, 0.5

For this KL divergence example, let:

• P = 0.6, 0.4
• Q = 0.5, 0.5

Using natural logs for the KL calculation:

D_KL(P‖Q) = 0.6·ln(0.6/0.5) + 0.4·ln(0.4/0.5)

= 0.6·ln(1.2) + 0.4·ln(0.8) ≈ 0.6·0.1823 + 0.4·(-0.2231) ≈ 0.0201 nats

This small positive value means the two discrete distributions are close, but not identical, on these two outcomes. The same idea applies when the lists have more categories: the calculator is looking for where the probability mass shifts, not just whether the totals add up.

Metric comparison for KL divergence at a glance

KL divergence limitations and assumptions

• Same length / same outcomes: The KL calculator compares the i-th probability in P to the i-th probability in Q, so both lists must describe the same outcomes in the same order.
• Non-negative inputs: Probabilities must be ≥ 0. Negative values are not meaningful for KL/cross-entropy/JSD.
• Normalization: If you enter counts or weights, the calculator normalizes them to probabilities before computing KL, cross-entropy, and JSD. That is convenient for empirical data, but it means the output reflects the normalized distributions rather than the raw counts.
• Zeros in P: Terms with P(i)=0 contribute 0 to KL by the limit behavior, so they do not cause problems by themselves.
• Zeros in Q: If Q(i)=0 while P(i)>0, then KL(P‖Q) diverges to infinity because the reference distribution assigns zero probability to an event that occurs under P. This is not a bug; it reflects an impossible event under Q.
• Finite precision: Very small probabilities can lead to large logs and numerical instability. If you see surprising outputs, consider smoothing (for example, adding a tiny epsilon to each probability and renormalizing), and report the method if used.
• Discrete-only: These formulas apply to discrete distributions. Continuous distributions require integrals and careful handling of densities (and units) rather than probabilities.

These assumptions matter most when you compare real data to a model prediction. Before trusting the number, make sure the categories are in the same order, because KL cannot tell whether two labels have been swapped. If an outcome appears in P but not in Q, forward KL will blow up, which often means you should smooth the inputs or reconsider the binning scheme.

KL divergence references

• T. M. Cover & J. A. Thomas, Elements of Information Theory.
• Wikipedia: Kullback–Leibler divergence; Jensen–Shannon divergence (for quick reference).

How to use this KL divergence calculator

1. Enter Probabilities P as a comma-separated list for the reference distribution.
2. Enter Probabilities Q with the same number of outcomes, and keep the order aligned with P.
3. Choose Log base as Natural (ln) for nats or Base 2 for bits.
4. Click Compute Divergences, then compare how much probability mass shifts between P and Q before you rely on the numbers in a model, report, or explanation. The summary and table show the main divergences together so you can read the forward and reverse directions side by side.