Information gain
Information gain is a way of measuring how useful a piece of information is. Think of it like trying to figure out which pieces of a puzzle are the most important. Information gain helps us decide which pieces will help us to find an answer the quickest. It does this by looking at how much each piece will help reduce the uncertainty we have about the answer. The more helpful a piece of information is, the more information gain it will have.
Related topics others have asked about:
Akaike information criterion,
Bayesian information criterion,
Bhattacharyya distance,
Bregman divergence,
Cross-entropy,
Deviance information criterion,
Entropic value at risk,
Entropy power inequality,
Hellinger distance,
Information gain in decision trees,
Information gain ratio,
Information theory and measure theory