Number Representations & States

"how numbers are stored in computers"

Cancellation

Without a guard digit, the relative error committed when subtracting two nearby quantities can be very large. In other words, the evaluation of any expression containing a subtraction (or an addition of quantities with opposite signs) could result in a relative error so large that all the digits are meaningless. When subtracting nearby quantities, the most significant digits in the operands match and cancel each other.

Catastrophic cancellation

Catastrophic cancellation occurs when the operands are subject to rounding errors. For example, the quadratic formula () contains the expression , where the quantities and are subject to rounding errors since they are the results of floating-point multiplications. Suppose that they are rounded to the nearest floating-point number, and are therefore accurate to within ulp. When they are subtracted, cancellation can cause many of the accurate digits to disappear, leaving behind a higher proportion of digits contaminated by rounding error. Therefore, the difference may have an error of several ulps.

For example, consider . The exact value of is , but rounds to and rounds to , yielding a final result of which is an error of ulps. The subtraction did not introduce any error, but rather exposed the error introduced in the earlier multiplications.

Benign cancellation

Benign cancellation occurs when subtracting exactly known quantities. If and have no rounding error, then by subtracting with a guard digit, the difference x-y has a very small relative error (less than 2).

A formula that exhibits catastrophic cancellation can sometimes be rearranged to eliminate the problem. Again consider the quadratic formula