Computers operate on integers \(x\in \mathbb {Z}\) using modular arithmetic. Basically, given an unsigned integer
we have
The smallest possible value is when
and the largest is when
Let
For a given \(x\), the remaining distance to the maximum value is
Thus, for \(x+y\), we experience unsigned overflow if
or if you will
When this event happens, computers signal it using the carry flag \(C\):
The inverse of \(y\), i.e. \(-y\), in \(\mathbb {Z}_{2^n}\) is
since
Thus computers can compute
If
a carry is generated. In particular,
Thus we can compare two unsigned integers \(x\) and \(y\) using subtraction, if
then
Otherwise, the carry/borrow information tells us whether
Now consider signed integers
so
For \(x>0\) and \(y>0\), \(x+y\) experiences signed overflow if
or for a given \(x\),
For example,
But \(2^{n-1}\) is not representable as a positive signed integer. Its \(n\)-bit representation corresponds to
so we went from a large positive value to a very small negative value (signed overflow).
Computers usually signal the occurence of this event using an overflow flag \(V\). For example,
implies
Similarly, if
the addition of two very small negative values can overflow to large positive values. Signed overflow occurs if
For example, if
then any \(y\) such that
will lead to signed overflow. In particular,
Modulo \(2^n\),
Thus a negative result wraps to the largest positive signed value.
Computers detect this case when
and
giving
Now consider
First, we can only experience signed overflow in subtraction when
If
then there is no signed overflow.
If
then
If
then
And if
then
Similarly, if
there is no signed overflow.
If
then
If
then
Again,
Now observe that if
then
The subtraction \(x-y\) can either have
or, if it overflows,
Thus in both cases
On the other hand, if
then
The subtraction can either have
or, if it overflows,
Thus in both cases
Therefore we can decide whether
by checking
where \(V\) detects signed overflow.
and