Math operations such as addition, multiplication and shifts can produce a result that is too large to store -- an integer overflow. Depending on the compiler, this integer overflow can result in a sign error, truncating the largest or smallest portion of the result, or another type of error. An integer overflow could lead to a security problem if the overflow affects the value of a pointer that references other code or data in memory. An attack could exploit the integer overflow in a way that allows execution of arbitrary code, resulting in a complete takeover of the vulnerable program's process.
These flaws can be extremely tricky to find and eliminate. The best approach is to use a safe integer class that has been built to avoid these problems. David LeBlanc's column "Integer Handling with the C++ SafeInt Class" provides a detailed mathematical analysis of integer overflows.