Cryptography is very complicated, difficult to understand, and perilously easy to get wrong. The #1 rule of cryptography is “Don't invent your own.”
But even if you use existing components, there are myriad ways you can introduce subtle bugs that make your application dramatically less secure. Here are some of the ways — although this is far from complete!
Mistake #1 — Improper or inadequate seeding of the random number generator
In 2006, a bug was introduced to Debian (and Ubuntu) that broke the seeding of the random number generator. With the bug, the only “random” number that debian-based operating systems were using was the process ID. However, on Linux the maximum process ID is 32,768 — so only 32,768 possible keys were getting generated. That's easily crackable. (Check out the vulnerability report for more information.)
Mistake #2 — Using the MD5 hashing algorithm
We've known MD5 to be insecure since 1996.
The security of the MD5 hash function is severely compromised. A collision attack exists that can find collisions within seconds on a computer with a 2.6 GHz Pentium 4 processor (complexity of 224.1). Further, there is also a chosen-prefix collision attack that can produce a collision for two chosen arbitrarily different inputs within hours, using off-the-shelf computing hardware (complexity 239). The ability to find collisions has been greatly aided by the use of off-the-shelf GPUs. On an NVIDIA GeForce 8400GS graphics processor, 16–18 million hashes per second can be computed. An NVIDIA GeForce 8800 Ultra can calculate more than 200 million hashes per second.
Mistake #3 — Storing SHA1(password) in the database
For passwords, it's better to use a unique salt for each account. That is, instead of storing SHA1(password) in the database, store SHA1(salt + password). Without a salt, an attacker who has access to the password hashes can hash the first word in his password dictionary, check whether the calculated hash is in his sorted list of stolen hashes, and probably get access to multiple accounts at once. On the other hand, if the passwords have been salted before storage, the hacker needs to attack each account separately. In 2012, LinkedIn got hacked. The hashes published by LinkedIn did not contain salts, and more than 60% of the passwords were cracked.
Mistake #4 — Length extension vulnerability
Several older hashing algorithms, including SHA1, are vulnerable to a length extension attack. If you're using SHA1 for authenticating your API, that's going to be a problem. An app is vulnerable to a hash length extension attack if it prepends a secret value to a string, hashes that (with a vulnerable algorithm such as SHA1), and makes the string and the hash available to a would-be attacker. A hacker can attack such an authentication scheme by taking the original hashed string, adding in whatever values he wants to (e.g., to transfer money) and then continuing the hashing. The hacker will then have a new hash (based off the still-unknown prefix) that will authenticate him and perform whatever action he wants. Flickr's API had this vulnerability in 2009.
Mistake #5 — Using DES
Don't use Data Encryption Standard (DES). It was designed with an effective key length of 56 bits, which is vulnerable to exhaustive search. Instead, use Advance Encryption Standard (AES).
Mistake #6 — Using ECB mode
Electronic Codebook (ECB) mode encryption basically loops over the plaintext one block at a time, encoding one block at a time. However, it's really bad at hiding data patterns (let's face it, most data has patterns) — especially images and BLOBs.
Instead of ECB, use Cipher-block chaining (CBC) mode.
Mistake #7 — Using the same keystream to encrypt two different documents
Microsoft has made this mistake more than once. In 2005, Hongjun Wu found a flaw in Microsoft Word and Excel: When an encrypted document gets modified and saved, the initialization vector (IV) remained the same, so that the same keystream was used to encrypt both the original version and the modified version of the document. That makes it easy to recover a lot of the information in the document.
Mistake #8 — Not verifying the hostname of the SSL certificate
Verify that the hostname on the SSL certificate is the hostname you're expecting. There was actually a bug earlier this year where ruby was not properly checking hostnames in certificates that contained null bytes, allowing hackers to spoof websites and conduct man-in-the-middle attacks.
Mistake #9 — Not using certificate pinning
At the time of writing, Mozilla has 166 certificate authorities that it trusts by default. (Source.) Unfortunately, unless you've set up certificate pinning, your app is only as secure as the weakest of all the trusted root certificate authorities.
Let's say you have an app with an SSL certificate signed by Verisign. No problem, right? WRONG. If any one of those 166 certificate authorities gets hacked, then the hackers can issue new certificates for your domain and impersonate you. In fact, that's exactly what happened in 2011 when DigiNotar was attacked. Iranian hackers issued a wildcard certificate for Google, then conducted man-in-the-middle attacks against Google services.
To protect your app from this sort of attack, you can use certificate pinning to associate your SSL certificate with a specific certificate authority. That way, if a random sketchy certificate authority gets hacked, your site is still secure. You only run into trouble if your specific certificate authority gets hacked. You can read more about SSL pinning here.
Mistake #10 — Not testing your SSL configuration
Once you've configured your web server, you can use Qualys's SSL Labs to run a free online test of the configuration of your SSL web server. You can check SSL server configuration, certificate chain, and protocol and cipher suite support. You can also check for known vulnerabilities such as the renegotiation vulnerability.