If you're writing a network server like I am with ZitaFTP, then you may be tempted to create a custom encryption algorithm to store some data. Your reasoning would be: it'll be secure because I'm the only person who knows the algorithm. Good old "security through obscurity."

I'm here to tell you that writing your own algorithm is a very bad idea, and here's why.

Cryptography 101

I bet your first idea for a super secure encryption algorithm would be to use a randomly generated string of numbers as key. Then, you'd add the key to incoming data on a per-byte basis, allowing it to overflow and wrap around. To be more sophisticated you might switch to a subtraction operation for some bytes, and maybe even add an XOR operation here or there.

The encrypted data would be a combination of the input data plus random noise. Perfect! or so you think...

Newsflash! You're not the first person to dream up this algorithm (or some variation thereof). It's crypto 101, and has been done before. Cryptography has been around far longer than modern computers. Such algorithms have been analysed, and possible weaknesses are known.

It's Easy to Reverse Engineer

Reverse engineering such algorithms is easy. Anyone with a debugger can step through the code, and extract the algorithm you used. Once done, flaws can be uncovered an exploited.

That's not a worry if only you and your dog are using the code. However, I assume you'd like others to use your server too. If there's a big enough potential reward (e.g., valuable data to be stolen), then there's a strong incentive for criminals to put in the effort.

Even Experts Struggle to Create Bullet-Proof Algorithms

SSL/TLS is used to secure website traffic (over HTTPS). We don't use the original SSL specification any more, or even the next few revisions. Why? Because flaws have been found in either the encryption algorithms or the protocol

SSL was created by experts in cryptography. Don't think you can get it right over a weekend or two when experts couldn't do so after years of effort.

Algorithm Implementation Is Easy to Screw Up

Even if you invented the best encryption algorithm ever, you could still easily screw up the implementation. It's ridiculously easy.

Remember your randomly generated encryption key? If you used the rand() function, then you've already screwed up. Rand() generates a pseudo-random number. If I know the algorithm and the seed, then I can regenerate your key without any extra info. This is why cryptography libraries have their own random number generators. It's hard for a deterministic machine to generate true randomness.

Here's another simple example. Let's say you stored passwords in plain text, or using your fancy encryption algorithm. Then you use memcmp() to check the password. Congratulations! You just created a side-band leak.

Side-band (a.k.a., side-channel) leaks occur when a code side effect leaks information about the data. Memcmp() is often implemented as a per-byte comparison. It stops when it encounters a byte that's different. So, its execution time is proportional to how many bytes in the password are correct. Very useful information, if you're trying to crack passwords.

Yes, the time difference is small, measured in CPU cycles. However, if you think it's impossible to measure, then have a look at the Spectre attack. Variations in code execution time allowed code to access memory that it wasn't supposed to have access to. It broke memory protection on a lot of CPUs.

This particular side-band leak can be eliminated by using a constant-time compare function.

 

P.S., If you are working on a server, passwords should be stored salted and hashed, using a currently recommended hashing algorithm. Do NOT write your own hash function, or use older ones with known vulnerabilities. Don't know what salting and hashing are? Your homework is to find out. As a server developer you need know this.