On Friday, 17 January 2014 at 14:06:57 UTC, Kagamin wrote:
> Doesn't TCP take care of that?

For a packet, yes. In general you can assume that if a transfer completes under TCP then it is very likely correct. That's the way TCP is designed. If you want to check the entire file at the end, MD5 could theoretically be done as a sanity check. I'd still use SHA-2 (in the form of a digital signature, obviously) minimally if your intention is to ensure it hasn't been tampered with, but the speed, simplicity, and availability of MD5 can be arguments for its use as long as you're not actually concerned about security.

On Friday, 17 January 2014 at 14:04:29 UTC, Kagamin wrote:
> There's no successful preimage attack on MD5, which is the only deadly attack on a hash function.

For one amusing example of why MD5 is broken (from years ago using a video game console, of all things), consider:
http://www.win.tue.nl/hashclash/Nostradamus/

Here, let me use an analogy. Note, this analogy comes from me so it's not necessarily endorsed by any professional cryptographers... but it explains where my concerns are coming from.

Let's say that everyone has something very precious ... like a diamond. Let's imagine a multiverse. Each universe can "represent" an algorithm. One is for MD5, another is for SHA1 and so on. In each universe, there exists a planet that holds a vault that has your diamond. Some universes are similar to others (for instance, the SHA1 universe has some passing resemblance to the MD5 universe).

When you say "specialized hardware can be made to speed up the process", I think "I can visit ten planets in this universe looking for the vaults instead of just one at a time if I invest in a special ship". Can you see why this doesn't concern me in the slightest?

When you say "Moore's law", I think "every 18 months I can know which half of the remaining universe contains the diamonds". Again, can you see why this doesn't concern me?

The universe is _just too big_ for this stuff to matter. If you can't find the vaults containing the diamonds, then these things don't bother me nor should it bother you.

For the MD5 universe, people have found a way of discovering the planet the diamonds are on very quickly (something like being able to find which millionth of the remaining universe contains the diamond each attempt ... a very fast solution). They've overcome the challenge of finding the diamonds (collision resistance) and now we wonder who is going to invent the blowtorch capable of opening the vaults up (preimage attack). All any cryptographer is saying is move the diamonds you care about into another universe before the MD5 blowtorch is invented. It may never get invented (we don't have any idea whether it's possible or if it is possible, when it might materialize) but moving it into another universe where the diamonds haven't even been found yet is simply safer. Ideally you don't want a universe that is too similar (such as MD5 and SHA1) because it might make the planet easier to be found. Also note that despite their similarities, no planets with diamonds have been found in the SHA1 universe. Yet still cryptographers are recommending moving on to SHA2 for new projects. Just food for thought.

Plus knowing which planet the diamond is on is dangerous enough since it can be used in certain circumstances to do some nefareous things (already showed the example where someone can lie about a prediction for the future). It's only in special circumstances, but ignoring subtleties isn't recommended in cryptography since people almost always figure out a way to use things incorrectly.

To summarize, do not use MD5 intending it to be secure against attackers. Just don't. I'd also like to say that you really shouldn't suggest to anyone that it's a reasonable approach either. Obviously, I can't force you to, but do know that generally people (not just me) are going to argue against you on this, so you can save yourself a lot of trouble by just never mentioning MD5 as being acceptable again. At this point I think there's been enough discussion on the matter that exceptionally few will wander into this topic and be mislead into accepting MD5 for security purposes, so I'm going to retire from this discussion. If you use MD5 for its "security", just remember this for later: You were warned™. (Also, remember that people have been saying "Don't use MD5" since 1996, so when you do get bit by it, not only "You were warned" but "you are 20 years out of date, find a new job because you're fired" is also likely)

On Friday, 17 January 2014 at 14:53:04 UTC, Chris Cain wrote:
> Your first part is true, but the second part is not. A generic collision attack can cause issues depending on what you're trying to do. For general use, MD5 is not safe enough based on that alone.

No technology is safe, and SHA3 doesn't solve this problem. Cryptography is particularly not fool-proof. Remember issues in RSA? The factorization math behind it is bulletproof, yet, it's still considered hackable.

> The wikipedia article on MD5 covers this reasonably well enough:
> http://en.wikipedia.org/wiki/MD5
>
> Do note that SHA1 is similar enough to MD5 that many are recommending against SHA1 as well (and I recommend following that advice). It's not as broken as MD5 but since we have SHA2 there's no good reason not to use the better version.
>
> For general messages requiring any degree of security against an attacker, use SHA2 (this does not necessarily apply to hashing passwords which have their own concerns, obviously). Stay away from MD5 and no new projects should use SHA1 (although my understanding is that no one is saying "jump from the ship" for SHA1 right now). These are generally accepted practices in the cryptography community. If you want to roll the dice with MD5, by all means, but don't spread acceptance of MD5 because someone might follow your advice against the clear recommendations of the cryptography community. As far as I'm concerned, it's unethical to recommend MD5 for security purposes at this point.

Ethics don't help in cryptography, only real factors play role.
Your insistence on blind following to recommendations and expecting them to magically protect security - by encouraging thoughtless behavior such voodooism is way more troublesome.

>> SHA3 is just more convenient than MD5 because when you want to change the hash function, you don't have to ditch the whole system, only change its parameters.
>
> I'm sorry but no. The reason we wanted a SHA3 is because we needed a message digest algorithm that is different enough from MD5, SHA1, and SHA2 so that an attack discovered on those algorithms should not be applicable to SHA3 as well (as was the case with MD5 and SHA1). The things you describe are just bonuses.

This works both ways: an attack on SHA3 doesn't affect other hash functions.
The older algorithms are more trustworthy exactly because they are well studied, have no dark corners and evolve gradually by fixing discovered weaknesses.

>> That's rather inconvenient, that you can't have an efficient implementation of the algorithm on common hardware. MD5 family has no such flaw.
>
> You seem to not know enough about this topic to discuss it. SHA3 is fast on common hardware, as it is designed. Specialized hardware can be made and that's fine, intentional, and even beneficial. Most cryptography is designed that way. DES was, AES is (AES instructions exist on many modern hardware, in fact), and SHA3 is as well. This is not a flaw and if you think it is, go talk with cryptographers instead of me because you're clearly well beyond my skill level (that said, since you're discussing the viability of MD5, you're not anywhere close to as competent on this matter as you think you are, no offense).

It's even worse that new algorithms force hardware vendors to add specialized instructions which are of no use for other software.

On Friday, 17 January 2014 at 15:13:27 UTC, Chris Cain wrote:
> On Friday, 17 January 2014 at 14:04:29 UTC, Kagamin wrote:
>> There's no successful preimage attack on MD5, which is the only deadly attack on a hash function.
>
> For one amusing example of why MD5 is broken (from years ago using a video game console, of all things), consider:
> http://www.win.tue.nl/hashclash/Nostradamus/

They didn't predict the result of elections, not sure what you want to prove by a lie.

On Friday, 17 January 2014 at 15:00:38 UTC, Chris Cain wrote:
> On Friday, 17 January 2014 at 14:06:57 UTC, Kagamin wrote:
>> Doesn't TCP take care of that?
>
> For a packet, yes. In general you can assume that if a transfer completes under TCP then it is very likely correct. That's the way TCP is designed. If you want to check the entire file at the end, MD5 could theoretically be done as a sanity check. I'd still use SHA-2 (in the form of a digital signature, obviously) minimally if your intention is to ensure it hasn't been tampered with

It's hard and expensive to use digital signatures in public projects and doesn't protect from tampering. In fact, direct tampering in such setup is cheaper than a collision attack, not even speaking, that a collision attack doesn't work here, only a preimage attack.

On Saturday, 18 January 2014 at 12:48:28 UTC, Kagamin wrote:
> ..snip almost everything..

I'm not going to be doing this. Everything you say suggests you have no clue about anything about cryptography and you're going to be bit sooner rather than later. Don't spread your information, just keep it to yourself so that the damage is limited. Thanks.

> They didn't predict the result of elections, not sure what you want to prove by a lie.

You're correct. It was an example of a collision attack which means they can put up a single hash and it will match many possible documents that include all the names of the candidates. If you had _read_ the source, you may have understood that. In fact, if you have read anything on cryptography at all before, you'd be staying away from MD5, but ignorance is bliss I suppose. Just don't spread your ignorance to others because you could easily cost people their livelihood. If you lose your job because of this, then that's on you, so I won't feel badly about it. But I am concerned about an innocent third party walking in thinking you might have a clue of what you're talking about, when you clearly don't.

Take care.

On Saturday, 18 January 2014 at 15:17:52 UTC, Chris Cain wrote:
> You're correct. It was an example of a collision attack which means they can put up a single hash and it will match many possible documents that include all the names of the candidates.

Consequence of a 2-document collision is big enough, that a 12-document collision doesn't pose any notably bigger threat. Nostradamus is all they could come up with, which is laughable. If they upgrade this to an actual partial 3.6-bit preimage attack, and one collision takes 2 days as they say, for a comparison, with such partial preimage attack, full preimage attack on a 32-bit hash like crc32 will take 2^^(32-3.6)*2 days = 1959870 years in an ideal case.

> If you had _read_ the source, you may have understood that. In fact, if you have read anything on cryptography at all before, you'd be staying away from MD5, but ignorance is bliss I suppose.

Isn't it you, who insist on ignorance to how a collision attack works and how it doesn't work? You insist on a magical approach to cryptography, that MD5 magic doesn't work and SHA3 magic works, but you should know that magic is a delusion, and delusion leads to failures and damage, so by spreading delusions, it's you who cause damage, not me.

On Sunday, 19 January 2014 at 15:09:46 UTC, Kagamin wrote:
> Isn't it you, who insist on ignorance to how a collision attack works and how it doesn't work? You insist on a magical approach to cryptography, that MD5 magic doesn't work and SHA3 magic works, but you should know that magic is a delusion, and delusion leads to failures and damage, so by spreading delusions, it's you who cause damage, not me.

Excuse me? Straw men arguments don't work here. Take that crap elsewhere. You keep asserting I say things that I don't say like crap like "You assume that Moores law doesn't work". Get out of here with that.

The fact that MD5 is weak against a collision attack means that it strictly provides weaker guarantees than the stronger SHA1, SHA2, and SHA3. No magic is necessary to know that with cryptography, attackers look for the weakest point in a chain to attack. Intentionally using something that provides weaker guarantees is foolish. And indeed a collision attack can cause problems, depending on what you want to do. I can't predict (nor can you) where some one will use a hash function expecting it to be secure, so the right thing to do is just suggest people use the stronger hash functions.

Furthermore, when talking about anything related to cryptography, we discuss things in terms of how much of a safety buffer we get. You're right that nothing is "perfectly safe" but when our estimations of safety suggest it should remain safe for at least 20 years, then we're fine with it. That's why we don't use MD5. Because it has known flaws AND because it no longer has the "shield" of collision resistance. For a hash function, if it is "collision resistant" that strictly means that no pre image attack exists either. You'd have to break through two walls to break the hash function. With MD5, there is no such safety buffer. It's one discovery away from being destroyed. Ergo, don't use it because it can turn from "no pre image attacks exist" to "here's the pre image attack" overnight.

Preempting your counterargument: Indeed such a thing can happen with SHA2, but the the likelihood of such a thing is essentially non existent. If we haven't even discovered a single collision ever, we don't find the likelihood of generating collisions at will likely nor do we find the likelihood of generating a specific collision likely either.

Finally, I have never suggested anyone work off of "this stuff is magic so be ignorant of it while working with it". Quite the opposite. You should be well educated with this stuff prior to working with it or else you risk exposing confidential information which can be quite damaging. Part of this is _don't deliberately ignore the recommendations of cryptography experts_. What you suggest people do, "despite being a non-expert, I suggest you do use MD5 because those experts clearly don't know what they're talking about" is dangerous. And stop with the straw men attacks because I'm sick of it. Actually, I'm sick of all of your crap. You can have the last word and if it's "ignore the cryptography experts because I'm smarter than them", then so be it. You might want to also let your employer know as well so they can give you a raise *wink*.

On Sunday, 19 January 2014 at 15:09:46 UTC, Kagamin wrote:
> Consequence of a 2-document collision is big enough, that a 12-document collision doesn't pose any notably bigger threat. Nostradamus is all they could come up with, which is laughable. If they upgrade this to an actual partial 3.6-bit preimage attack, and one collision takes 2 days as they say, for a comparison, with such partial preimage attack, full preimage attack on a 32-bit hash like crc32 will take 2^^(32-3.6)*2 days = 1959870 years in an ideal case.

By the way, I didn't respond to this, but I suppose I should.

https://code.google.com/p/tweakcrc/
http://blog.fortinet.com/trash-crc32/

Hence why you should keep your butt out of cryptography discussions until you take any class on it at the very least. Your calculation is garbage and shows how you haven't a single clue about what you are talking about.

Again, you can have the last word, but I'm certain I'm done here regardless. If you can't see the fact that you aren't smarter than the experts by now (with the above) then I don't think I can convince you with my words. Experience is the best teacher sometimes. I just hope that I never accidentally use a program you write expecting my data to be secure.

One more thing, if you're curious about taking your first step into actually learning the basics of cryptography, you can use this:
https://www.coursera.org/courses?search=cryptography

Dan Boneh is a well respected expert in cryptography and his class is very well structured (it wasn't the first one I took, but it's definitely comparable to an actual college course on cryptography). Do not take it expecting that you'll even be remotely capable of going against the recommendations by experts by the end (in fact, you'll learn that such things is idiocy if you pay any attention to it).

If you do take it, do the assignments and homework. Despite such things having the reputation of being just a bother, the assignments are necessary to fully understand the material for advanced topics like this. They are tough but they'll teach you pretty quickly that looking at, for instance, the number of bits in crc32 is meaningless for creating attacks.