On the Apple Developer Forums someone asked that very question and got an insightful answer from Chris Lattner:
The way to think about it is that "let" in Swift's current implementation is about physical immutability, not about logical immutability. Physical immutability means that ones the bits are set in memory (i.e. that the let value is initialized) they can never be changed.
Read the whole response here (needs an Apple Developer account).