Recently I had to do this, and couldn't figure out how to do it (maybe I need to start reading documentation more closely). But I found the way to do it:

type User struct {  
    Id       int64
    Contacts []User `gorm:"foreignkey:user_id;associationforeignkey:contact_id;many2many:user_contacts;"`
}