Click your avatar, click "Friends", click "Add a Friend", then type your friend's account name in. It needs to be exactly the way they spell it. Once they show up on the list, select them and click "Send a Friend Request".

Once they have accepted you can invite them to custom matches.