I started by generating the SSH key-pair and transferring the public key to the server. This is quite simple, so I won't go into details, except to provide a link to the ArchWiki article I used as reference.
Next, edit the file /etc/ssh/sshd_config on the server side, and make sure that PubkeyAuthentication is set to Yes.
If you want to make things even more secure, you might want to think about disabling password login completely. Just remember, if you do that, and you loose you private key file, you won't be able to access the server remotely anymore.
After a restart of the ssh daemon on the server, it should now be possible to login without being prompted for a password. This was not the case however...
The problem turns out to be that SSH is a bit cranky when it comes to file permissions. After a fair bit of googling, it turns out that not only does this apply to the authorized_keys files, and the .ssh directory, but also to the $home directory used (this last one caused me a bit of trouble).
Here are the commands for setting the correct file permissions:
chmod go-w ~/Once this was done, I was able to access the server without providing a password, both for SSH and SFTP.
chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys
Yeay, no more Samba for me!