Issue
Seems like most people that are having this problem are experiencing the issue on the remote server (i.e. running Git on the remote). I am having this problem at the local level (no AgentForwarding involved).
When I attempt to connect to my server with SSH-Remote, I am immediately prompted with an "Enter your passphrase" text box at the top (followed by a second immediately after I connect to the server). If I open my local terminal:
Load the key/passphrase by running a
ssh user@remote
, entering passphrase, connecting and then exiting.I can confirm
ssh-agent
is running viaps -e | grep ssh
.I can use
ssh-agent
locally--if I sshuser@remote
again, I am NOT prompted for a passphrase, which is the expected behavior.
After all this confirmation, if I open up VSCode and run SSH-Remote to connect to server, I am again immediately prompted for a passphrase (again, twice--once before connection and once after).
I have no extensions installed, this isn't for Git, nor am I trying to use my ssh-agent on the remote. I just don't want to be asked for a passphrase multiple times every single time I open the connection, open a new folder, etc...
Any advice is appreciated. Thanks!
Additional Context:
I am using zsh
as my default shell (its also set as the default shell in VSCode) and the ssh-agent
zsh
plugin to start ssh-agent
and automatically add keys.
This is the content of my ~/.ssh/config
:
AddKeysToAgent yes
Host server
HostName 192.168.1.155
User user
IdentityFile ~/.ssh/id_ed25519
IdentitiesOnly yes
Host github.com
HostName github.com
IdentityFile ~/.ssh/id_ed25519_git
IdentitiesOnly yes
Debug Level 2 Logs
I tried to get as much logging as possible from VSCode's SSH startup process, and I couldn't even find a reference to the use of ssh-agent
or $SSH_AUTH_SOCK
.
[22:26:29.832] stderr> debug1: Found key in /home/user/.ssh/known_hosts:1
[22:26:29.834] stderr> debug1: rekey out after 134217728 blocks
[22:26:29.834] stderr> debug1: SSH2_MSG_NEWKEYS sent
[22:26:29.834] stderr> debug1: expecting SSH2_MSG_NEWKEYS
[22:26:29.834] stderr> debug1: SSH2_MSG_NEWKEYS received
[22:26:29.835] stderr> debug1: rekey in after 134217728 blocks
[22:26:29.835] stderr> debug1: Will attempt key: /home/user/.ssh/id_ed25519 ED25519 XXXXXXXXXXXXXXXXXXXXXXXXXX explicit
[22:26:29.835] stderr> debug1: SSH2_MSG_EXT_INFO received
[22:26:29.835] stderr> debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,[email protected],ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,[email protected],[email protected],ssh-dss,ssh-rsa,rsa-sha2-256,rsa-sha2-512>
[22:26:29.835] stderr> debug1: kex_ext_info_check_ver: [email protected]=<0>
[22:26:29.880] stderr> debug1: SSH2_MSG_SERVICE_ACCEPT received
[22:26:29.892] stderr> debug1: Authentications that can continue: publickey
[22:26:29.892] stderr> debug1: Next authentication method: publickey
[22:26:29.892] stderr> debug1: Offering public key: /home/user/.ssh/id_ed25519 ED25519 XXXXXXXXXXXXXXXXXXXXXXXXXX explicit
[22:26:29.893] stderr> debug1: Server accepts key: /home/user/.ssh/id_ed25519 ED25519 XXXXXXXXXXXXXXXXXXXXXXXXXX explicit
[22:26:29.893] stderr> debug1: read_passphrase: can't open /dev/tty: No such device or address
[22:26:29.956] Got askpass request: {"request":"Enter passphrase for key '/home/user/.ssh/id_ed25519':"}
[22:26:29.956] Detected passphrase message
[22:26:29.957] Listening for interwindow password on /run/user/1000/vscode-ssh-askpass-a712a2dfc9c305b1ae74bbab98be91ebec02eafd.sock
[22:26:29.957] Writing password prompt to globalState
[22:26:31.870] Got passphrase response
[22:26:31.870] Interactor gave response: ********************
[22:26:31.870] Cleaning up other-window auth server
[22:26:32.000] stderr> Authenticated to 192.168.1.120 ([192.168.1.120]:22) using "publickey".
Debug Level 3 Logs
Finally managed to figure out a way to force -vvv
. They do not make it easy for absolutely 0 reason whatsoever. In this block, I found one mention of agent
and it was a log entry saying there isn't one. Not sure how it decides that (which environment variable).
[13:41:11.503] > local-server-1> Running ssh connection command: "-vvv -T -D 33699 -o ConnectTimeout=15 -F /home/user/.ssh/config server bash"
[13:41:11.505] > local-server-1> Spawned ssh, pid=51235
[13:41:11.508] stderr> OpenSSH_9.5p1, OpenSSL 3.1.4 24 Oct 2023
[13:41:11.508] stderr> debug1: Reading configuration data /home/user/.ssh/config
[13:41:11.508] stderr> debug1: /home/user/.ssh/config line 3: Applying options for *
[13:41:11.508] stderr> debug1: /home/user/.ssh/config line 6: Applying options for server
[13:41:11.508] stderr> debug2: resolve_canonicalize: hostname 192.168.1.120 is address
[13:41:11.508] stderr> debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/home/user/.ssh/known_hosts'
[13:41:11.508] stderr> debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/home/user/.ssh/known_hosts2'
[13:41:11.508] stderr> debug3: ssh_connect_direct: entering
[13:41:11.508] stderr> debug1: Connecting to 192.168.1.120 [192.168.1.120] port 22.
[13:41:11.508] stderr> debug3: set_sock_tos: set socket 3 IP_TOS 0x48
[13:41:11.508] stderr> debug2: fd 3 setting O_NONBLOCK
[13:41:11.508] stderr> debug1: fd 3 clearing O_NONBLOCK
[13:41:11.508] stderr> debug1: Connection established.
[13:41:11.508] stderr> debug3: timeout: 15000 ms remain after connect
[13:41:11.508] stderr> debug1: identity file /home/user/.ssh/id_ed25519 type 3
[13:41:11.508] stderr> debug1: identity file /home/user/.ssh/id_ed25519-cert type -1
[13:41:11.508] stderr> debug1: Local version string SSH-2.0-OpenSSH_9.5
[13:41:11.519] stderr> debug1: Remote protocol version 2.0, remote software version OpenSSH_9.2p1 Debian-2+deb12u1
[13:41:11.519] stderr> debug1: compat_banner: match: OpenSSH_9.2p1 Debian-2+deb12u1 pat OpenSSH* compat 0x04000000
[13:41:11.519] stderr> debug2: fd 3 setting O_NONBLOCK
[13:41:11.519] stderr> debug1: Authenticating to 192.168.1.120:22 as 'ssh-user'
[13:41:11.519] stderr> debug3: record_hostkey: found key type ED25519 in file /home/user/.ssh/known_hosts:1
[13:41:11.519] stderr> debug3: record_hostkey: found key type RSA in file /home/user/.ssh/known_hosts:2
[13:41:11.519] stderr> debug3: record_hostkey: found key type ECDSA in file /home/user/.ssh/known_hosts:3
[13:41:11.519] stderr> debug3: load_hostkeys_file: loaded 3 keys from 192.168.1.120
[13:41:11.519] stderr> debug1: load_hostkeys: fopen /home/user/.ssh/known_hosts2: No such file or directory
[13:41:11.519] stderr> debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
[13:41:11.519] stderr> debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
[13:41:11.519] stderr> debug3: order_hostkeyalgs: have matching best-preference key type [email protected], using HostkeyAlgorithms verbatim
[13:41:11.519] stderr> debug3: send packet: type 20
[13:41:11.519] stderr> debug1: SSH2_MSG_KEXINIT sent
[13:41:11.520] stderr> debug3: receive packet: type 20
[13:41:11.520] stderr> debug1: SSH2_MSG_KEXINIT received
[13:41:11.520] stderr> debug2: local client KEXINIT proposal
[13:41:11.520] stderr> debug2: KEX algorithms: [email protected],curve25519-sha256,[email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,ext-info-c
[13:41:11.520] stderr> debug2: host key algorithms: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,[email protected],[email protected],rsa-sha2-512,rsa-sha2-256
[13:41:11.520] stderr> debug2: ciphers ctos: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected]
[13:41:11.520] stderr> debug2: ciphers stoc: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected]
[13:41:11.520] stderr> debug2: MACs ctos: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1
[13:41:11.520] stderr> debug2: MACs stoc: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1
[13:41:11.520] stderr> debug2: compression ctos: none,[email protected],zlib
[13:41:11.520] stderr> debug2: compression stoc: none,[email protected],zlib
[13:41:11.520] stderr> debug2: languages ctos:
[13:41:11.520] stderr> debug2: languages stoc:
[13:41:11.520] stderr> debug2: first_kex_follows 0
[13:41:11.520] stderr> debug2: reserved 0
[13:41:11.520] stderr> debug2: peer server KEXINIT proposal
[13:41:11.520] stderr> debug2: KEX algorithms: [email protected],ecdh-sha2-nistp521,ecdh-sha2-nistp384,ecdh-sha2-nistp256,diffie-hellman-group-exchange-sha256
[13:41:11.520] stderr> debug2: host key algorithms: rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519
[13:41:11.520] stderr> debug2: ciphers ctos: [email protected],[email protected],[email protected],aes256-ctr,aes192-ctr,aes128-ctr
[13:41:11.520] stderr> debug2: ciphers stoc: [email protected],[email protected],[email protected],aes256-ctr,aes192-ctr,aes128-ctr
[13:41:11.520] stderr> debug2: MACs ctos: [email protected],[email protected],hmac-sha2-512,hmac-sha2-256,[email protected]
[13:41:11.520] stderr> debug2: MACs stoc: [email protected],[email protected],hmac-sha2-512,hmac-sha2-256,[email protected]
[13:41:11.520] stderr> debug2: compression ctos: none
[13:41:11.520] stderr> debug2: compression stoc: none
[13:41:11.521] stderr> debug2: languages ctos:
[13:41:11.521] stderr> debug2: languages stoc:
[13:41:11.521] stderr> debug2: first_kex_follows 0
[13:41:11.521] stderr> debug2: reserved 0
[13:41:11.521] stderr> debug1: kex: algorithm: [email protected]
[13:41:11.521] stderr> debug1: kex: host key algorithm: ssh-ed25519
[13:41:11.521] stderr> debug1: kex: server->client cipher: [email protected] MAC: <implicit> compression: none
[13:41:11.521] stderr> debug1: kex: client->server cipher: [email protected] MAC: <implicit> compression: none
[13:41:11.521] stderr> debug3: send packet: type 30
[13:41:11.521] stderr> debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
[13:41:11.526] stderr> debug3: receive packet: type 31
[13:41:11.526] stderr> debug1: SSH2_MSG_KEX_ECDH_REPLY received
[13:41:11.526] stderr> debug1: Server host key: ssh-ed25519 SHA256:+wYnNtg/GzeBjkCVw2VW+1KuZHq1808nstVdkZh1+bU
[13:41:11.526] stderr> debug3: record_hostkey: found key type ED25519 in file /home/user/.ssh/known_hosts:1
[13:41:11.526] stderr> debug3: record_hostkey: found key type RSA in file /home/user/.ssh/known_hosts:2
[13:41:11.526] stderr> debug3: record_hostkey: found key type ECDSA in file /home/user/.ssh/known_hosts:3
[13:41:11.526] stderr> debug3: load_hostkeys_file: loaded 3 keys from 192.168.1.120
[13:41:11.526] stderr> debug1: load_hostkeys: fopen /home/user/.ssh/known_hosts2: No such file or directory
[13:41:11.526] stderr> debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
[13:41:11.526] stderr> debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
[13:41:11.526] stderr> debug1: Host '192.168.1.120' is known and matches the ED25519 host key.
[13:41:11.526] stderr> debug1: Found key in /home/user/.ssh/known_hosts:1
[13:41:11.528] stderr> debug3: send packet: type 21
[13:41:11.528] stderr> debug2: ssh_set_newkeys: mode 1
[13:41:11.528] stderr> debug1: rekey out after 134217728 blocks
[13:41:11.528] stderr> debug1: SSH2_MSG_NEWKEYS sent
[13:41:11.528] stderr> debug1: expecting SSH2_MSG_NEWKEYS
[13:41:11.528] stderr> debug3: receive packet: type 21
[13:41:11.528] stderr> debug1: SSH2_MSG_NEWKEYS received
[13:41:11.528] stderr> debug2: ssh_set_newkeys: mode 0
[13:41:11.528] stderr> debug1: rekey in after 134217728 blocks
[13:41:11.528] stderr> debug1: Will attempt key: /home/user/.ssh/id_ed25519 ED25519 SHA256:XXXXXXXXXXXXXXXXXXXXXXXXX explicit
[13:41:11.528] stderr> debug2: pubkey_prepare: done
[13:41:11.528] stderr> debug3: send packet: type 5
[13:41:11.528] stderr> debug3: receive packet: type 7
[13:41:11.528] stderr> debug1: SSH2_MSG_EXT_INFO received
[13:41:11.528] stderr> debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,[email protected],ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,[email protected],[email protected],ssh-dss,ssh-rsa,rsa-sha2-256,rsa-sha2-512>
[13:41:11.528] stderr> debug1: kex_ext_info_check_ver: [email protected]=<0>
[13:41:11.575] stderr> debug3: receive packet: type 6
[13:41:11.576] stderr> debug2: service_accept: ssh-userauth
[13:41:11.576] stderr> debug1: SSH2_MSG_SERVICE_ACCEPT received
[13:41:11.576] stderr> debug3: send packet: type 50
[13:41:11.599] stderr> debug3: receive packet: type 51
[13:41:11.599] stderr> debug1: Authentications that can continue: publickey
[13:41:11.599] stderr> debug3: start over, passed a different list publickey
[13:41:11.599] stderr> debug3: preferred publickey,keyboard-interactive,password
[13:41:11.599] stderr> debug3: authmethod_lookup publickey
[13:41:11.599] stderr> debug3: remaining preferred: keyboard-interactive,password
[13:41:11.599] stderr> debug3: authmethod_is_enabled publickey
[13:41:11.599] stderr> debug1: Next authentication method: publickey
[13:41:11.599] stderr> debug1: Offering public key: /home/user/.ssh/id_ed25519 ED25519 SHA256:XXXXXXXXXXXXXXXXXXXXXXXXX explicit
[13:41:11.599] stderr> debug3: send packet: type 50
[13:41:11.599] stderr> debug2: we sent a publickey packet, wait for reply
[13:41:11.599] stderr> debug3: receive packet: type 60
[13:41:11.599] stderr> debug1: Server accepts key: /home/user/.ssh/id_ed25519 ED25519 SHA256:XXXXXXXXXXXXXXXXXXXXXXXXX explicit
[13:41:11.600] stderr> debug3: sign_and_send_pubkey: using [email protected] with ED25519 SHA256:XXXXXXXXXXXXXXXXXXXXXXXXX
[13:41:11.600] stderr> debug3: sign_and_send_pubkey: signing using ssh-ed25519 SHA256:XXXXXXXXXXXXXXXXXXXXXXXXX
[13:41:11.600] stderr> debug1: read_passphrase: can't open /dev/tty: No such device or address
[13:41:11.663] Got askpass request: {"request":"Enter passphrase for key '/home/user/.ssh/id_ed25519':"}
[13:41:11.663] Detected passphrase message
[13:41:11.663] Listening for interwindow password on /run/user/1000/vscode-ssh-askpass-3c669b661bf05bd38516123cc0c4977123a4eed3.sock
[13:41:11.663] Writing password prompt to globalState
[13:41:23.598] Got passphrase response
[13:41:23.599] Interactor gave response: ********************
[13:41:23.599] Cleaning up other-window auth server
[13:41:23.724] stderr> debug3: no authentication agent, not adding key
[13:41:23.724] stderr> debug3: send packet: type 50
[13:41:23.731] stderr> debug3: receive packet: type 52
[13:41:23.731] stderr> Authenticated to 192.168.1.120 ([192.168.1.120]:22) using "publickey".
At this point, the only thing I can imagine, given that the local-server can't find the agent, is that (despite setting the shell to ZSH in VSCode settings), Remote SSH is still using bash
or sh
, and because of that, has no access to the environment variables.
Not sure how I can tell if this is the case. Even more unsure of how I would change that.
Anywhere else I should look for logs to diagnose the root cause of this issue?
Update
Probably will be my last update, since I found a somewhat workable solution, but I would like to still understand why this is giving me trouble and fix it if possible.
If I make sure that no other VSCode instances are running, and run code
from the command line, that specific VSCode instance can interface with the ssh-agent
and doesn't ask for passphrases. If I run it from KRunner (on Arch Linux KDE), it can not. Not sure if this is a permissions or environment variable thing at this point.
Given this new information, any ideas on next step troubleshooting? Thanks again.
Solution
TL;DR
Resolved. I was using the oh-my-zsh ssh-agent
plugin which initialized and exported the ssh-agent
environment variables for individual interactive shell instances on shell startup, since it was sourced in .zshrc
. VSCode can't access that if it is run via xdg-open
/kde-open
. Running it in the console actually allowed it to work which was the hint I needed to resolve the issue.
The important lesson is understanding the differences between interactive, non-interactive, and login shells and how environment variables may or may not be set in one context or another. Depending on the functionality you are aiming for, you should be using one of (not exhaustive): .zshrc
, .zprofile
, OR .bashrc
, .bash_profile
. I recommend you initialize your ssh-agent
in a profile
file for your respective shell IF you plan to use ssh
from beyond the shell. That way, even those programs can access the necessary env variables.
Solution Details
It was somewhat disappointing. Heres the solution plus reasoning and sources in case others run into this:
- If I have NO other instances of
code
open and opened it via the console (code
) instead of thecode.desktop
, it actually worked with thessh-agent
! I had tried this before, but I suspect it was done incorrectly for two reasons:
- I likely had another instance of
code
open, and seemingly even if I runcode
from the terminal, if I had a running instance started from the .desktop, it wouldn't work. My guess is that consecutive calls tocode
trigger theNew Window
functionality instead of allowing another instance completely. This makes sense as under the process tree, there remains only one parentcode
instance. - Other threads on the topic advised that if the VSCode terminal had access to the environment variables (i.e. if
echo $SSH_AUTH_SOCK
yielded correct results from INSIDE the VSCode terminal) then the entire instance of VSCode also had access to those variables. This is not true. I got duped by this and overlooked the issue of environment variables after trying it, hence why this took me so long.
This meant it must have been an environment variable issue. To confirm, I added to the
.desktop
the currentSSH_AUTH_SOCK
andSSH_AGENT_PID
env variables. Lo and behold, it worked.Quick Google search regarding why the terminal might have access to env variables and not
xdg-open
(or possiblykde-open
in my case) yielded this thread. In my case, I am using theoh-my-zsh ssh-agent
plugin that is started in.zshrc
--which is only run for interactive terminals, on each terminal instance startup. My guess was thatxdg-open
/kde-open
are not interactive terminals, as they only run theExec
portion of the .desktop with no user input. Thus, the plugin wasn't starting and the variables were not being exported for the VSCode startup. Rather, if I runcode
at the terminal, since it was born from that terminal, it inherits the environment variables the terminal initialized which is why it was working in that context. Ugh.More research pointed to moving
ssh-agent
initialization out to.zprofile
. So, I moved thessh-agent
initialization out to$HOME/.zprofile
and made an equivalent initialization (checks ifssh-agent
is running first before so they don't each start their own instance) in$HOME/.bash_profile
.It worked at last.
This was the result of ~4 hours of debugging/researching each day for the last 4 or 5 days, so kinda pathetic it was a relatively simple issue lol.
Answered By - Sa'id Kharboutli Answer Checked By - David Marino (WPSolving Volunteer)