Forticlient VPN on GNU/Linux: Blank screen

Preamble

On a Debian GNU/Linux 11 Bullseye box, with the latest updates and with a working NVIDIA graphics card, the Forticlient GUI binary showed a blank-screen with no widgets in it:

The widget-rendering is not working

We did not have Forticlient source code, so we needed to resort to conventional reversing & debugging techniques in order to know more about this binary so we could pinpoint where the issue was. First of all, we ran the binary on a working GNU/Linux box to see what happened when the client worked as expected:

/opt/forticlient/gui/FortiClient-linux-x64/FortiClient
xdg-settings: default-url-scheme-handler not implemented for xfce
Platform detected: fedora 1
08:52:39.134 › Platform detected: fedora
[ ‘/opt/forticlient/gui/FortiClient-linux-x64/FortiClient’ ] 1
08:52:39.138 › [ ‘/opt/forticlient/gui/FortiClient-linux-x64/FortiClient’ ]
Saml – init 1
08:52:39.196 › Saml – init
ready-to-show 1
08:52:39.538 › ready-to-show
compliance configDir=/home/USER/.config/FortiClient/config 1
08:52:39.841 › compliance configDir=/home/USER/.config/FortiClient/config
did-finish-load 1
08:52:39.845 › did-finish-load
Events – IPC_RENDERER_REQUEST.FETCH_INVITATION_CODE inviteCode=null 1
08:52:40.725 › Events – IPC_RENDERER_REQUEST.FETCH_INVITATION_CODE inviteCode=null
IPC_RENDERER_REQUEST.LOADED 1
08:52:40.770 › IPC_RENDERER_REQUEST.LOADED
Events – processArgv [“/opt/forticlient/gui/FortiClient-linux-x64/FortiClient”] 1
08:52:40.770 › Events – processArgv [“/opt/forticlient/gui/FortiClient-linux-x64/FortiClient”]

On the GNU/Linux box with the issue, we got this instead:

/opt/forticlient/gui/FortiClient-linux-x64/FortiClient
xdg-settings: default-url-scheme-handler not implemented for xfce
Platform detected: fedora 1
08:52:39.134 › Platform detected: fedora
[ ‘/opt/forticlient/gui/FortiClient-linux-x64/FortiClient’ ] 1
08:52:39.138 › [ ‘/opt/forticlient/gui/FortiClient-linux-x64/FortiClient’ ]
Saml – init 1
08:52:39.196 › Saml – init
ready-to-show 1
08:52:39.538 › ready-to-show
compliance configDir=/home/USER/.config/FortiClient/config 1
08:52:39.841 › compliance configDir=/home/USER/.config/FortiClient/config
did-finish-load 1
08:52:39.845 › did-finish-load

So, apparently no event “IPC_RENDERER_REQUEST.FETCH_INVITATION_CODE“, and the window showed no widgets inside at all. At this point, it was crystal clear that this binary was using some sort of “rendering” engine in order to show all the widgets (text-boxes, radio-buttons, labels, whatever) within its main window. Because we were not getting this event at all, no widgets were rendered and so the blank screen!

ElectronJS

We googled “IPC_RENDERER_REQUEST” because we though this was a very good place to start. We got this:

Our search leads us to the ElectronJS framework

Okay, so Forticlient was using the ElectronJS framework to render its widgets. That was great, because now we could read ElectronJS documentation in order to determine how we could get some debugging information from it.

So, armed with this new option “–enable-logging”, we could re-run the program and see what we got:

[6297:1026/145315.375104:INFO:CONSOLE(20)] “WebSocket connection to ‘ws://127.0.0.1:36527/websocket’ failed: Connection closed before receiving a handshake response”, source: file:///home/opt/forticlient/gui/FortiClient-linux-x64/resources/app.asar/assets/js/bundle.min.js (20)
[6297:1026/145315.375274:INFO:CONSOLE(25)] “Error: [object Event]”, source: file:///home/opt/forticlient/gui/FortiClient-linux-x64/resources/app.asar/assets/js/bundle.min.js (25)
[6297:1026/145315.375459:INFO:CONSOLE(25)] “Closed! [object CloseEvent]”, source: file:///home/opt/forticlient/gui/FortiClient-linux-x64/resources/app.asar/assets/js/bundle.min.js (25)
[6297:1026/145315.383580:INFO:CONSOLE(20)] “TypeError: Failed to fetch”, source: file:///home/opt/forticlient/gui/FortiClient-linux-x64/resources/app.asar/assets/js/bundle.min.js (20)

So the client tried to connect via websockets to a server running locally, it managed to establish the connection and then the local web server closed it immediately, yielding the error message in red.

Webserver confighandler

We used systemctl to determine which binaries were executed by the forticlient service:

systemctl status forticlient
● forticlient.service – Forticlient Scheduler
Loaded: loaded (/lib/systemd/system/forticlient.service; enabled; vendor preset: enable>
Active: active (running) since Thu 2022-10-27 06:13:47 CEST; 3h 25min ago
Main PID: 854 (fctsched)
Tasks: 64 (limit: 38385)
Memory: 55.2M
CPU: 3min 1.004s
CGroup: /system.slice/forticlient.service
├─ 854 /opt/forticlient/fctsched
├─ 967 /home/opt/forticlient/confighandler

We needed to identify which one was the local web server our Forticlient binary was trying to connect to. We tried with the second one first because its name stood out:

sudo lsof -p 967|grep LISTEN

confighan 967 root 7u IPv4 18773 0t0 TCP localhost:34069 (LISTEN)

So, the confighandler binary was the web server and in our case it was listening for connections on TCP localhost:34069.

Instead of trying more things with the GUI, we found out that Forticlient also included a command line client: fortivpn. So we ran the client within strace and we confirmed that, somehow, the web server accepted the connection first and then it immediately closed it:

socket(AF_INET, SOCK_STREAM|SOCK_NONBLOCK, IPPROTO_IP) = 9
connect(9, {sa_family=AF_INET, sin_port=htons(34069), sin_addr=inet_addr(“127.0.0.1”)}, 16) = -1 EINPROGRESS (Operation now in progress)
epoll_ctl(6, EPOLL_CTL_ADD, 9, {EPOLLOUT, {u32=9, u64=9}}) = 0
epoll_wait(6, [{EPOLLOUT, {u32=9, u64=9}}], 32, 3000) = 1
getsockopt(9, SOL_SOCKET, SO_ERROR, [0], [4]) = 0
getpeername(9, {sa_family=AF_INET, sin_port=htons(34069), sin_addr=inet_addr(“127.0.0.1”)}, [28->16]) = 0
epoll_ctl(6, EPOLL_CTL_DEL, 9, 0x7fffdb67ebfc) = 0
getsockopt(9, SOL_SOCKET, SO_ERROR, [0], [4]) = 0
epoll_ctl(6, EPOLL_CTL_ADD, 9, {EPOLLOUT, {u32=9, u64=9}}) = 0
epoll_ctl(6, EPOLL_CTL_MOD, 9, {EPOLLIN|EPOLLOUT, {u32=9, u64=9}}) = 0
epoll_wait(6, [{EPOLLOUT, {u32=9, u64=9}}], 32, 3000) = 1
writev(9, [{iov_base=”GET /vpn HTTP/1.1\r\nContent-Type:”…, iov_len=68}], 1) = 68
epoll_ctl(6, EPOLL_CTL_MOD, 9, {EPOLLIN, {u32=9, u64=9}}) = 0
epoll_wait(6, [{EPOLLIN, {u32=9, u64=9}}], 32, 3000) = 1
ioctl(9, FIONREAD, [0]) = 0
readv(9, [{iov_base=””, iov_len=4096}], 1) = 0
epoll_ctl(6, EPOLL_CTL_DEL, 9, 0x7fffdb67ebcc) = 0
shutdown(9, SHUT_WR) = 0
close(9) = 0
fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(0x88, 0x21), …}) = 0
write(1, “ERROR: Request failed.\n”, 23ERROR: Request failed.
) = 23

Okay, so we needed to find out why the web server was closing the connection right after the client was successfully connected to it. Obviously, iptables was allowing TCP connections within the loopback interface, so that was not he issue here.

CVE-2021-43205

In this particular GNU/Linux box, Forticlient was installed on /home/opt/forticlient, and there was a soft link here: /opt -> /home/opt. This was so because the root partition was running out of space. Because of this setup, confighandler was executed from /home/opt/fortinet/confighandler, so was the fortivpn command line client and the GUI version. Then, we sort of thought: okay, so we have this web server listening on localhost we can interact with. Surely we can write our own web-sockets client and send requests to it. But we shouldn’t get any responses back, right? Because that would be a vulnerability! So maybe the web server, because both clients (fortivpn and Forticlient GUI) were not run from /opt/fortinet/, was dropping all client connections!

We googled this, and we got a well-known vulnerability affecting, precisely, confighandler: https://www.fortiguard.com/psirt/FG-IR-21-226 !

Fixing it

We removed the /opt soft link and then we mounted /home/opt over /opt using the “bind” option:

rm /opt; mkdir /opt; mount -t none /home/opt /opt -o bind

We re-ran the client and this time the connection was not dropped, and thus both client and server were communicating via websockets with no issues at all. And, of course, the Forticlient window showed all the widgets too!

Blank screen check list

This time the issue happened because we did not have an standard installation of GNU/Linux. Remember: the /opt directory was, in fact, a soft link to /home/opt. But we know there are some other issues been reported on Fortinet forums as long as on other online platforms. So what follows is kind of a check-list in order to solve the blank screen problem:

  1. Make sure the forticlient service is running before trying anything else by typing: systemctl status forticlient. You must have two processes running: fctsched and confighandler. If confighandler is NOT running, the client will show a blank screen.
  2. If both binaries are running and you are still getting a blank screen, run fortivpn command-line client and see if you can communicate with the confighandler web server without issues. If not, make sure that TCP loopback connections are allowed using iptables, and that Fortilcient is correctly installed on /opt/forticlient.
  3. If the issue is not communicating with the web server, run the GUI with –enable-logging. Chances are that the issue may be related to some GPU glitch or bug. If you get something like this: The GPU process has crashed N times, then you have an issue with your graphics card and the OpenGL acceleration used by ElectronJS to render its widgets. In this case, run the GUI with: –disable-gpu.
  4. If that does not fix your issue, and you have an NVIDIA card, try running the GUI forcing software rendering: __GLX_VENDOR_LIBRARY_NAME=mesa LIBGL_ALWAYS_SOFTWARE=1 /opt/forticlient/gui/FortiClient-linux-x64/FortiClient
  5. If all this fails, make sure you have the XDG_RUNTIME_DIR environtment variable correclty set. When this variable is not set, apart from the blank screen, if you try to run the command-line utility “fortivpn” you get the following error: “Failed to set up FortiClient directories.”. So make sure to run the following: export XDG_RUNTIME_DIR=/run/user/$UID and then try again. Don’t forget to add this to your ~./bashrc.

Issues concerning gnome-keyring

On another GNU/Linux box, after being able to log-in using valid VPN credentials, the GUI did not show the TCP/IP settings of the VPN, although it showed the “Disconnect” button. The computer was not connected to the VPN though. Using the “–enable-logging” again, we got this:

14 [2120:1027/132520.954416:INFO:CONSOLE(9)] “Uncaught (in promise) Error: Cannot create an item in a locked collection”, source: file:///opt/forticlient/gui/FortiClient-linux-x64/resources/app.asar/assets/js/bundle.min.js (9)

This quickly pointed us to the gnome-keyring daemon. The user did not remember their password to unlock the secrets repository, and thus they were pushing the “Cancel” button after logging into their computer. So the keyring repository remained locked, and thus Forticlient was not able to store the credentials in the keyring. Therefore, the previous error message.

We restore the keyring by renaming the keyrings directory (the user did not store credentials there anyway):

mv ./local/share/keyrings .local/share/keyrings.old