Mixing GUIs and CLIs on Windows: A Cautionary Tale

I f you’ve used desktop Linux, then I’m sorry for you.1

You will, however, most likely be familiar with the practice of using the same app from either the CLI or the GUI, depending on how you invoke it and what you want to do with it. In some cases, the CLI merely replicates the functionality of the GUI, but (due to being, you know, a CLI) is much easier to incorporate in scripts and such. In other cases (Wezterm is a good example) the GUI app acts as a “server” with which the CLI communicates to cause it to do various things while it runs.

On Linux, this is as natural as breathing. There’s nothing in particular that distinguishes a “GUI app” from a “CLI app”, other than that the GUI app happens to ultimately call whatever system APIs are involved in creating windows, drawing text, and so on. Moreover, even when running its GUI, a Linux app always has stdout, stderr, etc. In most day-to-day usage, these get stashed in some inscrutable location at the behest of Gnome or XFCE or whatever ends up being responsible for spawning and babysitting GUI apps most of the time, but you can always see them if you launch the app from a terminal instead. In fact, this is a common debugging step when an app is misbehaving: launch it from a terminal so you can see whether it’s spitting out errors to console that might help diagnose the problem.

Since Windows also has both GUIs and CLIs, you might naively expect the same sort of thing to work there, but woe betide you if you try to do this. Windows thinks every app must be either a GUI app or a CLI app, and never the twain shall meet, or at lest never the twain shall meet without quite a significant degree of jank.

Every Windows executable is flagged somehow2

as “GUI app” or “CLI app”, and this results in different behavior on launch. CLI apps are allocated a console, on which concept I’m not entirely clear but which seems somewhat similar to a pty on Linux. GUI apps, on the other hand, are not expected to produce console output and so are not allocated a console at all, which means that if they try to e.g. write to stdout they just… don’t. I’m not sure what exactly happens when they try: in my experience e.g. println!() in Rust just becomes a no-op, but it’s possible that this is implemented on the Rust side because writing to stdout from a GUI app would crash your program otherwise, or something.

Aha, says the clever Windows developer, but I am a practitioner of the Deep Magicks, and I know of APIs such as AllocConsole and FreeConsole which allow an app to control the existence and attached-ness of its Windows consoles. But not so fast, my wizardly acquaintance. Yes, you can do this, but it’s still janky as hell. There are two basic approaches: you can either a) flag the executable as a GUI app, then call AllocConsole and AttachConsole to get a console which can then be used for stdout/err/etc, or you can b) flag the executable as a CLI app, so it gets allocated a console by default, then call FreeConsole to get rid of it if you decide you don’t want it.

If you do a), the problem is that the app doesn’t have a console at its inception, so AllocConsole creates an entirely new console, with no connection to the console from which you invoked the app. So it pops up in a new window, which is typically the default terminal emulator3

rather than whatever you have set up, and - even worse - it disappears as soon as your app exits, because of course its lifecycle is tied to that of the app. So the extremely standard CLI behavior of “execute, print some output, then exit” doesn’t work, because there’s no time to read that output before the app exits and the window disappears.

Alternatively, you can call AttachConsole with the PID of the parent process, or you can just pass -1 instead of a real PID to say “use the console of the parent process”. But this almost as terrible, because - again - the app doesn’t have a console when it launches, so whatever shell you used to launch it will just assume that it doesn’t need to wait for any output and blithely continue on its merry way. If you then attempt to write to stdout, you will see the output, but it will be interleaved with your shell prompt, keyboard input, and so on, so again, not really usable.

Ok, so you do b) - flag your app as a CLI app, then call FreeConsole as soon as it launches to detach from the console that gets automatically assigned to it. Unfortunately this doesn’t work either. When you launch a CLI app in a context that expects a GUI, such as the Start menu, it gets assigned a brand-new console window, again using whatever is the default terminal emulator. In my experience, it isn’t consistently possible (from within the process at least) to call FreeConsole quickly enough to prevent this window from at least flashing briefly on the desktop. Livable? Sure, I guess, but it would be a sad world indeed if we never aimed higher than just livable.

Up until now, my solution has been to simply create two copies of my executable, one GUI and one CLI, put them in different directories, and add the directory of the CLI executable to my PATH so that it’s the one that gets invoked when I run mycommand in a terminal. This works ok, despite being fairly inelegant, but just today I discovered a better way via this rant.4

Apparently you can specify the CREATE_NO_WINDOW flag when creating the process, which prevents it from creating a new window. Unfortunately, as that page notes, this requires you to control the invocation of the process, so the only way to make proper use of it is to create a “shim” executable that calls your main executable (which will be, in this case, the CLI-first version) with the CREATE_NO_WINDOW flag, for when you want to run in GUI mode. That post also points out that if you have a .exe file and a .com file alongside each other, Windows will prefer the .com file when the app is invoked via the CLI, so your shim can be app.exe and your main executable app.com. I haven’t tried this yet myself, but it sounds like it would work, and it’s a general enough solution5 that app frameworks such as Tauri6 might eventually handle it for you.

Another solution that was suggested to me recently (I think this is the more “old school” way of handling this problem) is to create a new virtual desktop, then ensure that the spurious console window gets created there so that it’s out of sight. I haven’t tried this myself, so I’m not familiar with the details, but my guess is that like the above it would require you to control the invocation of the process, so there isn’t really any advantage over the other method, and it’s still hacky as hell.

Things like this really drive home to me how thoroughly Windows relegates the CLI to being a second-class citizen. In some ways it almost feels like some of the products of overly-optimistic 1960s-era futurism, like designing a fighter jet without a machine gun because we have guided missiles now, and obviously those are better, right? But no, in fact it turns out that sometimes a gun was actually preferable to a guided missile, because surprise! Different tools have different strengths and weaknesses.

Of course, if Microsoft had been in charge of the F-4 it would have taken them 26 years to finally add the machine gun, and when they did it would have fired at a 30-degree angle off from the heading of the jet, so I guess we can be thankful that we don’t have to use our terminal emulators for air-to-air dogfights, at least.