State of multi-player Wayland

> This is obviously not a great solution, and VNC in general feels like ancient technology

It's not obvious to me. This looks like software ageism. :)

There are at least 2 things that could be done to improve performance in the author's setup:

  1. Use the --gpu flag which enables dmabuf capture and h264 encoding via vaapi. 
  2. Connect using a dedicated VNC client like TigerVNC. This reduces latency by roughly 50 ms.

For more details on 2. see second paragraph in the following blog post https://andri.yngvason.is/measuring-latencies.html

That being said, noVNC has gained hardware decoding of h264 via WebCodecs since the writing of that blog post, so performance will have improved some. It's unlikely that the 50 ms latency has gone down, but it might be worth measuring again since browsers may opt to render video through a dedicated overlay plane that bypasses regular rendering. However, it would be silly not to synchronise frame presentation between overlays.

To add to the long list of caveats: This also breaks assumptions GUI toolkit and applications make about only one of their windows being focused at a time: `gtk_application_get_active_window`, `QApplication::activeWindow`, etc.

I have never liked split keyboards but I might if each keyboard was also a mouse with its own cursor, it seems fun to develop the skill to work that way. Would also be useful at times if the the Thinkpad trackpoint and trackpad could each have their own cursor and I could leave one somewhere. Or maybe a cursor stack like pushd/popd, registers would probably work better; instead of virtual desktops just give me one massive desktop with cursor locations I can jump between, switch to cursor 3 it saves the location of the current cursor and jumps.

The original title, "State of multi-player Wayland" is a much better summary of the article. This ain't just about the pointers! In fact, the author is very explicit about that:

> I think there could be a term that’s describing this concept even better – in the title of this post, I picked “multi-player”, to make clear that it’s mostly about multiple people who are collaborating. I’ve also seen “multi-cursor” or “multi-pointer” (which are missing the keyboard component).

I'm building a compositor now, and I've been thinking about multi-seat from the beginning, but I know there are lots of places that assume a single seat, and so this sort of thing won't work for me. It's actually a lot more work than you'd think to support this...

I wish libinput gave the option for pointer inertia, where the pointer glides to a gradual rather than a hard halt after you stop moving it. I’m limited to a trackpad these days and it’s a really nice feature on my Steamdeck that I’d love to have on my Thinkpad.

Come to think of it, Steamdeck is Wayland? How do they implement it?

cant read the entire article on ios because the left part is just somehow cut off :(

This feels like a step in the right direction from this talk on desktop UX. Finally some cool innovation here!

https://youtube.com/watch?v=1fZTOjd_bOQ

On the web, PointerEvents and touch events both support multiple different cursors. https://developer.mozilla.org/en-US/docs/Web/API/Touch_event... https://developer.mozilla.org/en-US/docs/Web/API/PointerEven...

I've always thought this was a cool idea. It's obviously been on the minds of Wayland protocol developers since the beginning, otherwise it wouldn't be possible!

    xinput list  # find second mouse id
    xinput create-master second
    xinput reattach <mouse-id> "second pointer"

    xinput reattach <mouse-id> 'Virtual core pointer'
    xinput remove-master 'second pointer'

one interesting use case is surface management in ways that cant be solved easily by swapping desktops or which is clunky this happens most on single-screen where have to swap apps out that have related actions and want to rearrange or copy paste between or scroll sync and native app so cant scroll sync by extension think rapid fire macro mouse events except not just serial which can run the experience eg browser tab and electrion app that sync scroll or scroll c/p and where macro recorder might come close but not really

I have been dreaming about a split keyboard with dual trackballs and two cursors for the past few months. This may be the kick i need to actually get on it.

I remember the multiplayer of Mahjong on the Amiga worked by just plugging in a second mouse. Voila, two pointers.

I immediately recognized the avatar of this person. They are the one with the DIY soft drinks which has been posted on HN before: https://blinry.org/diy-soft-drinks/

After using a space mouse for CAD it always seems weird how dual mice isn't a more common thing.

While op uses systemd/udev's internal property mutation mechanism to configure seats, not that sway (and likely other compositors) allow you to just configure the device-to-seat assignment in your configuration or even at runtime.

Finally I get to play The Settlers II in split screen mode again

I use two mice just to be able to switch from the fatigued hand to the other without shifting the mouse repeatedly. I wish it was easy to have two cursors in KDE.

The more "typical" use for a feature like this is drawing tablets and touch screens. You don't necessarily want a touch/drawing tablet input to move your actual "main" mouse cursor.

Very timely post. I wonder, is it possible to clone one single mouse cursor, to duplicate exactly same actions on two windows placed side by side?

Maybe RedHat should contract this guy to finally get pointer warp implemented for Wayland everywhere it needs to be.

Focus of controls is the hard part. You end up needing a Per-seat version of focus in an application.

Innovative stuff. I wonder if similar things are possible on Windows and macOS.

This is actually something I've always wanted, but people told me it's dumb. I want two cursors and two keyboards, with two "focused window" panes. If Mouse A clicks Pane1, keyboard A presses should go into Pane1. I think this would be so fun for multiplayer games on the same computer for example (think two Minecraft instances running).

It's not as weird as you might think: laptops may have touchscreens (which can be multi-touch) on top of the "normal" cursor. IIRC, X.org can't really handle such cases cleanly.

Cool! Where can I get the turtle mouse cursor?

and we can also assign this to particular apps like, mouse 1 and keyboard one can only interact with notepad !! etc !!

This is cool. I feel like I shouldn’t be the only one who as a kid connected up an extra mouse to their computer with their buddy and was disappointed to still only have one cursor.

The way screen sharing solutions like Teams handle the multi cursor thing is to just keep the main cursor jumping around, which works, but meh.

Having multiple cursors within the same environment theoretically enables such interesting collaborative experiences. Imagine a large surface in which everyone has their own workspace (cluster of windows), but you can seamlessly go between them …

Another possible use case might be mixed reality/VR displays, for simultaneous mouse cursor + eye tracking/control.

Sexy Flanders

Imagine working with two separate mouses at the same time - no keyboard.

On the note of the two different kinds of seats, a tangent: better multi physical seat support is something I am passionate about, specifically, I want Windows-style "Fast User Switching" on Linux with RDP. Today's Linux RDP support usually acts like screen mirroring; you get a mirror of a physical seat. With GNOME I believe there is also now support to do remote login: you can log in remotely, and even disconnect and reconnect to a remote login session. But what you can't do is continue a physical session remotely or a remote session physically, and the reason why is sorta to do with seats. Sort of.

See, when you start a graphical session on modern systemd-based Linux, a new logind session is created, and it is immutably tied to a seat. So you can't just remove the seat from a running logind session and then make it remote, or add the seat to a running logind session and make it physical. I actually tried patching systemd-logind to do this, but when I proposed it upstream, Lennart Poettering himself actually took the time to explain that this wasn't the intended way to do it: instead, you would actually just create a new logind session and drop the old one entirely, or literally whatever you want.

Which to me sounded perposterous, but then I realized that it really wasn't that ridiculous. As it was, when you boot a desktop using systemd-logind, the desktop itself doesn't typically run much of anything directly in the session context anyway - the compositor and almost all services actually run as user services, as in systemd-land having more than one active desktop session per user isn't really supported. Okay so we can do this!

And so I set out to prototype this with KDE, but there are still a few issues...

- First and foremost, $XDG_SESSION_ID. Believe it gets set by the systemd PAM module somehow? I don't fully recall. Either way, it gets used by various programs to determine the current logind session, but if our graphical session can "roam" between them we need a mechanism that can handle changing over time. I propose we stuff the current logind session ID into a file in $XDG_RUNTIME_DIR instead.

- Secondly, compositors need to be redesigned to handle this. Compositors today can hotplug displays and input devices - that's a good start. Unfortunately, at least kwin can't switch backends at runtime. You can really go one of two ways, and I tried a bit of both and either can work but both are a bit ugly: one is to make a single backend that can handle either case and the other is to allow switching the backend at runtime. Either way, when running remotely the card* DRM nodes are unavailable so you need to switch to renderD* nodes instead. You can also use both, and only use card* for scanout when available, but it seems a bit trickier.

- Finally, I genuinely have no idea how this could or should be handled in non-systemd environments. I have admittedly forgotten how that all works. It would be rather lame to simply require logind for this to work, but sadly it is the path of least resistance at least at first.

And maybe some more, but otherwise the concept does indeed broadly work. I mainly have only tested it in virtual machines with DRM native context, but I can see everything working switching in either direction at least once before things start to get screwed up. Even OpenGL clients and audio playback can move seamlessly! It's just like what Windows accomplished over 20 years ago!

Aside from convenience and being cool, it would also solve a couple problems. No need to worry about the limit of only one session per user: just move your session with you. No need to worry about your physical session being unlocked when remotely connected: when you log in this way your physical seat gets disconnected from the session entirely and the display manager takes it back over.

I actually would really like to at least make some progress on ground work that would be needed to make this happen, although I'm not really that sure that anybody cares. But either way this post's brief discussion about physical vs logical seats activated a few neurons so I couldn't help but dump this out.

Now I know why Web devs, frontend developers hate us