An aspect of video calls that many of us take for granted is the way they can switch between feeds to highlight whoever’s speaking. Great — if speaking is how you communicate. Silent speech like sign language doesn’t trigger those algorithms, unfortunately, but this research from Google might change that.
It’s a real-time sign language detection engine that can tell when someone is signing (as opposed to just moving around) and when they’re done. Of course it’s trivial for humans to tell this sort of thing, but it’s harder for a video call system that’s used to just pushing pixels.
A new paper from Google researchers, presented (virtually, of course) at ECCV, shows how it can be done efficiency and with very little latency. It would defeat the point if the sign language detection worked but it resulted in delayed or degraded video, so their goal was to make sure the model was both lightweight and reliable.
The system first runs the video through a model called PoseNet, which estimates the positions of the body and limbs in each frame. This simplified visual information (essentially a stick figure) is sent to a model trained on pose data from video of people using German Sign Language, and it compares the live image to what it thinks signing looks like.
Image Credits: Google
This simple process already produces 80 percent accuracy in predicting whether a person is signing or not, and with some additional optimizing gets up to 91.5 percent accuracy. Considering how the “active speaker” detection on most calls is only so-so at telling whether a person is talking or coughing, those numbers are pretty respectable.
In order to work without adding some new “a person is signing” signal to existing calls, the
The new Microsoft Flight Simulator is an immersive beast of a PC game, and we can only imagine how immersive it might get in VR — but you might not have to imagine much longer, because Microsoft has just opened signups (via Eurogamer) for a closed beta of the virtual reality experience.
There are quite a few requirements if you want to be considered, though. Not only do you have to own the game, have a Windows Mixed Reality headset, be a registered Microsoft Flight Simulator “Insider” and sign an NDA, you’ll need a slightly beefier PC than the base game — with an Nvidia GeForce GTX 1080 or better sporting 8GB of VRAM, as well as 16GB of system memory.
And, you’ll need to prove your PC qualifies by submitting your DxDiag (press your Windows start button, type “dxdiag”, hit Enter) so Microsoft can confirm those specs and that your system isn’t prone to BSODs or other major system errors.
(The DxDiag file is basically just an inventory of your system, but it does also list recent application crashes.)
Intriguingly, it doesn’t look like you’ll need the HP Reverb G2, the specific headset that Microsoft originally said would get first dibs on the game. Any Windows Mixed Reality headset will apparently do to start, which should be neat for those who got deep discounts on items like the Lenovo Explorer and Samsung Odyssey over the past couple of years.
The company isn’t saying when the closed beta will begin, but it had previously promised a free VR update this fall, so it probably won’t be long. If you’ve got an Oculus, HTC, or other VR headset, there’ll be a second wave of closed beta for other devices as well.
It’s not surprising Microsoft might want to be cautious