{"id":192,"date":"2022-10-04T20:57:43","date_gmt":"2022-10-04T20:57:43","guid":{"rendered":"https:\/\/www.andyhartzell.com\/staging\/?page_id=192"},"modified":"2022-10-11T02:01:12","modified_gmt":"2022-10-11T02:01:12","slug":"iui","status":"publish","type":"page","link":"https:\/\/www.andyhartzell.com\/gallery\/iui\/","title":{"rendered":"iUi"},"content":{"rendered":"\n<figure class=\"wp-block-embed is-type-video is-provider-vimeo wp-block-embed-vimeo wp-embed-aspect-16-9 wp-has-aspect-ratio\"><div class=\"wp-block-embed__wrapper\">\n<iframe loading=\"lazy\" title=\"Jim Marggraff \u2014 The Language of Looking (FoST)\" src=\"https:\/\/player.vimeo.com\/video\/173686386?dnt=1&amp;app_id=122963\" width=\"500\" height=\"281\" frameborder=\"0\" allow=\"autoplay; fullscreen; picture-in-picture; clipboard-write\"><\/iframe>\n<\/div><\/figure>\n\n\n\n<p>Eyefluence was in the midst of a pivot when I came aboard. Its proprietary eye-tracking technology had been developed to enable \u201clocked-in\u201d quadriplegics to communicate. The system had been a life-saver for many, but the company was floundering. It needed to expand its market to survive, and it had set its sights on the emerging XR industry.<\/p>\n\n\n\n<p>We weren\u2019t the only player in this arena. But our competitors were mostly pitching passive use-cases: foveated rendering and heatmap analysis. Eyefluence founder Jim Marggraff had a hunch that gaze might serve as an active input, a central component of an interaction system that might in fact be superior to manual controls.<\/p>\n\n\n\n<p>This went against prevailing wisdom. It\u2019s well-known that our gaze isn\u2019t entirely under our conscious control: it flits about distractedly while we think, or it can be captured by passing shiny objects. The \u201cdwell-based\u201d input system that had been devised for quadriplegics was functional but slow: to signal intent, the user generally had to focus on an affordance until a progress-circle filled.<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><a href=\"https:\/\/www.andyhartzell.com\/gallery\/wp-content\/uploads\/2022\/10\/pan_zoom_vivie_expo.gif\"><img loading=\"lazy\" decoding=\"async\" width=\"600\" height=\"299\" src=\"https:\/\/www.andyhartzell.com\/gallery\/wp-content\/uploads\/2022\/10\/pan_zoom_vivie_expo.gif\" alt=\"\" class=\"wp-image-235\"\/><\/a><\/figure>\n\n\n\n<p>Marggraff understood that in order to be viable for a general audience, a gaze-based UI needed to operate at the speed of thought. The eyes are, after all, the fastest-moving organs in the human body, and the organs most closely wired to the brain. How to develop a UI that was both lightning fast and foolproof, one that might enable the user to perform all the functions of a traditional interface without triggering \u201cinadvertent activations?\u201d This was the challenge I took on as Creative Director.<\/p>\n\n\n\n<p>We dove into the physiology, psychology and physics of our subject. The eye exhibits two types of motion: the saccade, a staccato glance between two fixed points, and the pursuit, the fluid motion that obtains when gaze is locked on a moving subject. Our tracking technology enabled us to reliably distinguish between these two actions, and with this assurance we set about constructing our interaction system.<\/p>\n\n\n\n<p>The centerpiece of the system was the \u201cactivation saccade,\u201d an intentional glance between two points. Because the saccade is ballistic, we found we could accurately determine the endpoint before the fact. This enabled us to serve up feedback before the triggering glance was complete. The user experienced this as the system \u201creading my mind,\u201d something akin to magic.<\/p>\n\n\n\n<p>Keeping the user on-task is a prime challenge of the gaze-based UI: if I\u2019m using my eyes both to read my environment and to work the controls, I can easily lose focus. We pioneered a technique called \u201ceye-holding,\u201d a subtle moving particle to lead the gaze to the desired spot.<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><a href=\"https:\/\/www.andyhartzell.com\/gallery\/wp-content\/uploads\/2022\/10\/ViveDemo-1.gif\"><img loading=\"lazy\" decoding=\"async\" width=\"600\" height=\"367\" src=\"https:\/\/www.andyhartzell.com\/gallery\/wp-content\/uploads\/2022\/10\/ViveDemo-1.gif\" alt=\"\" class=\"wp-image-236\"\/><\/a><\/figure>\n\n\n\n<p>By utilizing saccade and pursuit in various combinations, we enabled a range of interactions: scroll and rotate, pan and zoom, search and organize, and more. We even tested a gaze-based version of the Android swipe keyboard. The system was instantiated in a suite of demos, which we exhibited at conventions and private presentations. We demonstrated that, with very little practice, the user could learn to perform complex tasks, rapidly and intuitively, using just her eyes. This showcase eventually attracted the notice of Google, which acquired Eyefluence in 2016.<\/p>\n\n\n\n<p>I worked closely with Marggraff to define the system and spec out the demos. We typically brainstormed them together on wipe boards. I would capture the boards and turn them into documentation and wireframes, working mostly in Illustrator. I then directed a small team of Unity developers through the iterative build process, occasionally making adjustments in-engine.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Eyefluence was in the midst of a pivot when I came aboard. Its proprietary eye-tracking technology had been developed to enable \u201clocked-in\u201d quadriplegics to communicate. The system had been a life-saver for many, but the company was floundering. It needed to expand its market to survive, and it had set its sights on the emerging [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-192","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/www.andyhartzell.com\/gallery\/wp-json\/wp\/v2\/pages\/192"}],"collection":[{"href":"https:\/\/www.andyhartzell.com\/gallery\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/www.andyhartzell.com\/gallery\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/www.andyhartzell.com\/gallery\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.andyhartzell.com\/gallery\/wp-json\/wp\/v2\/comments?post=192"}],"version-history":[{"count":3,"href":"https:\/\/www.andyhartzell.com\/gallery\/wp-json\/wp\/v2\/pages\/192\/revisions"}],"predecessor-version":[{"id":237,"href":"https:\/\/www.andyhartzell.com\/gallery\/wp-json\/wp\/v2\/pages\/192\/revisions\/237"}],"wp:attachment":[{"href":"https:\/\/www.andyhartzell.com\/gallery\/wp-json\/wp\/v2\/media?parent=192"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}