My smartphone as a shortcut helper

August 30, 2017

Exception this month : this is not a personnal project. With Loic (@loicdesroc), we designed this POC and all apps together.
No code published now, we are thinking about the best licence for publication.

Purpose

Nowadays almost every software uses keyboard shortcuts to allow fast and productive user experience. When mastered by the user those shortcuts have proven their efficiency and usefulness. Despite the fact that shortcuts have many advantages they also have some major drawback. On some softwares you can have more than 500 shortcuts. The use of a large number of shortcuts leading inevitably to their complexification and makes them hard to remember for users especially for casual users.

In the 80’s, function keys were used, with this famous banner at the bottom of the 80x25 16 colors text app :

This active banner minds me…. oh wait !

To work around this problem software editors and power user have created the concept of shortcut cheatsheet. This involves placing as much informations as possible on a single sheet. Keep it close to the keyboard so you can take a quick look at. But this concept also has its limitations.

To illustrate here is a partial cheat sheet for the well known vectorial design tool Inkscape :

As you can see there is a lot of differents and complex combinations in order to bind every useful tools to a shortcut. These shortcuts often involve 3 or more keys.

Some of them are common use and conventional like the classic Ctrl+C and Ctrl+V to copy and paste. But the vast majority are cryptique to the casual user who don’t need to use them every day. Also a casual user don’t want to learn them because it won’t be worth the effort for their usage.

Nonetheless, casual users would like to improve their productivity without having to learn anything. Who would not ?

Moreover more and more users are using dedicated smartphone applications to avoid having to print the shortcut list.

https://play.google.com/store/apps/details?id=com.positiveapps.cheatsheetforphotoshop

Now what if we make this application interactive ?
What if your smartphone can directly drive your computer software through these shortcuts ?

What we built

The main idea is really simple : Put your smartphone on top of your keyboard and use it as a “shortcuts touchscreen device”.

A custom software (written in C# for Windows) send the name of the active window you are using. The application on the phone loads a plugin (written in HTML/CSS/JS). It displays shortcuts as buttons.
This is how it looks like :

It works with plugins. The system detects which application is running and shows a custom interface. Here are some examples of customs plugins we built :

Eclipse IDE

Google Chrome (also works with Firefox)

Visual Studio Code

Inkscape, with a 3 tabs interface

Gimp

Libreoffice / Open Office / Ms Office

How it works

This software contains 2 parts :

Server (written in C#)
Client (Web App)

Server

Retreive active window

Retreive the active window on Windows (7 - 8 - 10) is easy by using system calls.

            
            [DllImport("user32.dll")]
            static extern IntPtr GetForegroundWindow();

            [DllImport("user32.dll")]
            static extern int GetWindowText(IntPtr hWnd, StringBuilder text, int count);

            [DllImport("user32.dll")]
            static extern int GetWindowThreadProcessId(IntPtr hWnd, out int lpdwProcessId);

With 3 calls, we got :

Current window
The title of the window
The name of the process who runs the window

We need the current title of the window for plugins running inside webapps for example. With GMail openned in a chrome tab, with only the process name it is impossible to load a specific plugin for GMail.

Communication

We built a web application, running on the smartphone. The easiest way to communicate is Websockets. And it also gives security, nobody else can retreive informations or even send keycodes.

2 websockets are used :

Window informations, gives to the webapp which window is active
Keystrokes, send key combinations to achieve.

Keystrokes

This is a list of key codes (ANSI) given to a KeyboardSimulator instance. By this way, we can send and invoke every keyboard shortcut we want.

Client

We built an Android App, with only a webview for now. The screen's phone is always up, that is not possible with a browser.

View

Examples are build with Bootstrap and JQuery.
2 javascript functions are available :

sendCodes(int[]) : send keystrokes. If you want a 2 key combination, it requires a 4 elements array, ["keypressCode", "keypressCode", "keyreleaseCode", "keyreleaseCode"]
loadPlugin() : here are located conditions to load a plugin, with window title or process name.

There is also a big list of constants for each keyboard key. Examples :

            
                var VK_ENTER;
                var VK_BACK_SPACE;
                var VK_TAB;
                var VK_CANCEL;
                var VK_CLEAR;
                var VK_SHIFT;
                var VK_CONTROL;
                var VK_ALT;
                var VK_PAUSE;
                var VK_CAPS_LOCK;
                var VK_ESCAPE;
                var VK_SPACE;
                var VK_PAGE_UP;
                var VK_PAGE_DOWN;
                var VK_END;
                var VK_HOME;
                var VK_LEFT;
                var VK_UP;
                var VK_RIGHT;
                var VK_DOWN;
                var VK_COMMA;
                var VK_MINUS;

Those constants are extracted from this file : https://docs.oracle.com/javase/7/docs/api/java/awt/event/KeyEvent.html
It also contains multimedia keys from special keyboards. Play, Pause, Previous, Next or Volume keys are available.

Conclusion

Such a software will need a lot more development to be really used by anyone. In particular more plug-ins will be needed to make it useful for the major part of casual and professional users.

That being said, this simple proof of concept have proven its usefulness and high compatibility with various software both open and closed-source. Using directly keyboard shortcuts makes this software virtually compatible with almost every existing front-end softwares.

Its only major disadvantage is that it needs spécific plug-in for each compatible software. But this can be turned into a major strength by adding a plug-in marketplace and building a strong community of software users, editor and developer.

We are seriously thinking about developing this marketplace, the future of this project will depend of your interest in our proof of concept and article. So don’t hesitate to share it as much as you like it.