Platypus

WHAT IS IT?

Platypus takes spoken text from speech recognition programs like Dragon NaturallySpeaking® (that run in the Wine Windows® emulator) and relays it as keystrokes, so it can be used anywhere, including Directx games, Virtualbox, VMWare and Qemu sessions, VNC, other users' desktops, remote terminals or anywhere else dictation didn't work before.

Platypus also serves as a basic text editor, a global "Paste anywhere" and voice-enabled macro utility to automate certain tasks. It can type user configurable keyboard commands, such as 'Alt_L F4' (file close) and 'Control_L s' (file save) by voice command. It can even launch Windows or Linux programs and scripts, e.g. "start terminal". See customization hints below.

INSTALLATION

1. Install Wine. (Follow link to official website or use your operating system's update tool.)
2. Create a separate wine prefix named ".dictation" (What is this?)
   1. Open a terminal.
   2. Make sure you are not the root user. (Why?)
   3. Type or paste the following command:

   export WINEPREFIX=$HOME/.dictation

   Important: Keep this terminal open. The WINEPREFIX environment variable has been set to "$HOME/.dictation" so that Wine software will install to that location.

3. Use the above terminal to install Dragon NaturallySpeaking®. (Installation instructions for different versions.). Fedora users: For best results, install alsa-plugins-pulseaudio.i686, run winecfg and choose "alsa" under the audio drivers tab.
4. Make sure that dictation into DragonPad is working.
5. Some libraries will be needed to compile Platypus. Download them using your operating systems update tool or package manager to pull in any dependenocies. Keep in mind that if you are planning to cross-compile to 32 bit on 64 bit hosts, you will need 32 bit (?86) versions of these libraries:

       wine-devel
       xorg-x11-proto-devel
       libX11-devel
       libXtst-devel

6. Download Platypus. Unzip the archive and install to $HOME/.dictation. Again, do not use root.

    export WINEPREFIX=$HOME/.dictation
    ./configure
    make
    make install

USAGE

1. Select which microphone to use:

       alsamixer -c0 -V capture
       (or make a custom amixer script)

2. Start Dragon NaturallySpeaking (Applications--Wine--Programs)
3. Turn on Dragon's microphone
4. Start Platypus from the desktop icon.
5. Open any Linux application and start dictating.

CUSTOMIZATION

Desktop Voice Control

Edit the file, "platypus_commands"

Spoken text:            What to do:

"start Firefox"        "firefox&"
"start terminal"       "xterm&"
"file close"           "Alt_L F4"
"file save"            "Control_L s"
"file print"           "Control_L p"
"banana radio"         "T y p e space s o m e t h i n g ."

Commands ending with the ampersand (&) character, like "firefox&" are executed directly on the host system. The rest are interpreted as keystrokes separated by spaces. Be aware that "make install" copies this file to the program directory, so don't edit that one or it will be overwritten if you install again.

Special keys (keysyms), like "space" and "Control_L", use the quaint X-style syntax. A full list of those special keysym codes (with "KX_" prepended to them) is defined in /usr/include/X11/keysymdef.h (but due to keyboard limitation only 255 keycodes are available to type with (0x00-0xff) and selection is limited to what can actually be typed on the particular keyboard layout currently in use [System Settings - Keyboard - Layout].)

Special Characters©

In order to output a character to the screen, Platypus has to emulate the exact keys a user would press when typing that character on the keyboard. A different platypus_keys is needed for every keyboard layout. Only two layouts are supported at the moment, US and US (International AltGr with dead keys). Luckily, most symbols can be typed on these keyboards.

US Keyboard (Default)

As of version 0.8, Platypus will now type unicode characters on any keyboard layout using ISO 14755-conformant input (GTK+: Firefox, OpenOffice, and most other Gnome applications, terminals respond to Ctrl-Shift-u followed by the unicode character number followed by the Enter key). To override the keys Platypus presses, put the code sequence in platypus_keys prefixed by "Ux". For example, 0x0080 (EuroSign) has no corresponding keypress on US keyboards, so the the unicode number in platypus_keys is Ux20ac. Platypus, when it encounters the EuroSign, types Ctrl-Shift-u 20ac Enter. If a symbol you want is not available, override some other key in platypus_keys with the new key, keysym string, A_... AltGr combination, or Ux... unicode number sequence.

International Keyboards©

X11 AltGr keybords support a wider variety of applications than GTK+'sISO 14755-conformant input. Platypus 0.81 can type out most extended characters using International keyboard layouts with AltGr and dead keys. The included platypus_keys.altgr supports the English (International AltGR dead keys) keyboard layout. Other non-English, dvorak, custom keyboard layouts will require creating a matching platypus_keys file. Feel free to use platypus_keys.altgr as a starting point. It should be fairly straightforward, if a little tedious, switching the symbols around.

Those using various other non-US keyboard layouts may need to override letters of the alphabet in platypus_keys with their own. In the future, several versions of platypus_keys may become available through user submission.

To select an international keyboard in Gnome (Metacity), go to Applications==>System Tools==>System Settings==>Keyboard==>Layout Settings. Click the [+] button to add a new keyboard layout. Add English (International AltGR dead keys) and select it. Now rename platypus_keys.altgr to platypus_keys and copy it to /$HOME/.dictation/drive_c/Program Files/Platypus/ (or re-run make install).

If there is an unusual symbol that Platypus can not type out for some reason, worry not. Virtually any character can be typed on a Linux keyboard through custom keyboard layouts.

Multiple X Servers, Multiple Users

A $DISPLAY variable can be passed as an optional argument. Keyboard commands will then go to that particular Xserver display. This could give multiple users access to voice recognition. X authorization may be required.

platypus.exe :0.0
platypus.exe $DISPLAY
platypus.exe :2.0

FAQ

Will this work on mac?

We have not heard of anyone using this on a Mac yet. It should compile and run on any system that has wine and X11 development libraries installed, however. Note: The configure script has only been tested on Fedora so far. It may need editing to work with your particular system.

Does not compile

Requires wine-devel. If cross-compiling Platypus as 32 bit on a 64 bit host, make sure to install glibc-devel.i686 (Redhat) or libc6-dev-i386 (Debian). The configure file only checks a few standard places for libraries. If your libraries are somewhere else you may have to set the path manually with ./configure --libdir option. ./configure --help for options. If all else fails, edit configure and send me the updated file so I can include changes in the next version.

Platypus is not working.

Newer versions of wine may install the 64 bit version of Dragon NaturallySpeaking by default. This may have problems interacting with Platypus compiled as 32 bit (though that could be a wine bug). The interim solution is to either uninstall NaturallySpeaking and re-install it as 32 bit (which could use up a license) or compile Platypus as 64 bit (--with-64bit configure option). The author does not have a 64 bit Dragon NaturallySpeaking to test with, but it should work...

If you play video games on Wine, launch windows applications, or mess with Dragon's settings, Platypus will lose the focus and you will have to click on it to bring the window on top of other windows again. It also helps to run the dictation software and Platypus together within a separate $WINEPREFIX from the one your game or application is on. This is the recommended setup. On some versions of Wine it may be necessary to edit the registry to start the dictation software on the same wine desktop as Platypus. The included .reg file attempts to do this for DNS10 during the install.

But Platypus refuses to stay on top!

Running Platypus in a Wine desktop window is how we get around this problem. The included registry file automatically configures this during installation. The Wine desktop can be minimized, hiding it from view.

Text appears twice.

It is normal for text to appear twice when dictating into Platypus as it converts the "Windows messages" from Dragon NaturallySpeaking into Linux keystrokes. Click on a Linux application to bring that into the foreground and you will be able to dictate text into it.

The cursor is in the wrong place, messing up.

Sometimes the dictation buffer can fill up or go out of sync and it may be necessary to clear it. This can be done through the application menu, (Alt_L a r) or by voice command, "reset dictation". Another voice command hard-coded into the source is "undo all that" which undoes recent editing (as if you had pressed Control_L z in both windows).

Text is all jagged.

CHANGELOG

TECHNICAL

Platypus is a strange creature that lives in the swampy marshland between Windows and Linux, taking over the X display, sending keystrokes and making Linux system() calls from within a Windows application inside the Wine desktop environment. This might scare some security-minded people. News flash: Wine is not a sandbox.

Running make install with a different $WINEPREFIX exported in the terminal is supported.

Platypus can be made into a Windows-only application by linking with Windows' msvcrt -mno-cygwin compiler option with winegcc or MingW "make mingw". But then it will need the full pathname to launch anything other than Windows applications. And since there is no X server to send keys to, it will output spoken text to the terminal.

The file, platypus_keys can be edited to include keysims for your particular keyboard. Later definitions overwrite earlier ones and only the last 2 hexidecimal digits are meaningful (bitmask 0x00ff). Beware because of that, if you enter a character like 0xff7a Muhenkan, it will replace 0x007a z and z will be replaced with Muhenkan in everything you dictate. Unicode key sequences start with Ux followed by the unicode number. Keys that require other key combinations to type can be programmed internally by taking any unused keycode and modifying the source so as to press those extra keys whenever that keycode is encountered. For example, 0x0014 Left is a made-up shifted version of the left arrow key in our implementation.

If you have an X server installed in Windows along with the necessary development libraries (cygwin?) you could experiment with the -D__WINE__ commandline option to MingW and see if it compiles that way. Then the fake keypresses might actually be generated in Windows. I don't know. Or you could try linking it with the Windows version of wine, equally strange, but I'd like to hear about it if it works.

Should you experience trouble with missing characters as you type, there is a Delay setting in xkeys.h that you can set and re-compile. The default is 0. Try something like 5, 10 or 50.

Windows® is a registered trademark of The Microsoft Corporation. Linux® is a registered trademark of Linus Torvalds in the US and other countries. Mac® is a trademark of Apple Inc., registered in the U.S. and other countries. Any other trademarks mentioned belong to their rightful holders. The owner of this site disclaims any affiliation with the provided links.

This documentation is Copyright © 2010 Henry Kroll III. You are free to share, adapt and re-distribute this work with attribution under the terms of the Creative Commons SA license inasmuch as it agrees with the terms of the LGPL. Any of the above conditions can be waived if you get permission from the copyright holder.

Tweet