Push-to-talk dictation: voxtype with exclusive evdev capture, whisper large-v3-turbo served from ef, and a rime-dictionary post-processor
README.md

Dictation using Voxtype #

Hold Meta+V anywhere, speak, release. The words appear at the cursor of whatever window has focus. English, Chinese, and Japanese can be mixed inside one sentence. The audio crosses the LAN to ef and comes back as text; nothing leaves the house.

This document covers how that pipeline is put together on my laptop (X11 + QTile, Intel i5-10300H, NVIDIA GTX 1650 Ti, PipeWire) and how to reproduce it.

The files beside it are the working copies: config/config.toml is the live voxtype configuration, config/dinit/ holds the user service and its env file, and ef/ holds whisper-server, the OpenAI shim and its unit file.

What actually runs #

Piece Role
voxtype daemon push-to-talk voice-to-text
evdev listener owns the Meta+V chord at kernel level; no compositor binding
uinput keyboard mirrors re-emit keystrokes minus the chord, so the app never sees it
Whisper large-v3-turbo on ef multilingual speech recognition, 1.6 GB, RX 7650 GRE, over the LAN
voxtype-osd-gtk4 waveform overlay while recording
xclip + xdotool clipboard write, then a synthetic Ctrl+V
voxtype-rime post-processor: rime user dictionary, then punctuation

The flow: hold the chord → record 16 kHz mono from the PipeWire default source → release → transcribe → post-process → paste at the cursor. A single recording is capped at 60 s. voxtype status prints idle, recording, or transcribing.

Corrections from the IME's dictionary #

The recogniser cannot be given a dictionary. SenseVoice is a SAN-M encoder with a CTC head, so there is no decoder to warm-start and nothing in the ggml runtime that takes a hotword list; what a rime user dictionary holds is a pinyin key and a per-word weight, which is what an IME needs to rank candidates, not what an acoustic model consumes.

What can be used is the pinyin, after recognition. A misheard word keeps all but one of its syllables, and rime wrote down which syllables each word it knows is typed with:

开饭时间   ->  开放时间        kai fan shi jian -> kai fang shi jian

voxtype-rime reads the user-dictionary snapshot rime writes on sync (~/.local/share/fcitx5/rime/sync/<id>/luna_pinyin.userdb.txt) and the schema dictionaries, for the readings rime accepts per character, then rewrites a window only when it reaches every syllable of a stored word but one, and that one is a single sound away. It is wired in through voxtype's own hook:

[output.post_process]
command = "/home/user/cairn/voxtype-rime/bin/voxtype-rime"
timeout_ms = 20000

The same command adds punctuation. SenseVoice punctuates per utterance, so a short clip comes back closed (开饭时间早上9点至下午5点。) and a long fast run comes back as one bare stream; FunASR's CT-Transformer punctuator fills the marks in, and only on text that carries no sentence end of its own.

开饭时间早上9点至下午5点我们明天上午十点开会讨论这个项目的进度
-> 开饭时间早上9点至下午5点,我们明天上午十点开会,讨论这个项目的进度。

It does not choose between two words the dictionary stores under one pinyin, it does not rescue a word that gained or lost a syllable, and it does not touch three-character windows (the measured false positive was 供養無 -> 供養佛, since 無 also reads mo). Without opencc installed a traditional entry stays traditional inside simplified text. Both stages together take about 0.7 s, most of it loading the 281 MB punctuation model; --no-punct and --no-repair drop either one, and --explain prints each rewrite to stderr, which lands in the daemon's log.

A span the dictionary cannot speak about is left exactly as it arrived, which is the property that makes this safe to put in front of every dictation: on 2 524 characters of correct prose, in both simplified and traditional, it rewrote nothing.

The tool is voxtype-rime.

Why a source build #

Two things forced local patches, and neither is a packaging accident:

  • QTile cannot bind a held chord. Its keybinds are X grabs, so a binding fires once on press, and hold-to-record is not expressible in the config. The daemon therefore listens on evdev directly and owns the chord itself.
  • A chord owned by evdev also reaches the focused application, and Chromium inserts a literal v into whatever text box is open. The exclusive-capture patch is what stops that.

The rest is upstream 1.0.1 with three open PRs attached: the xdotool output driver, the clipboard fallback on the paste path, and the fix for a chord whose modifier is released first (which used to leave the recording running until the 60 s cap). What runs here is that tree, on the fork's custom branch:

Commit Patch Upstream
7580f98 xdotool output driver (X11 via XTEST, also works in XWayland) PR #559
a4a6eaa paste chain falls back to the clipboard when the modifier guard skips the keystroke PR #588
ae050ce stop recording when the modifier is released before the hotkey key PR #589
d641cd8 [hotkey] grab = true: exclusive EVIOCGRAB capture, chord withheld from applications not upstreamed, only in custom

The last one is mine and is not upstreamed; the other three are waiting in review. All four are on the fork's custom branch, which is the tree this machine runs: iacore/voxtype.

An older 0.7.5 install is still in /usr/lib/voxtype, with /usr/bin/voxtype pointing at it; that variant was hand-patched at the time to add xdotool, which is why pacman -Qkk voxtype reports a checksum mismatch. Nothing uses any of it now.

Reproducing #

1. Packages #

# build toolchain
sudo pacman -S rustup alsa-lib clang cmake pkgconf patchelf

# runtime
sudo pacman -S xdotool xclip ydotool vulkan-icd-loader

# on-screen visualiser
sudo pacman -S gtk4 gtk4-layer-shell

# evdev hotkeys and exclusive capture
sudo usermod -aG input "$USER"

Log out and back in. The input group matters twice: /dev/input/event* for the hotkey listener, and /dev/uinput (mode 0660, group input) for the keyboard mirrors that exclusive capture creates.

2. Source tree with the four patches #

git clone -b custom https://github.com/iacore/voxtype
cd voxtype

custom is upstream 320a737 (the merge of PR #746) plus the four patches above, and two follow-ups that only touch printed help text, so it is the tree that runs here.

Three of those patches are also open against upstream dev, if the goal is to get them merged rather than to run them — the fork branch can be rebuilt from public refs:

git clone https://github.com/peteonrails/voxtype
cd voxtype && git checkout -b custom 320a737
git fetch origin pull/559/head:pr559 pull/588/head:pr588
git fetch https://github.com/iacore/voxtype fix/hotkey-release-without-modifier:release-and-grab
git merge pr559 pr588 release-and-grab

That merge applies cleanly and produces the same code; it lags the fork branch only by two cosmetic commits — the xdotool row in voxtype setup check and the driver order printed by voxtype config show.

3. ONNX Runtime #

The ONNX engines (SenseVoice, Paraformer, Dolphin, Omnilingual) link against ONNX Runtime. Here it is linked dynamically against a local prefix, which keeps the binary small and lets the runtime be replaced without a rebuild:

mkdir -p ~/.local/lib/onnxruntime
curl -L https://github.com/microsoft/onnxruntime/releases/download/v1.24.4/onnxruntime-linux-x64-1.24.4.tgz \
  | tar -xz --strip-components=1 -C ~/.local/lib/onnxruntime

4. Build #

export ORT_LIB_PATH=$HOME/.local/lib/onnxruntime/lib
export ORT_PREFER_DYNAMIC_LINK=1

cargo build --release \
  --features "sensevoice,paraformer,dolphin,omnilingual,ml-diarization,gpu-vulkan" \
  --bin voxtype

# bake the runtime location into the binary (the build only records -L)
patchelf --set-rpath "$HOME/.local/lib/onnxruntime/lib" target/release/voxtype

install -Dm755 target/release/voxtype ~/.local/lib/voxtype/voxtype

gpu-vulkan is what gives Whisper the GTX 1650 Ti; SenseVoice and the other ONNX engines run on the CPU. voxtype info variants prints the feature set that ended up in the binary.

The waveform overlay is a separate build, and the daemon spawns it as a sibling of its own executable, so both files have to sit next to voxtype:

cargo build --release --features osd-gtk4 --bin voxtype-osd --bin voxtype-osd-gtk4
install -Dm755 target/release/voxtype-osd target/release/voxtype-osd-gtk4 \
  ~/.local/lib/voxtype/

Before switching feature sets, run cargo clean. Upstream warns about this loudly, and rightly: with stale artifacts the binary still compiles, still reports the right version, and silently runs without GPU or ONNX support.

5. Models #

voxtype setup --download --model large-v3-turbo

That lands in ~/.local/share/voxtype/models/ggml-large-v3-turbo.bin, 1.6 GB. --model sensevoice-small fetches the 239 MB SenseVoice model into its own directory instead, and downloads never change which engine loads unless --activate is passed. voxtype info models lists what each engine has installed.

6. Configuration #

~/.config/voxtype/config.toml. Everything omitted is a default; the file that ships with a fresh install is a commented-out copy of every option, and voxtype config schema lists the keys that the CLI can reach directly.

state_file = "auto"          # $XDG_RUNTIME_DIR/voxtype/state, for `voxtype status`
engine = "whisper"

[hotkey]
key = "EVTEST_47"            # kernel keycode 47 = V (`evtest` prints "Event code 47 (KEY_V)")
modifiers = ["LEFTMETA"]     # so the chord is Meta+V
grab = true                  # exclusive capture; the chord stops leaking to applications

[audio]
device = "default"
sample_rate = 16000
max_duration_secs = 60

[output]
mode = "paste"               # clipboard + synthetic Ctrl+V
fallback_to_clipboard = true
wait_for_modifier_release = true   # send the paste after Meta comes up, not while it is held
modifier_release_timeout_ms = 2000

[whisper]
mode = "remote"                                  # transcription runs on ef
remote_endpoint = "http://192.168.1.32:8090"
remote_model = "large-v3-turbo"
model = "large-v3-turbo"                         # used if mode goes back to "local"
language = "auto"            # detect per utterance; a list would pin the first entry
remote_send_auto_language = true  # send language=auto, not the server default
threads = 8
flash_attention = true
on_demand_loading = false

[sensevoice]                 # kept for switching back; unused while engine = "whisper"
model = "sensevoice-small"
language = "auto"            # per-utterance identification: en, zh, ja, ko, yue
use_itn = true               # punctuation and written-out numbers
threads = 8
on_demand_loading = false

Four of these earn their keep:

  • grab = true (exclusive capture). Every keyboard is held with EVIOCGRAB and its events are re-emitted through a uinput virtual keyboard that voxtype owns, minus the hotkey chord. Without it Meta+V reaches the focused application as well, and Chromium inserts a stray v into whatever text box is open. In xinput list this shows up as one voxtype key mirror per grabbed keyboard — five here.
  • wait_for_modifier_release: a paste fired while Meta is still down arrives as Super+Ctrl+V, which nothing binds, so the text is silently lost.
  • on_demand_loading = false: the alternative would unload an idle model and pay roughly two seconds on the first dictation after every pause.
  • engine = "whisper" with language = "auto". SenseVoice tags one language per utterance, and a sentence carrying a single English word comes back either with that word turned into characters or with the Chinese around it degraded. Whisper with the language fixed keeps commit as commit; whisper in remote mode pins whatever the first element of a language list says, so a list has to be replaced by language = "auto" plus remote_send_auto_language = true: with ["zh", "en"] an English sentence came back in Chinese characters. Measured: SenseVoice 0.4 s and 229 MB, whisper large-v3-turbo 2.5 to 3.4 s and 1.6 GB, both including model load per run. Names and English words now survive, at the cost of VRAM and battery.

Transcription on ef #

The 1650 Ti in this laptop takes 2.1 s for a 5.6 s clip and holds the whole 4 GB of VRAM while it does. ef does the same work in 0.45 s of model time, so voxtype runs in mode = "remote": it records here, posts the WAV to ef over the LAN, and types the text it gets back. End to end that is 0.8 s to 1.1 s for a sentence, and the spread is round-trip latency rather than payload: a 16 KB request costs the same 0.59-0.94 s over the LAN as a 179 KB one, against 0.45 s on ef's own loopback. ping between the machines reads min 3-4 ms but averages 171-182 ms with maxima near 500 ms, so the link gives either 4 ms or half a second depending on the moment. No SSH is in the path: the endpoint is a LAN address, ip route get 192.168.1.32 resolves to dev wlan0, and there is no local listener or port-forward process.

ef was on 2.4 GHz (tcpip, 2412 MHz) while 17 sits on tcpip-5G (5180 MHz, -52 dBm), so every request crossed bands at the access point. Moving ef onto 5 GHz on 2026-11-07 — nmcli connection clone tcpip tcpip-5G, set the SSID, autoconnect-priority 10, activate — cut the round trip from avg 171 ms (mdev 148, max 500) to avg 30 ms (mdev 23, max 75), and the slowest requests from 1.0 s to 0.69 s. The remaining 0.15 s over loopback is still there and still independent of payload size.

iw dev wlan0 get power_save says on on both machines; that is the next lever, and the only one left short of a cable.

17 keeps the dictionary repair and the punctuation stage; only the audio crosses the wire, unencrypted, on a wired-free home network.

On ef that is whisper.cpp's whisper-server, plus a thirty-line shim because whisper-server serves /inference while voxtype posts to /v1/audio/transcriptions. The trap worth writing down: ef has two Vulkan devices and ggml takes device 0, which is the Ryzen's integrated Radeon. The same clip took 6.9 s there and 0.77 s on the discrete RX 7650 GRE, so the server is started with -dev 1. The service files and the measurements are in deploy/ef-whisper.

7. Autostart #

This machine has no systemd; the user session runs under dinit --user. ~/.config/dinit.d/voxtype is the whole service:

type = process
# Custom build: upstream v1.0.1 + PRs #559 (xdotool output driver), #588, #589,
# plus the SenseVoice/ONNX engines. NOT the /usr/bin/voxtype pacman binary.
command = /home/user/.local/lib/voxtype/voxtype
env-file = xserver.env

with ~/.config/dinit.d/xserver.env holding DISPLAY=:0, since xdotool needs a display but the daemon can start before X is up. Enabling it drops the symlink into boot.d:

dinitctl enable voxtype     # once
dinitctl start voxtype
dinitctl restart voxtype    # after every rebuild

QTile starts xserver.target from its hooks, and that target has waits-for.d = xserver.d, so the empty marker file ~/.config/dinit.d/xserver.d/voxtype makes the X session start the daemon as well.

8. Verify #

voxtype setup check                       # dependency and output-chain report
voxtype status                            # idle
xinput list | grep -c 'voxtype key mirror'   # one per grabbed keyboard

voxtype record toggle                     # for scripts and keybindings; the chord does the same

Then hold Meta+V, say something, and let go.

Gotchas #

  • Exclusive capture is per keyboard and fails soft. If /dev/uinput is not writable, or another program (keyd, kmonad, interception-tools) already holds a keyboard, that keyboard is skipped: dictation still works, but the chord reaches applications again. Startup logs one line, Exclusive capture active: N of M keyboard(s) grabbed.
  • While it is on, every keystroke in the session is routed through voxtype's virtual keyboards. A broken build therefore means broken typing, which is why there is a copy of the previous binary in ~/.local/lib/voxtype-backup/.
  • The chord is grabbed at the kernel level, so do not also bind it in QTile.
  • If Meta is still held two seconds after the key is released, the paste is skipped and the text only reaches the clipboard.
  • The pacman package stays installed for its /usr/lib/voxtype variants, but its sidecars are 0.7.5 and must not be mixed with the 1.0.1 daemon.
  • When the three open PRs land, only the exclusive-capture patch is left to carry. PR #589 currently bundles it with the release-order fix, so upstream may split it out during review.