- Python 100%
Dictation using Voxtype #
Hold Meta+V anywhere, speak, release. The words appear at the cursor of
whatever window has focus. English, Chinese, and Japanese can be mixed inside
one sentence. The audio crosses the LAN to ef and comes back as text; nothing
leaves the house.
This document covers how that pipeline is put together on my laptop (X11 + QTile, Intel i5-10300H, NVIDIA GTX 1650 Ti, PipeWire) and how to reproduce it.
The files beside it are the working copies: config/config.toml is the live
voxtype configuration, config/dinit/ holds the user service and its env file,
and ef/ holds whisper-server, the OpenAI shim and its unit file.
What actually runs #
| Piece | Role |
|---|---|
| voxtype daemon | push-to-talk voice-to-text |
| evdev listener | owns the Meta+V chord at kernel level; no compositor binding |
| uinput keyboard mirrors | re-emit keystrokes minus the chord, so the app never sees it |
Whisper large-v3-turbo on ef |
multilingual speech recognition, 1.6 GB, RX 7650 GRE, over the LAN |
voxtype-osd-gtk4 |
waveform overlay while recording |
xclip + xdotool |
clipboard write, then a synthetic Ctrl+V |
voxtype-rime |
post-processor: rime user dictionary, then punctuation |
The flow: hold the chord → record 16 kHz mono from the PipeWire default source
→ release → transcribe → post-process → paste at the cursor. A single recording
is capped at 60 s. voxtype status prints idle, recording, or
transcribing.
Corrections from the IME's dictionary #
The recogniser cannot be given a dictionary. SenseVoice is a SAN-M encoder with a CTC head, so there is no decoder to warm-start and nothing in the ggml runtime that takes a hotword list; what a rime user dictionary holds is a pinyin key and a per-word weight, which is what an IME needs to rank candidates, not what an acoustic model consumes.
What can be used is the pinyin, after recognition. A misheard word keeps all but one of its syllables, and rime wrote down which syllables each word it knows is typed with:
开饭时间 -> 开放时间 kai fan shi jian -> kai fang shi jian
voxtype-rime reads the user-dictionary snapshot rime writes on sync
(~/.local/share/fcitx5/rime/sync/<id>/luna_pinyin.userdb.txt) and the schema
dictionaries, for the readings rime accepts per character, then rewrites a
window only when it reaches every syllable of a stored word but one, and that
one is a single sound away. It is wired in through voxtype's own hook:
[output.post_process]
command = "/home/user/cairn/voxtype-rime/bin/voxtype-rime"
timeout_ms = 20000
The same command adds punctuation. SenseVoice punctuates per utterance, so a
short clip comes back closed (开饭时间早上9点至下午5点。) and a long fast run
comes back as one bare stream; FunASR's CT-Transformer punctuator fills the
marks in, and only on text that carries no sentence end of its own.
开饭时间早上9点至下午5点我们明天上午十点开会讨论这个项目的进度
-> 开饭时间早上9点至下午5点,我们明天上午十点开会,讨论这个项目的进度。
It does not choose between two words the dictionary stores under one pinyin, it
does not rescue a word that gained or lost a syllable, and it does not touch
three-character windows (the measured false positive was 供養無 -> 供養佛,
since 無 also reads mo). Without opencc installed a traditional entry
stays traditional inside simplified text. Both stages together take about
0.7 s, most of it loading the 281 MB punctuation model; --no-punct and
--no-repair drop either one, and --explain prints each rewrite to stderr,
which lands in the daemon's log.
A span the dictionary cannot speak about is left exactly as it arrived, which is the property that makes this safe to put in front of every dictation: on 2 524 characters of correct prose, in both simplified and traditional, it rewrote nothing.
The tool is voxtype-rime.
Why a source build #
Two things forced local patches, and neither is a packaging accident:
- QTile cannot bind a held chord. Its keybinds are X grabs, so a binding fires once on press, and hold-to-record is not expressible in the config. The daemon therefore listens on evdev directly and owns the chord itself.
- A chord owned by evdev also reaches the focused application, and Chromium
inserts a literal
vinto whatever text box is open. The exclusive-capture patch is what stops that.
The rest is upstream 1.0.1 with three open PRs attached: the xdotool output
driver, the clipboard fallback on the paste path, and the fix for a chord whose
modifier is released first (which used to leave the recording running until the
60 s cap). What runs here is that tree, on the fork's custom branch:
| Commit | Patch | Upstream |
|---|---|---|
7580f98 |
xdotool output driver (X11 via XTEST, also works in XWayland) | PR #559 |
a4a6eaa |
paste chain falls back to the clipboard when the modifier guard skips the keystroke | PR #588 |
ae050ce |
stop recording when the modifier is released before the hotkey key | PR #589 |
d641cd8 |
[hotkey] grab = true: exclusive EVIOCGRAB capture, chord withheld from applications |
not upstreamed, only in custom |
The last one is mine and is not upstreamed; the other three are waiting in
review. All four are on the fork's custom branch, which is the tree this
machine runs: iacore/voxtype.
An older 0.7.5 install is still in /usr/lib/voxtype, with /usr/bin/voxtype
pointing at it; that variant was hand-patched at the time to add xdotool, which
is why pacman -Qkk voxtype reports a checksum mismatch. Nothing uses any of
it now.
Reproducing #
1. Packages #
# build toolchain
sudo pacman -S rustup alsa-lib clang cmake pkgconf patchelf
# runtime
sudo pacman -S xdotool xclip ydotool vulkan-icd-loader
# on-screen visualiser
sudo pacman -S gtk4 gtk4-layer-shell
# evdev hotkeys and exclusive capture
sudo usermod -aG input "$USER"
Log out and back in. The input group matters twice: /dev/input/event* for
the hotkey listener, and /dev/uinput (mode 0660, group input) for the
keyboard mirrors that exclusive capture creates.
2. Source tree with the four patches #
git clone -b custom https://github.com/iacore/voxtype
cd voxtype
custom is upstream 320a737 (the merge of PR #746) plus the four patches
above, and two follow-ups that only touch printed help text, so it is the tree
that runs here.
Three of those patches are also open against upstream dev, if the goal is to
get them merged rather than to run them — the fork branch can be rebuilt from
public refs:
git clone https://github.com/peteonrails/voxtype
cd voxtype && git checkout -b custom 320a737
git fetch origin pull/559/head:pr559 pull/588/head:pr588
git fetch https://github.com/iacore/voxtype fix/hotkey-release-without-modifier:release-and-grab
git merge pr559 pr588 release-and-grab
That merge applies cleanly and produces the same code; it lags the fork branch
only by two cosmetic commits — the xdotool row in voxtype setup check and the
driver order printed by voxtype config show.
3. ONNX Runtime #
The ONNX engines (SenseVoice, Paraformer, Dolphin, Omnilingual) link against ONNX Runtime. Here it is linked dynamically against a local prefix, which keeps the binary small and lets the runtime be replaced without a rebuild:
mkdir -p ~/.local/lib/onnxruntime
curl -L https://github.com/microsoft/onnxruntime/releases/download/v1.24.4/onnxruntime-linux-x64-1.24.4.tgz \
| tar -xz --strip-components=1 -C ~/.local/lib/onnxruntime
4. Build #
export ORT_LIB_PATH=$HOME/.local/lib/onnxruntime/lib
export ORT_PREFER_DYNAMIC_LINK=1
cargo build --release \
--features "sensevoice,paraformer,dolphin,omnilingual,ml-diarization,gpu-vulkan" \
--bin voxtype
# bake the runtime location into the binary (the build only records -L)
patchelf --set-rpath "$HOME/.local/lib/onnxruntime/lib" target/release/voxtype
install -Dm755 target/release/voxtype ~/.local/lib/voxtype/voxtype
gpu-vulkan is what gives Whisper the GTX 1650 Ti; SenseVoice and the other
ONNX engines run on the CPU. voxtype info variants prints the feature set
that ended up in the binary.
The waveform overlay is a separate build, and the daemon spawns it as a
sibling of its own executable, so both files have to sit next to voxtype:
cargo build --release --features osd-gtk4 --bin voxtype-osd --bin voxtype-osd-gtk4
install -Dm755 target/release/voxtype-osd target/release/voxtype-osd-gtk4 \
~/.local/lib/voxtype/
Before switching feature sets, run cargo clean. Upstream warns about this
loudly, and rightly: with stale artifacts the binary still compiles, still
reports the right version, and silently runs without GPU or ONNX support.
5. Models #
voxtype setup --download --model large-v3-turbo
That lands in ~/.local/share/voxtype/models/ggml-large-v3-turbo.bin, 1.6 GB.
--model sensevoice-small fetches the 239 MB SenseVoice model into its own
directory instead, and downloads never change which engine loads unless
--activate is passed. voxtype info models lists what each engine has
installed.
6. Configuration #
~/.config/voxtype/config.toml. Everything omitted is a default; the file
that ships with a fresh install is a commented-out copy of every option, and
voxtype config schema lists the keys that the CLI can reach directly.
state_file = "auto" # $XDG_RUNTIME_DIR/voxtype/state, for `voxtype status`
engine = "whisper"
[hotkey]
key = "EVTEST_47" # kernel keycode 47 = V (`evtest` prints "Event code 47 (KEY_V)")
modifiers = ["LEFTMETA"] # so the chord is Meta+V
grab = true # exclusive capture; the chord stops leaking to applications
[audio]
device = "default"
sample_rate = 16000
max_duration_secs = 60
[output]
mode = "paste" # clipboard + synthetic Ctrl+V
fallback_to_clipboard = true
wait_for_modifier_release = true # send the paste after Meta comes up, not while it is held
modifier_release_timeout_ms = 2000
[whisper]
mode = "remote" # transcription runs on ef
remote_endpoint = "http://192.168.1.32:8090"
remote_model = "large-v3-turbo"
model = "large-v3-turbo" # used if mode goes back to "local"
language = "auto" # detect per utterance; a list would pin the first entry
remote_send_auto_language = true # send language=auto, not the server default
threads = 8
flash_attention = true
on_demand_loading = false
[sensevoice] # kept for switching back; unused while engine = "whisper"
model = "sensevoice-small"
language = "auto" # per-utterance identification: en, zh, ja, ko, yue
use_itn = true # punctuation and written-out numbers
threads = 8
on_demand_loading = false
Four of these earn their keep:
grab = true(exclusive capture). Every keyboard is held withEVIOCGRABand its events are re-emitted through a uinput virtual keyboard that voxtype owns, minus the hotkey chord. Without it Meta+V reaches the focused application as well, and Chromium inserts a strayvinto whatever text box is open. Inxinput listthis shows up as onevoxtype key mirrorper grabbed keyboard — five here.wait_for_modifier_release: a paste fired while Meta is still down arrives as Super+Ctrl+V, which nothing binds, so the text is silently lost.on_demand_loading = false: the alternative would unload an idle model and pay roughly two seconds on the first dictation after every pause.engine = "whisper"withlanguage = "auto". SenseVoice tags one language per utterance, and a sentence carrying a single English word comes back either with that word turned into characters or with the Chinese around it degraded. Whisper with the language fixed keepscommitascommit; whisper in remote mode pins whatever the first element of a language list says, so a list has to be replaced bylanguage = "auto"plusremote_send_auto_language = true: with["zh", "en"]an English sentence came back in Chinese characters. Measured: SenseVoice 0.4 s and 229 MB, whisper large-v3-turbo 2.5 to 3.4 s and 1.6 GB, both including model load per run. Names and English words now survive, at the cost of VRAM and battery.
Transcription on ef #
The 1650 Ti in this laptop takes 2.1 s for a 5.6 s clip and holds the whole
4 GB of VRAM while it does. ef does the same work in 0.45 s of model time, so
voxtype runs in mode = "remote": it records here, posts the WAV to ef over
the LAN, and types the text it gets back. End to end that is 0.8 s to 1.1 s for
a sentence, and the spread is round-trip latency rather than payload: a 16 KB
request costs the same 0.59-0.94 s over the LAN as a 179 KB one, against 0.45 s
on ef's own loopback. ping between the machines reads min 3-4 ms but averages
171-182 ms with maxima near 500 ms, so the link gives either 4 ms or half a
second depending on the moment. No SSH is in the path: the endpoint is a LAN
address, ip route get 192.168.1.32 resolves to dev wlan0, and there is no
local listener or port-forward process.
ef was on 2.4 GHz (tcpip, 2412 MHz) while 17 sits on tcpip-5G
(5180 MHz, -52 dBm), so every request crossed bands at the access point. Moving
ef onto 5 GHz on 2026-11-07 — nmcli connection clone tcpip tcpip-5G, set the
SSID, autoconnect-priority 10, activate — cut the round trip from avg 171 ms
(mdev 148, max 500) to avg 30 ms (mdev 23, max 75), and the slowest requests
from 1.0 s to 0.69 s. The remaining 0.15 s over loopback is still there and
still independent of payload size.
iw dev wlan0 get power_save says on on both machines; that is the next
lever, and the only one left short of a cable.
17 keeps the dictionary repair and the punctuation stage; only the audio crosses the wire, unencrypted, on a wired-free home network.
On ef that is whisper.cpp's whisper-server, plus a thirty-line shim because
whisper-server serves /inference while voxtype posts to
/v1/audio/transcriptions. The trap worth writing down: ef has two Vulkan
devices and ggml takes device 0, which is the Ryzen's integrated Radeon. The same
clip took 6.9 s there and 0.77 s on the discrete RX 7650 GRE, so the server is
started with -dev 1. The service files and the measurements are in
deploy/ef-whisper.
7. Autostart #
This machine has no systemd; the user session runs under dinit --user.
~/.config/dinit.d/voxtype is the whole service:
type = process
# Custom build: upstream v1.0.1 + PRs #559 (xdotool output driver), #588, #589,
# plus the SenseVoice/ONNX engines. NOT the /usr/bin/voxtype pacman binary.
command = /home/user/.local/lib/voxtype/voxtype
env-file = xserver.env
with ~/.config/dinit.d/xserver.env holding DISPLAY=:0, since xdotool needs
a display but the daemon can start before X is up. Enabling it drops the
symlink into boot.d:
dinitctl enable voxtype # once
dinitctl start voxtype
dinitctl restart voxtype # after every rebuild
QTile starts xserver.target from its hooks, and that target has
waits-for.d = xserver.d, so the empty marker file
~/.config/dinit.d/xserver.d/voxtype makes the X session start the daemon as
well.
8. Verify #
voxtype setup check # dependency and output-chain report
voxtype status # idle
xinput list | grep -c 'voxtype key mirror' # one per grabbed keyboard
voxtype record toggle # for scripts and keybindings; the chord does the same
Then hold Meta+V, say something, and let go.
Gotchas #
- Exclusive capture is per keyboard and fails soft. If
/dev/uinputis not writable, or another program (keyd, kmonad, interception-tools) already holds a keyboard, that keyboard is skipped: dictation still works, but the chord reaches applications again. Startup logs one line,Exclusive capture active: N of M keyboard(s) grabbed. - While it is on, every keystroke in the session is routed through voxtype's
virtual keyboards. A broken build therefore means broken typing, which is why
there is a copy of the previous binary in
~/.local/lib/voxtype-backup/. - The chord is grabbed at the kernel level, so do not also bind it in QTile.
- If Meta is still held two seconds after the key is released, the paste is skipped and the text only reaches the clipboard.
- The pacman package stays installed for its
/usr/lib/voxtypevariants, but its sidecars are 0.7.5 and must not be mixed with the 1.0.1 daemon. - When the three open PRs land, only the exclusive-capture patch is left to carry. PR #589 currently bundles it with the release-order fix, so upstream may split it out during review.