Sheng-KaiHuang

CUDA2MLX

Built a CUDA/PyTorch to Apple Silicon MLX porting toolkit that turns repeated model-port work into analyzable tiers, reusable rules, cookbook kernels, and parity gates.

• Designed a four-tier conversion framework covering LLM, vision, 3D, sparse ops, custom Metal kernels, and signal workloads.
• Documented 11/11 E2E parity and 37/37 smoke coverage; TinyViT max diff 8.49e-7 and LLaMA-style block max diff 2.38e-7.
• Ran the analyzer on TRELLIS, ProPainter, LaMa, InstantMesh, and LGM, exposing Tier 1/2/3/4 migration work before porting begins.
• Relevance to Hermes: an agent can use this as a structured playbook for codebase analysis, port planning, parity testing, and repair loops.

MLX Inpaint Ports

Ported image and video inpainting pipelines into MLX for Apple Silicon, using numerical parity and visual outputs as acceptance gates rather than visual plausibility alone.

• LaMa image inpainting: 51M-parameter FFCResNetGenerator, 96.67 dB PSNR against PyTorch on the test image, and 8-12x MLX speedups across useful resolutions.
• ProPainter video inpainting: RAFT optical flow, recurrent flow completion, modulated deformable convolution, sparse transformer, and top-level pipeline wired in MLX.
• Reported full ProPainter pipeline parity at PSNR 59.4 dB vs PyTorch, plus practical ~450 ms/frame at 240x432 in the sliding-window path.
• Relevance to Hermes: strong evidence for model-port debugging, tensor-layout reasoning, parity testing, and Apple Silicon runtime optimization.

Generative Image Speedup

Built a measured image-generation speed study and product UI around Boogu Image, separating the fast product generation path from the slower offline vision-teacher path used by Meadow.

• Kept Turbo-16bit / bf16 as the main image-generation route after comparing Base 4bit, Turbo-8bit, Turbo-bf16, degrid, and poster workflows.
• Measured generation from 121.81s to 12.99s, a 9.38x speedup; DiT denoise moved from 120.61s to 11.23s, a 10.74x speedup.
• Compared Boogu Edit with OpenCV, DINO, YOLOv8n-seg, and SAM, concluding that Boogu belongs in offline labeling, hazard hints, object memory, and rule extraction, not every-frame control.
• Connected the research to Meadow hot paths: Boogu acts as a teacher, while compact detectors and Meadow policies keep runtime decisions in the ms range.
• Relevance to Hermes: agents need to choose model routes, cache expensive work, explain quality and speed tradeoffs, and turn heavy vision models into controllable product workflows.

Open the Boogu Image report

STT and TTS Pipeline

Optimized the speech loop as a pipeline problem: capture, transcribe, normalize, rewrite, synthesize, and expose enough state for the user to repair errors.

• Used waveform-level boundary checks and context decode / declick artifacts to reason about audio continuity.
• Separated latency, repair turns, audio artifacts, and context carry so voice quality is not judged by transcript accuracy alone.
• Relevance to Hermes: voice agents need low-friction recovery, not just a good first answer.

Englisher

Built an English communication surface that treats speaking, translation, rewriting, and correction as one continuous user loop instead of separate tools.

• Designed the meeting translation UI around live captions, bilingual context, voice isolation, notes, and repairable output.
• Compared speech recognition and rewrite paths so the user can see what changed and recover when the model misunderstands intent.
• Relevance to Hermes: agent quality depends on turn-taking, editable output, and trust during imperfect model behavior.

OpenAI Parameter Golf

Participated in OpenAI Parameter Golf / Model Craft Challenge as a solo research line. The final ranking was not strong, but the work produced several useful architecture probes for compact language models under a hard 10-minute training and 16MB artifact budget.

• Best mature competition run: 1.2146 BPB base and 1.1854 compliant no-SLOT eval on the final partial-retrodiction 8xH100 run.
• Built a retrodiction training line: a causal model learns both normal forward prediction and reversed-sequence auxiliary prediction, inspired by recovery-map style reasoning.
• Built a shared-weight AR + masked-denoising line where the same model supports causal drafting and two-pass coarse-to-fine infill; 11L final-checkpoint verification showed -0.0205 +/- 0.005 BPB against a matched causal-only control.
• Explored v4096 tokenizer, TTT, SLOT-style eval-time adaptation, CDM sequential unmasking, model compression, and H100 throughput tradeoffs; the strongest stable eval-time probe reached 0.7688 BPB with SLOT-24 stride=96 on a 1.5M-token / 3% validation slice.
• The key lesson was budget-aware architecture search rather than leaderboard chasing: separate stable competition numbers from lower-confidence eval-time probes, then keep the useful mechanism.
• Relevance to Hermes: agents need the same discipline: honest metrics, route selection, test-time adaptation, and knowing when a novel mechanism is a research signal rather than a product claim.

Open the Meadow Golf research diary

Meadow Core / MLX Runtime

Moved local model work from scripts into reusable runtime surfaces for CLI, desktop, browser, and phone-preview workflows.

• Worked on MLX profiles, vLLM serve bridges, KV cache policy, warm service lifecycle, model switch cleanup, and streaming generation fixes.
• Wrapped local inference as OpenAI-compatible APIs so product surfaces can share the same model contract.
• Measured Gemma 4 12B TTFT from 2.67s to 0.89s and throughput from 17.60 to 20.19 tok/s; shared warm runs reached 397ms.
• Relevance to Hermes: agent UX is shaped by managed inference, streaming behavior, and runtime recovery.

Meadow CLI

Worked on command-line local agent execution where startup time, model service reuse, memory lookup, and reconnect behavior directly change developer experience.

• Kept the CLI contract close to OpenAI-compatible local services so runtime upgrades do not break users.
• Validated local chat flow, reconnect smoke, model status, and MemPalace lookups instead of only checking build success.
• Relevance to Hermes: developer agents need predictable command surfaces and evidence-backed runtime states.

Open the Meadow CLI dynamic demo / Open the actual Meadow Go interface preview

Meadow Desktop

Optimized the desktop layer as a managed local-AI product surface: service lifecycle, user waiting states, response streaming, model switching, and predictable recovery.

• Focused on launch feel, warm service availability, sidecar state cleanup, reconnect snapshots, and streaming completion.
• Separated UI reliability from model quality so the app can explain what is loading, what failed, and what recovered.
• Relevance to Hermes: desktop agents need durable local runtime states, not fragile demo windows.

Open the actual Meadow Go interface preview

Meadow Browser

Explored browser operation as an agent tool: deterministic navigation, DOM reading, visual context, overlay interaction, and shared-core chat fallback.

• Kept navigation prompts local and deterministic while chat uses the shared Meadow Core endpoint.
• Worked on package/runtime reduction, release-only optimization, lazy console creation, and opt-in Web Inspector.
• Relevance to Hermes: real agents live inside browsers, forms, docs, and unstable web apps.

VSMONSTER

Built an early phone-to-VS-Code agent harness concept where mobile and social messages become reviewed coding tasks, with a UFO control center and BlueMonster worker surface.

• Designed the Holography message bridge, UFO Control Center webview, BlueMonster chat/task worker, and review-gated task lifecycle.
• Connected Gateway WebSocket, VS Code extension HTML surfaces, Copilot bridge, MCP control, tunnel service, and task-board concepts.
• Relevance to Hermes: direct experience turning agent workflows into controllable operating surfaces rather than one-off prompts.

Meadow WB Tools

Worked on world-building experiments that convert visual evidence into usable 3D or embodied state: spaces, objects, and skeleton or motion state from video.

• Separated 3DGS environment capture, 3DGS object reconstruction, human motion capture, fast preview, and high-quality SAM 3D branches.
• Measured Meadow Cube 10view + LoRA at 1.6s on M1 Max and SAM 3D Object at 25s to 53.5s, depending on path.
• Measured WB 3DGS planner proxy error reduction across environment, object, and imagined path state.
• Added MLX image/video inpainting ports as repair tools for missing pixels, removed objects, and world-state cleanup.
• Relevance to Hermes: future agents need physical context, object identity, and motion memory.

Meadow WM Framework

Built and evaluated a compact physics-grounded world-model stack for real-time action selection: causal tree teacher policies, compact reaction students, and neural scorers/cost maps.

• Covered object manipulation, arm reaching, and cube pick-and-place tasks across PushT, Reacher, and OGBench-style probes.
• Kept the reaction loop small enough for interactive control: the public report lists 0.095-0.122 ms Core ML reaction inference on Apple Silicon.
• Used benchmark evidence carefully: the report lists PushT, Reacher, and OGBench Cube task-family successes while separating task-contract limits from broader robot-learning claims.
• Relevance to Hermes: agents need cheap action scoring, physical state memory, and recoverable action effects, not only language planning.

Meadow Balancer

Explored biped balancing where all components move toward MLX-backed local inference and a 3D state inventory.

• Framed balance as a real-time control problem with visible state, actuator constraints, and correction timing.
• Connected 3D spatial components to MLX local runtime assumptions for future embodied loops.
• Relevance to Hermes: embodied agents require grounded state and fast correction, not only plan text.

Meadow Motion Transfer

Explored how motion transfers across controlled worlds, where the useful unit is not a prompt but an action pattern that survives environment changes.

• Used angle, contact, and endpoint-error artifacts to inspect candidate motion transfer runs.
• Connected motion transfer back to world state and embodied memory rather than treating it as isolated animation.
• Relevance to Hermes: agents should reuse skills across tools and contexts, not relearn every interface from scratch.

Meadow Game

Used games and compact environments as controlled worlds for action choice, timing failure, reward recovery, and visual feedback loops.

• Kept the focus on action timing and measurable behavior, not only attractive demos.
• Used game probes to expose whether an agent can recover from wrong moves and delayed decisions.
• Relevance to Hermes: game-like probes are compact tests for tool use, planning, and embodiment.

Meadow Mind Latent Predictor

Converted a local 7B diffusion LLM from answer generation into a fixed-latency latent decision layer: state, policy, decision pass, and action.

• Built Perceiver / Rule / Mind / Actuator layers plus sanity-gate checks before deployment.
• Zero-training Gymnasium results include CartPole-v1 400/400, LunarLander-v3 +251, FrozenLake shortest path, and MountainCar flag in 103 steps.
• Added Meadow Memory: binary event log, SQLite projection, and semantic recall for meaningful state deltas.
• Relevance to Hermes: concrete action-loop prototype for agents that grow from interaction instead of only chatting.

Thinking Machines

Designed an active AI presence runtime where camera, screen, voice, chat, recent events, micro-turn policy, background workers, and selective memory share one frame.

• Defined InteractionFrame, PresenceState, MicroTurnDecision, and MicroTurnPolicy contracts.
• Separated foreground presence from background VLM/LLM/tool execution so the user still feels acknowledged while slow work runs.
• Built a benchmark ladder for visual attention, latent reaction, replay repair, and embodied/game probes.
• Relevance to Hermes: an agent should feel co-present, interruptible, and useful across terminal, desktop, and app surfaces.

Shader Music Lab

Built audio-reactive shader instruments where rhythm, synthesis parameters, and visual state influence each other instead of running as separate decoration layers.

• Connected beat, energy, tempo, and shader parameters so the interface behaves like a visual instrument rather than a passive animation.
• Explored 21 generative audio-visual systems as HMI probes: FOCUS, sequencing, realtime control, and understandable state feedback.
• Relevance to Hermes: agent products need feedback surfaces that make hidden state legible and controllable.

Meadow AD

Explored AI-assisted advertising and creative direction as a human-machine interface problem: the system must expose taste, options, constraints, and iteration state.

• Connected visual generation, copy direction, and interaction flow into a usable creative surface.
• Focused on psychological feel: control, surprise, comparison, and the confidence to select.
• Relevance to Hermes: creative agents need interfaces for taste and iteration, not only final image output.

2D & 3D Human-Machine Interface

Explored 2D and 3D human-machine interfaces for generative audio-visual systems, including rhythm-driven visuals, WebGL/Three.js scenes, and a prior brainwave music-selection experiment.

• Built 21-node generative HMI labs where users can scan, focus, orbit, and control complex system states.
• Kept FOCUS UI and 3D HMI captures fixed-camera so the visual system itself moves, not the screenshot crop.
• Designed visual and audio feedback loops that create agency, predictability, and control rather than passive spectacle.
• Relevance to Hermes: strong intuition for what agentic AI should feel like across terminal, web, app, and ambient surfaces.