LW Linji (Joey) Wang

RESEARCH NOTEBOOK / 21 SEPTEMBER 2026

From motion to
useful robot behavior.

A robot must do more than imitate a motion. It must choose a behavior that fits the scene, execute it under real constraints, recover from error, and still finish the task.

This research program connects adaptive training, whole-body traversal, physical feasibility and reliable execution. The central question is what evidence should guide the next decision: what to practice, which motion to request, and when to trust the result.

Scope of this review. Active and foundational robotics/ML projects identified in the workspace and connected research repositories. Related branches and packaging efforts are grouped. This is a synthesis of selected reports and recorded experiments, not a new experiment campaign or a claim to have replayed every historical run. Published studies, development results and prospective ideas remain distinct. Unrelated personal applications and historical forks are outside this research overview.

01 / DEFINE THE PROBLEM

The gap is between an intended motion
and a completed task.

Imagine a humanoid approaching a low beam. A collision-free reference tells us where its body should move. It does not tell us whether the controller will lower the torso enough, keep its feet supported, recover after the beam, or stop at the requested destination.

Training has the same problem. High error can identify a useful challenge, an impossible reference, a broken interface, or a measurement artifact. Sampling the hardest examples more often can reinforce the wrong behavior.

Our research goal

Learn and select coordinated whole-body behavior from a destination, a scene and causal robot feedback, with explicit evidence that the chosen behavior remains physically supported and task-relevant.

Today: separate bounded components and known-map simulation studies.
Long term: robust perception-conditioned traversal and independently qualified hardware execution.

01 · RequestGoal + sceneWhat must be achieved?
02 · DecideMotion + admissionWhich option has support?
03 · ExecuteAdapter + motorWhat does the body do?
04 · EvaluateComplete taskDid it recover and stop?
Conceptual program map. Connections summarize research responsibilities; they do not depict a jointly validated end-to-end system.
Two schematic beam scenes show planned clearance failing in execution and a conservative margin rejecting a motion that executes successfully.
The admission problem has two sides: accepted motions can fail, and rejected requests can have a successful alternative. This illustration is conceptual; the clearance table below contains the measured evidence.

Reference ≠ execution

A planned pose or replay is an instruction or visualization. A controller must realize it through forces and contacts. We score the executed body and the whole task.

Assistance ≠ learned capability

A teacher rescuing a learner demonstrates a possible continuation. Improvement is established only when the updated student succeeds without that help.

Difficulty ≠ learning value

A useful curriculum must account for feasibility, retained skills, information access and cost. More error or more samples alone do not identify the best next practice.

02 / WHAT THE EXPERIMENTS SAY

Progress includes learning
which assumptions fail.

These comparisons belong to different protocols. Select one to inspect its own units, denominator and interpretation. They are not a shared leaderboard.

Recorded results, with experiment-specific interpretation
StudyResultWhat it changesSource
KimoNav clearanceLocal duck 5/6; IK lowering 5/6. Geometry shadow coverage 2/6; finite ledger 0/6.Available execution exceeds admitted coverage. Lowering adds no success; calibrated geometry is a necessary rival to a learned envelope.All requests
Hindsight distanceContinuous 2/2; Linear29 1/2.A missed one-time clearance check can decide the task; representation value must be measured through execution.Record
Motion2Scene learningInitializer 10/12; demonstration replay 10/12; history correction 8/12.Executable corrective labels did not earn an improved student. Retention is part of the learning problem.Synthesis
Terrain transferStarting policy 56 soil failures; staged 53; direct 54.Shallow practice became possible, but both training candidates failed promotion. Counts include resets, not independent trials.Decision
Flow checker20.136 vs 38.147 ms per valid return; both preserve 375/432 returns.A simple uniform prefilter plus exact fallback is a strong cost baseline. These are descriptive CPU batch timings.Report
LUCID replicationPrimary PLR effect +1.82 points, then a tie across two new student seeds.The registered directional screen fails. Secondary stronger-stress gains need independent confirmation.Replication

03 / PROJECT-BY-PROJECT REVIEW

One program.
Different scientific questions.

Every track below states the problem, goal, method, current evidence and an experiment that could change our view. “Next research” is a proposal unless a source explicitly records completion.

18 tracks shown

1

Traversal / Development simulation

KimoNav

Which whole-body motion can a robot actually execute to reach a destination through clutter?

Research goal

Coordinate route, posture, feet and speed, then recover upright and stop inside the destination region.

Method

Use a known map, goal and measured initial state to choose a qualified reference from a small bank. A native reference adapter feeds a frozen SONIC encoder/decoder and proprioceptive tracker. Current selection happens at startup; online replacement and perception are proposed.

Findings and current progress

Six new public requests completed in three familiar contexts and exactly matched their selected fixed controls. In a separate 12-execution clearance diagnostic, local duck and IK-lowered references each completed 5/6 requests. Geometry shadow admission covered only 2/6 requests; the finite ledger admitted none. Extra lowering added no task successes.

Evidence boundary

One inspected source ancestry and corridor. The latest shadow choices are not public rollouts or safe-stop tests. The successful public connection uses a different motor from older failed panels; the change cannot be credited to the planner alone.

Next research

Compare a calibrated scalar geometry margin with an execution-envelope predictor on separately frozen requests. Replace the hard-coded source-name assumption with stable bank metadata and qualify a second source before expanding geometry and entry conditions.

What would change the decision?

Prefer the simple margin if it matches useful coverage at the same admitted-failure rate. Add learned state dependence only if it improves separate-condition results without privileged dynamics, seed IDs or future trajectories.

Current project ↗ · Every clearance outcome ↗ · Plan ↗ · Public integration ↗

2

Traversal / Development simulation + preparation

Hindsight Motion / Student 01

Can execution-verified motion experience teach a policy to choose useful motion from a goal and scene?

Research goal

Learn complete approach–duck–recover–stop behavior, then compare hindsight-generated data with scene-first data at matched acquisition cost.

Method

Separate motion representations from their downstream usefulness. Retain continuous and Linear29 controls; first prepare a causal known-map behavior-cloning student using complete episodes and measured history.

Findings and current progress

Supplied same-phase continuations passed 8/8 per method. A separate scene screen passed 1/3 per method. The distance-responsive comparison passed continuous 2/2 versus Linear29 1/2: the farther-goal case missed a one-time clearance decision. A later memory-admission window launched no new native run. September 21 shifts priority to Student 01 preparation.

Evidence boundary

These are small, related development tasks. Supplied-reference and selected-reference successes do not prove learned navigation. The pending repair remains unqualified; CPU fit preparation is not a closed-loop result.

Next research

Audit complete episodes and causal pre-action inputs, fit the bounded development baseline, validate reference packing and rotations, then register a separate native student evaluation. Freeze new ancestry/scene splits before any transfer claim.

What would change the decision?

Continue to data/representation comparisons only after the student executes a complete task unaided. Keep a discrete head only if it improves execution, data efficiency or latency over continuous controls.

Project repository ↗ · Reviewed evidence ↗ · Student 01 protocol ↗

3

Traversal / Mixed development results

Motion2Scene / recursive student learning

Do corrections collected from a learner’s actual mistakes improve the next unaided student?

Research goal

Build a repeatable acquisition–qualification–learning cycle above a frozen motor while retaining previously successful behavior.

Method

Replay causal learner histories; execute teacher and motor continuations; admit only labels supported by complete-task and changed-command tests; continue from the incumbent with replay controls; evaluate without assistance.

Findings and current progress

The latest coherent-history panel records initializer 10/12, demonstration replay 10/12 and history correction 8/12. Assisted continuations reached 12/12, but that support did not become an improved student. A separate goal/map scene-diversity study completed 3/24 tasks in each arm, with all shifted cases failing.

Evidence boundary

Teacher support, inherited full-command capability and learned sparse control are different outcomes. The four sparse commands are reference-derived; they do not establish autonomous goal/map navigation. These panels must not be pooled.

Next research

Measure retained successes and individual regressions alongside correction response. Compare matched old-data replay, nominal demonstrations and learner-history corrections with the same initializer, optimizer and acquisition accounting.

What would change the decision?

Promote only an unaided student that earns complete-task gains while meeting declared retention criteria. If replay matches corrections, investigate coverage and optimization before collecting more expensive labels.

September 17 synthesis ↗ · Code and research ↗

4

Traversal / Qualification gate failed

Motion2Scene / inverse scene construction

Can a successful body motion tell us which obstacles make that motion necessary?

Research goal

Construct minimal critical scenes and matched alternatives, producing useful scene–motion supervision with physical preference reversal.

Method

Generate matched neutral/adapted motion carriers, build ordered beam-clearance ladders, and verify reference semantics, tracker survival, route retention and obstacle-conditioned preference in stages.

Findings and current progress

The fresh eight-carrier validation admitted 8/8 references, retained 7/8 tracker survivors and passed relative route retention in 5/7 survivors. It missed the registered 80% criterion. The matched reference ladders exist, but their next physics pilot remains gated.

Evidence boundary

A beam interval that clears a reference is not an executable demonstration or proof that ducking is necessary. Earlier absolute-route measurements were sensitive to measurement scale; their frozen outcomes remain unchanged.

Next research

Repair and separately validate route-retention measurement and carrier execution before spending on preference-reversal or learned scene generation. Compare against simple factorized geometry construction.

What would change the decision?

Only promote pairs where both controls are physically supported and obstacle changes alter the useful behavior under the same task. Stop if a simple sampler supplies equivalent qualified data.

Research record ↗ · Training package ↗

5

Traversal / Offline proxy experiments

Motion2Scene Flow / geometry and generation

What geometric information must a generator preserve, and how cheaply can its candidates be checked?

Research goal

Generate diverse useful traversal candidates while retaining the body–obstacle relationships that determine clearance.

Method

Compare scene readouts and conditional flow matching on continuous crouch/yaw/lateral-shift controls. Use conservative interval certificates, exact geometry fallback and explicit query/runtime budgets.

Findings and current progress

On the reused 13,824-candidate bank, hybrid checking retained all 9,962 valid accepts and 375/432 scene–model returns with identical selected actions. Uniform16 plus fallback measured 20.136 ms per valid return versus exact checking’s 38.147 ms for the checker. A prior bounded-output comparison did not beat simple clipping on candidate validity.

Evidence boundary

The 432 evaluations reuse three fits across 144 scenes. Times are CPU batch measurements, not online robot latency. Geometry certificates apply to the declared interpolant and authored geometry, not tracking safety. No native motor execution is established.

Next research

Freeze the checker choice and test new scenes. Return to the generator: compare its candidates with independent in-domain search at matched checking cost on the 57 scene–model cases with no valid candidate.

What would change the decision?

If search finds valid actions the generator misses, study coverage or conditioning. If neither finds them, report “not found,” not “infeasible”; test whether the three-control representation itself is too restricted.

Full experiment sequence ↗ · Hybrid checker report ↗

6

Traversal / Negative eligibility evidence

Scene2Motion-G1 / frozen-prior control

Does a frozen motion prior expose enough reliable control to place a body event where a scene needs it?

Research goal

Turn scene geometry and a route into physically executable whole-body adaptation programs.

Method

Audit command effects against sampling variation, then align coherent absolute/residual step-event programs to gait phase. Freeze sampler versions, cache identity and event-placement limits.

Findings and current progress

The candidate-pool preflight found only 1/8 eligible nominal motions and generated no final comparison arms. Exact replay showed a required +65-frame placement shift outside the locked ±8-frame bound in an earlier case.

Evidence boundary

This is evidence about addressability and eligibility, not a measured absolute-versus-residual benefit. Earlier long-clip results used a seed-restarting sampler and require version-specific interpretation.

Next research

Couple route progress to gait phase explicitly and test whether a supported event can be placed under fixed scene/task requirements. Preserve the old failed pool rather than extending it after inspection.

What would change the decision?

Proceed to adaptation comparisons only if the interface can produce an eligible cohort. If placement remains unsupported, simplify to measured body modes or change the motion interface.

Repository ↗ · Reviewed method/status ↗

7

Physics / Training completed; neither arm promoted

Newton terrain / learning on deformable support

Can shallow-soil practice teach a humanoid to retain tracking on deeper deformable terrain?

Research goal

Learn stable, contact-rich motion tracking under changing support while preserving rigid-ground performance.

Method

Couple native humanoid dynamics to Newton MPM material simulation. Hold the SONIC teacher fixed and train a leg residual; compare staged 2/6/14/14 cm practice with direct 14 cm training under matched retained budgets.

Findings and current progress

Original-depth soil failure counts were 56 for the starting policy, 53 after staged practice and 54 after direct training. Both candidates failed the usefulness and tracking gates. Two-centimeter practice was executable, but useful transfer to 14 cm soil was not demonstrated.

Evidence boundary

These are reset-containing fixed-duration failure counts, not independent success-rate trials. Three reused development clips; uncalibrated soils; one-cell shallow depth. Interrupted staged attempts consumed extra discarded compute despite equal retained budgets.

Next research

First compare explicit full-reference starts mixed with random-phase starts against current sampling. Then test competence-based depth promotion with target-depth rehearsal. Audit coupling mass/energy and material/grid sensitivity before scaling.

What would change the decision?

Advance only with lower target-depth failure counts, retained tracking, adequate material contact and ground retention. More surviving reference-clock endpoints alone do not qualify a policy.

Training decision ↗ · Integration history ↗ · Terrain project ↗

8

Physics / Measured screening; learning benefit unresolved

CLIMB + refeas / feasibility before curriculum

When a tracker fails repeatedly, is the motion useful practice or physically unsupported?

Research goal

Prevent adaptive sampling from concentrating on references whose required wrench lacks an admissible contact/actuator source.

Method

Screen robot-space references with explicit physical assumptions, route them to admit/contextualize/repair/quarantine, enumerate legal starts and apply learning-progress selection only inside a feasibility gate.

Findings and current progress

The inspected corpus screen flags 2,442/10,705 clips (22.8%); a separately filtered pairing has 7/4,950 (0.14%). A bounded repair panel qualifies 22/26 candidates and preserves 4/4 controls. The newer status records inconclusive allocation evidence and an admission campaign that trained but stopped before evaluation. Actuator-limit sensitivity moved flags from 99 to 98 of 900, refuting the registered monotonicity prediction.

Evidence boundary

Corpus percentages are pipeline-dependent. Repair qualification is not policy improvement. Feasibility is model-relative; the screen’s translational residual is not guaranteed to increase when actuator limits tighten. The failed Newton predictive gate did not qualify that instrument for its proposed training role.

Next research

Compare gated uniform sampling against progress-based allocation under the same eligible units, delivered exposure and training budget. Preserve sealed failures and distinguish screening accuracy, coverage and downstream policy value.

What would change the decision?

If allocation does not improve retained performance beyond the gate, keep the feasibility instrument and simplify the sampler. Do not reinterpret model conformance as predictive usefulness.

Project ↗ · Reviewed evidence ↗ · refeas tool ↗ · Updated public status ↗ · Updated status extract ↗

9

Learning / Preprint + separate continuation research

LUCID / robustness, retention and recovery

How do we make disturbances harder without teaching a robot to abandon the intended motion?

Research goal

Improve disturbance tolerance while retaining nominal tracking and recovering to the task after perturbations.

Method

The preprint uses command–execution history, a frozen temporal encoder and bounded PI/backoff domain-randomization scheduling. Separate continuation studies test anchoring and fresh-event uniform versus prioritized level replay (PLR). These are distinct methods and records.

Findings and current progress

In the fresh-event replication, PLR’s primary gain was +1.82 points in one training seed and a tie in the other, failing the frozen directional criterion. Both methods passed retention checks. Stronger-stress gains were secondary findings and did not replace the failed primary endpoint.

Evidence boundary

Two new student seeds are not a precise algorithm-level estimate. Survival and tracking quality can disagree; pre-termination errors use different observation lengths. The existing preprint’s results do not validate these newer continuation methods.

Next research

Build event-aligned, pre-reset recovery measurement with fixed horizons, dwell and censoring. Compare dense task-quality feedback with critic-error prioritization; use cap-only and schedule-replay controls before adding latent features.

What would change the decision?

Require simultaneous task retention and independently confirmed recovery improvement. If dense feedback matches a latent observer, retain the simpler signal. Confirm stronger-stress effects in a newly frozen study.

Published research story ↗ · Preprint ↗ · Fresh-event replication ↗ · Continuation history ↗

10

Infrastructure / Corrected simulation rehearsal

G1 deployment / validating the execution path

Does the exported policy receive the intended reference in the actual deployment runner?

Research goal

Preserve joint order, clocks, control activation and reference content across Python, ONNX, TensorRT and DDS.

Method

Bind policy/reference hashes, compare model outputs, inspect complete reference traces and rehearse start–stand–play–stop through the C++ runner and simulated robot.

Findings and current progress

The September 17 reference CSVs had a joint-order mismatch. The September 18 corrected converter regenerated them and all eight C++/TensorRT/DDS rehearsals completed under the declared low-pelvis criterion. Earlier DDS readiness claims were superseded; direct-player path drift remains documented.

Evidence boundary

Loading, balance and complete reference playback do not establish accurate locomotion tracking or hardware readiness. No robot was connected in this validation.

Next research

Validate path and body tracking, transitions, timing and fault behavior under the corrected export contract before a separately authorized hardware protocol.

What would change the decision?

Do not promote on inference parity alone. The policy must execute the intended motion and satisfy task-quality requirements in the runner that will be deployed.

Correction and evidence ↗ · Deployment repository ↗

11

Learning / Mixed preregistered findings

Curriculum-MaxRL / what makes practice useful?

Does a score derived from an RL estimator identify useful examples, or only opportunities for nonzero updates?

Research goal

Allocate examples and rollout counts effectively while accounting for how task aggregation changes the estimator’s signal.

Method

Derive expected coefficient activity from pass rates and rollout count, estimate it with posterior sampling, and compare score shapes and realized count-law corrections under matched budgets.

Findings and current progress

The N-aware shape helped in a fixed Acrobot pool but lost to p(1−p) in the coarser MAZE-SCORE task unit (−0.0032; reported CI [−0.0054, −0.0011]). A separate matched count-law correction improved coverage-AUC by +0.00666; this does not establish superiority over p(1−p).

Evidence boundary

The algebra predicts coefficient activity, not the sign of learning benefit. Cross-study patterns were interpreted after results. Pooling heterogeneous tasks can invalidate an i.i.d.-at-the-mean activity estimate.

Next research

Test activity calibration and learning utility separately. Match the scored task unit to the estimator’s rollout group; compare count-law activity, simple learnability and strong replay controls at equal cost.

What would change the decision?

Retain activity as a diagnostic if it predicts update availability but not useful learning. A more exact score earns deployment only through independently measured learning gains.

Repository ↗ · Full reviewed record ↗

12

Learning / Published · IROS 2025

GACL / adapting the tasks

How can generated training tasks remain useful and relevant to a partially observed target domain?

Research goal

Choose practice suited to the learner while retaining the structure of real target tasks.

Method

Encode structured tasks, give a teacher task/performance history, and alternate reference with generated samples.

Findings and current progress

The paper reports improved wheeled navigation and quadruped locomotion over its compared curricula. The existing research story shows the paper’s scoped simulation tables.

Evidence boundary

This published task-selection method is separate from reward adaptation and domain-randomization scheduling. The three have not been jointly validated as one system here.

Next research

A proposed extension is to compare performance history with execution-quality feedback while preserving the same grounding distribution and training budget.

What would change the decision?

Keep the simpler history if added execution features do not improve held-out task learning and retention.

Paper ↗ · Illustrated method ↗

13

Learning / Published · IROS 2025

Reward Training Wheels / adapting the guidance

Which auxiliary rewards help a learner now, while the primary objective stays fixed?

Research goal

Reduce manual reward-weight tuning and adapt shaping to the learner’s changing needs.

Method

A teacher observes reward/weight histories and adjusts auxiliary weights; the student continues optimizing the primary task with that guidance.

Findings and current progress

The paper reports improvements in constrained navigation and off-road mobility, including a small physical off-road comparison. Existing paper-linked charts retain the sample sizes and comparator.

Evidence boundary

Weights need not decrease monotonically. Reward adaptation is not a physical safety guarantee; small hardware demonstrations have a bounded scope.

Next research

A proposed extension compares adaptive weights with tuned fixed weights and schedule replay under matched budgets, then tests whether retained task quality explains any gain.

What would change the decision?

Prefer a fixed or replayed schedule if it matches adaptation; distinguish adaptive feedback from merely choosing better average weights.

Paper ↗ · Illustrated method ↗

14

Learning / Infrastructure; prospective method

TRIAGE / choosing what to adapt

Should the next intervention change tasks, rewards or domain randomization?

Research goal

Route limited training effort to the current bottleneck rather than independently increasing every curriculum axis.

Method

Task/reward/DR experts propose bounded interventions. A guarded router compares deployment-grounded value, switching cost and confidence against a null continuation.

Findings and current progress

The reviewed repository implements the router, state transitions, paired screening/confirmation machinery and an Isaac Lab Franka Lift adapter. Short PPO probes and checkpoint forking remain the next stage in that record.

Evidence boundary

Infrastructure and shadow predictions are not a demonstrated training advantage. Source-reported status is a snapshot, not a live run monitor.

Next research

Construct an intervention atlas to test whether short probes predict longer-block value. Compare fixed-axis, random, hold/null and guarded routing policies with probe costs charged.

What would change the decision?

Do not activate the learned router until probe validity and downstream benefit are demonstrated. A transparent fixed-axis strategy is the primary simplifying rival.

Repository ↗ · Pinned reviewed README ↗

15

Physics / Development research; confirmation pending in source

SNMR / shared neural motion retargeting

Can a shared motion representation preserve content when decoded for a different robot?

Research goal

Map human motion to embodiment-conditioned robot references that remain useful to a physical tracker.

Method

Use a skeleton-aware graph autoencoder and robot graph decoder with joint limits and differentiable forward kinematics. Compare GMR and learned references through separately trained Holosoma tracking policies.

Findings and current progress

The reviewed README reports an initial two-walk comparison passing the first seed’s content gates, with confirmation seeds still queued in that source. Its earlier specialist comparison closed after a quality-gate failure.

Evidence boundary

Kinematic reconstruction, explicit-reference tracking and latent-command control are separate questions. Confirmation status has not been verified from a live remote worker; no hardware conclusion follows.

Next research

Close the frozen confirmation with adequately trained GMR controls and loader-order checks. Evaluate content retention and physical tracking together before expanding embodiments or adding a shared policy.

What would change the decision?

If learned references cannot match a qualified IK baseline, diagnose representation/contact information before claiming shared-control value.

Repository ↗ · Pinned reviewed README ↗

16

Learning / Offline evidence; policy result unverified

Traversal Critic / learned visual rewards

Can a video model judge traversal well enough to improve a navigation policy?

Research goal

Use simulator-derived labels to train a visual critic, then supply training-time shaping to a small policy above a frozen whole-body motor.

Method

Separate label integrity, critic parsing/ranking, out-of-distribution ordering and matched baseline/oracle/critic policy tests.

Findings and current progress

The reviewed record reports that the selected critic parses validation outputs, but its impaired-gait out-of-distribution example scores above clean walking. The policy matrix is described as running in that source; no completed policy benefit is verified here.

Evidence boundary

Correlation with labels does not establish useful reward, causal visual understanding or policy improvement. Historical critic versions cannot be silently pooled.

Next research

Close the frozen matrix and independent replay before editing claims. Test a corpus-disjoint natural-fall/collision readout challenge to expose visually plausible but physically poor behavior.

What would change the decision?

If the critic cannot rank physical failures correctly or improve policy outcomes against an oracle/baseline at equal cost, retain it as an offline instrument rather than a navigation reward.

Project ↗ · Pinned reviewed README ↗

17

Infrastructure / Reusable research infrastructure

Motion2Scene Training Kit + Newton Terrain Lab

Can another researcher reproduce the intended data, simulator and policy interfaces?

Research goal

Make experiments inspectable and portable without confusing a packaged artifact with a validated learning result.

Method

Publish versioned datasets/manifests, extraction and doctor commands, selective resets, map generation, reference viewers and explicit simulator adapters.

Findings and current progress

The training kit bundles research code/data and workflow checks. The terrain package demonstrates native small-batch evaluation and a PPO smoke update. Newer local terrain training has since completed with negative qualification results, documented separately above.

Evidence boundary

Packaged September snapshots can lag active work. KimoNav’s repaired 84/reserved 20/excluded 16 split differs from the training kit’s reported 89-clip pool; these memberships must not be merged. A replay video is not live physics.

Next research

Version interfaces and data contracts, keep source/clip identities through every transformation, and publish both qualification failures and reproducible small examples.

What would change the decision?

Treat installability, numerical conformance, physical validity and learning benefit as separate release checks; a pass at one boundary does not imply the next.

Training kit ↗ · Kit snapshot ↗ · Terrain lab ↗ · Lab snapshot ↗

18

Infrastructure / Measured engineering study

Cosmos3-Edge / compute-efficient models

Can a multimodal model fit an older GPU without losing useful behavior?

Research goal

Reduce memory and runtime requirements while preserving task-relevant model quality.

Method

Use weight-only INT8, phase-separated memory probes and same-configuration comparisons; diagnose compiler kernel substitutions separately from quantization effects.

Findings and current progress

The local report measures 480p/189-frame image-to-video peak memory at 9.86 GiB with INT8 versus 12.48 GiB with BF16 on its A10G setup. It also documents a version-specific compiled INT8 performance trap.

Evidence boundary

These are measured configurations, not universal memory limits or a current installation recommendation. Small visual/LPIPS comparisons do not establish unchanged downstream robot decision quality.

Next research

Freeze software/hardware settings and broaden task-level quality checks, including traversal-relevant judgments if the model is used by the critic. Measure load transients and complete pipeline cost.

What would change the decision?

Adopt compression only if the intended task meets a predeclared quality tolerance and actual end-to-end resource budget.

Project ↗ · Reviewed measurements ↗

04 / HOW THE KIMONAV QUESTION EVOLVED

Follow the evidence
through the changes in direction.

  1. Objective and representation audits. Typed objectives and program/path representations made task information measurable. Retrieval and offline prediction did not establish control of generated or executed motion.
  2. Body conditioning and temporal supervision. Target-assisted corrections showed potential, but learned conditioning repeatedly failed its promotion gates. Lower imitation loss was not enough. Historical synthesis with supersession notes.
  3. Execution and terminal repair. The research moved to persistent feedback, transitions, braking and complete-task accounting. Earlier successes are bounded by their controller, reference bank and test conditions. Preserved September 10 notebook.
  4. Sparse navigation students. The September reassessment records zero full completions for the public student arms in its latest matched panel despite lower offline errors. It redirects the main prerequisite toward task/motor feasibility. Evidence and metric definitions.
  5. Startup, ducking and motor controls. Standing entry, posture realization and the motor itself changed outcomes. The stronger simple fixed baseline had to be established before connecting public selection. Progress · Motor comparison · Simple reference.
  6. Public connection and useful admission. The familiar public interface works; new fixed clearance probes show it rejects requests that can succeed. The next question is transferable acceptance under execution variation, compared with calibrated geometry. Public connection · Current plan.

These stages use different datasets, tasks, motors and metrics. The sequence explains decisions; it is not a monotone performance curve. Historical protocols and unsuccessful outcomes remain part of the record.

05 / RESEARCH INSIGHTS & NEXT STEPS

Make the next experiment
answer a decision.

The following is a proposed ordering, not a promise of positive results or an authorization to start new training. Each study should freeze its own budget, data access and acceptance criteria.

01 / USE THE CAPABILITY WE HAVE

Calibrate useful admission.

Hypothesis: execution-aware bounds can recover useful requests that a reference-only screen rejects.

Decisive comparison: scalar calibrated margin versus measured envelope, same motor/task/bank and fresh evaluation conditions.

Measure: completed requests over all requests, admitted failures, rejected-but-solvable requests, unknown choices and collection cost.

Change course: keep the scalar rule if it matches the envelope. Qualify another source rather than endlessly sweeping one beam.

02 / TURN DATA INTO A WORKING STUDENT

Prove the complete learning path.

Hypothesis: causal complete-episode supervision can train useful goal/map behavior above the existing motor.

Decisive comparison: simple continuous/Linear29 baselines, then hindsight versus scene-first acquisition with the same cost and independently defined splits.

Measure: unaided complete-task gains and regressions, support coverage, failure type and inference cost.

Change course: repair data/adapter support before adding a new tokenizer, LLM or larger architecture if the first student cannot execute.

03 / TEACH WITHOUT LOSING THE TASK

Measure recovery and retention.

Hypothesis: task-quality feedback targets useful practice more reliably than high error alone.

Decisive comparison: dense feedback versus critic priority with fresh events, matched support and schedule/cap controls; terrain start-distribution ablation before deeper-soil scaling.

Measure: event-aligned recovery with dwell, censored failures, fixed-horizon quality and nominal skill retention.

Change course: prefer simple replay, dense signals or a fixed schedule when adaptation adds no retained benefit.

04 / BROADEN ONLY AFTER QUALIFICATION

Test the assumptions we remove.

Hypothesis: the learned decision survives new ancestry, combined constraints and partial observations.

Decisive comparison: qualify new motion sources and entry states, then vary scene families; replace exact maps with timestamped partial sensing in a separate study.

Measure: transfer, observation age, unknown-space handling, end-to-end latency and complete task performance.

Change course: retain the known-map scope when perception fails. Hardware and new contact capabilities require their own qualification.

The shared research insight

Our current evidence suggests that the important bottleneck often lies at a boundary: a task becomes a reference, a reference becomes an action, an executed continuation becomes a label, or a label becomes a retained skill. The strongest next contribution would identify which information is missing at one of these boundaries and demonstrate useful improvement over a simple, well-qualified alternative.

This is a synthesis and a testable research direction. It is not a claim that every project shares one failure mechanism or that the combined system is already validated.

06 / EVIDENCE & READING

Inspect the record behind the story.

The source snapshots below preserve the research reports used here. They are plain text: internal relative links and local paths are archival references, not website navigation. Dates inside a source may be older than this review. “Running” and “pending” describe the reviewed record unless a completion report is cited. Numerical findings are attributed to those records, not independently reproduced by this website update.

Source manifest and SHA-256 hashes · Structured project overview · Published collaborations and bibliography · KimoNav primary-literature assessment

Open the 26 reviewed source snapshots and extracts

Terms used throughout

Causal observation
Information available before the current action, excluding future motion and outcomes.
Qualification
A pass under a declared task, motor, input and evaluation contract; not universal reliability.
Shadow admission
A rule’s saved choice scored against fixed executions; not a new rollout of that rule.
Motion ancestry
The original source lineage shared by related clips or variants; copies do not create independent data.
Retention
Preserving previously demonstrated skills or tracking quality while learning a new behavior.
MPM
Material Point Method: material particles and a computation grid model deformable support; presets still need calibration.