Stitched and Bound
Stitching the oracle and v32 into one end-to-end controller, finding the fuel budget that broke the hand-off, and training a perturbation-robust capture specialist that beats the oracle alone.
01 The Setup
Welcome back. v32 from last week captures Mars 20/20 across four cruise spawn points on window 0, and the oracle solves the full Earth-to-Mars problem at 97.5% across all four launch windows. Both pieces work; what was missing was a way to glue them together. Five attempts to extend v32 by retraining had failed, so the plan for this week was to compose the two solvers behind one predict interface and skip the retraining altogether.
Phase 0 and most of Phase 1 are oracle territory; the last 40 days of cruise plus the MOI brake go to v32. The hand-off boundary is v32's outermost training cruise spawn point, and the waypoint cache that defines that spawn was produced by running this same oracle. Under deterministic clean physics, the oracle's state at $T - 40$ days matches v32's training distribution by construction.
02 The Hand-Off Misaligned
The first version of the stitched controller dispatched on time-to-SOI: oracle while $t_\text{to\_SOI} > 40\,\text{d}$, PPO once it dropped under. Clean-physics runs showed the moment the controller switched, the spacecraft's heliocentric position differed from the cached $T - 40\,\text{d}$ waypoint by roughly 2.3 Mkm and the velocity by about 16 m/s. This meant the hand-off was off by exactly one decision step.
The cache is built at decision-step granularity by scripts/precompute_soi_entry.py. The oracle does not stop precisely at $T - 40\,\text{d}$; it stops at whichever decision-step boundary happens around that moment, which can be up to a full step on either side. So the cached waypoint labelled "T-40d" actually corresponds to $T - 40.32\,\text{d}$, and any fixed-offset rule lands the live spacecraft on the wrong step's worth of cruise.
The fix is to anchor the hand-off on the cached ET itself rather than on a relative time-to-SOI. The npz file records the ephemeris time of every cruise waypoint it stores. The controller now loads cruise_epoch_et[i, idx] alongside the position and velocity, and switches the moment the live ET reaches that exact value. Under deterministic clean physics the result is $\Delta r$ and $\Delta v$ at hand-off both round to zero.
if t_to_soi(env) > 40*86400: dispatch_to_oracle()# Live hand-off was off by ~1 decision step → Δr ≈ 2.3 Mkm at switch
# New (cached-ET rule):
if env.et_now() < cruise_epoch_et: dispatch_to_oracle()# Clean hand-off lands exactly on the training spawn state (Δr ≈ 0)
Fallback Gate
Under perturbations the live state will not match the cached waypoint perfectly even with the corrected dispatch rule. A pos / vel quality gate guards against off-distribution hand-offs: if $|\Delta r| > 150{,}000\,\text{km}$ or $|\Delta v| > 50\,\text{m/s}$, the controller stays on the oracle for the rest of the episode. The gate is a safety net for outliers and is not exercised in v34f's main results, but it gives the system a graceful fallback when the upstream drift is too large for the trained PPO to recover from.
03 The Fuel Budget
With the hand-off fix in place, the stitched v32 controller still failed at end-to-end. Eval runs all reached Mars's SOI with the right approach geometry, but $\sim$80% of episodes hit fuel depletion before the brake closed and the eccentricity hovered at 0.995, barely hyperbolic. The MOI burn was being initiated at full commitment but cut short. Something about the live state at hand-off was inconsistent with v32's training distribution after all.
Turns out the culprit was the fuel budget. v32 was trained from a SOI-edge spawn with the spacecraft at full tank: $m_0 = 1200\,\text{kg}$, all 1100 kg of propellant available for the MOI brake. The policy learned a burn profile that uses roughly 811 kg of that propellant over the 50–80 steps inside Mars's SOI. In the hierarchical run, by contrast, the oracle has already spent fuel on the TMI burn ($\sim$888 kg) and a small TCM allowance ($\sim$61 kg) during cruise. By the time the spacecraft reaches $T - 40\,\text{d}$, total mass is about 251 kg and the available propellant for capture is 171 kg. v32 then over-spreads its trained burn profile and exhausts the tank before periapsis.
A useful aside on Basilisk physics surfaced while debugging this. BSK's spacecraft hub keeps its inertial mass constant, so the dynamics integrator computes $\Delta v = F \cdot t / m_0$ regardless of how much propellant has actually been depleted. The propellant bookkeeping (Tsiolkovsky) runs in parallel but doesn't feed back into the integrator. The practical consequence is that a commanded burn magnitude produces the same per-step $\Delta v$ whether the tracked tank is full or nearly empty; what changes is only the fuel counter and the latched "fuel exhausted" termination flag. The fix therefore lives entirely in the training-time fuel counter that the policy sees, while the simulator dynamics stay unchanged.
04 v33: Realistic-Mass Specialist
A new cfg.spawn_mass_kg_override field plus a matching --spawn-mass-kg-override CLI flag let the SOI-edge spawn write any chosen total mass into the env at reset, while leaving the BSK hub mass alone (so per-step $\Delta v$ is unaffected). v33 is a fine-tune of v32 with the override set to 252 kg, the launch window pinned to 0, and the cruise offset fixed at $T - 40\,\text{d}$ to match the hand-off boundary:
python main_train.py --env mars_nasa \ --resume successful_models/v32_window0_multioffset_0to40d_5x5_5x5.zip \ --cruise-offset-seconds 3456000 --spawn-mass-kg-override 252 \ --epoch-weights "1,0,0,0" \ --lr 1e-4 --ent-coef 0.001 --reset-log-std -1.5 \ --clamp-log-std --clamp-log-std-min -2.5 --clamp-log-std-max -0.3
First capture event landed at the 200k-step evaluation: $+1521$ reward, ep_len 186, deterministic. Training plateaued from there but the best checkpoint was preserved. The recipe is the same v24 / v32 recipe (low entropy, clamped log_std) with the fuel budget as the only new variable.
Obs Spoof at the Hand-Off
v33 is trained from env._step = 0 but at the live hand-off env._step is around 197 (Phase 0 plus 196 cruise steps). The observation vector contains a TIME_TO_GO component that v33 saw as roughly 1.0 during training and would see as $\sim$0.55 in deployment if fed raw. To keep the live distribution close to training, the controller latches the step index at the first PPO dispatch and recomputes the observation's TIME_TO_GO dynamically:
# For v33 (trained at mass_kg=252): MASS_FRAC is already correct, leave it.
# For v32 (trained at full tank): also spoof MASS_FRAC = 1.0.
A --ppo-realistic-mass / --no-ppo-realistic-mass toggle picks the right spoofing behaviour for whichever PPO is being run, since v32 and v33 have different training-mass assumptions and the controller has to match each one.
05 End-to-End
The stitched controller with the v33 PPO, the cached-ET hand-off rule, and the obs spoof produces a complete Earth-to-Mars run from LEO to a bound Mars orbit. On clean physics, window 0, fixed parking orbit, 5 random seeds: 5/5 captures, mean reward $+1565$, mean capture eccentricity 0.940, mean periapsis altitude 84,614 km. The original project goal of an autonomous Earth-to-Mars transfer is met.
Five-episode evaluation of the hierarchical controller on window 0, clean physics. Heliocentric plane (left) with Earth's and Mars's full-year orbits as dashed circles, the spacecraft traces (coloured by episode), and burn events marked with cyan dots (oracle) or red stars (PPO). Distance to Mars (top right) decays from 170 Mkm at TMI to inside the SOI by step 200. Burn $\Delta v$ per step (bottom right, symlog) shows the TMI as a $\sim$3.5 km/s spike at step 0, small oracle TCMs through cruise, and a concentrated PPO MOI burn around step 400.
The trajectory plot makes the role of each solver visible. The blue Phase-0 spike is the oracle's Lambert TMI, sized at one decision step. The cyan dots scattered through cruise are oracle TCMs, each a few m/s of trajectory correction. The red star cluster near step 400 is v33's MOI burn, concentrated into roughly six high-magnitude steps right at periapsis. The cyan-to-red transition is the hand-off boundary, $T - 40\,\text{d}$ before SOI entry.
// What's Doing What //
Both solvers contribute to the final $+1565$ reward, but not symmetrically:
# Oracle Phase 0 TMI: ~3.5 km/s, ~888 kg fuel
# Oracle Phase 1 cruise TCMs: ~40 m/s, ~61 kg fuel
# PPO Phase 1 endgame (40d): ~3 m/s, ~5 kg fuel
# PPO Phase 2 MOI brake: ~430 m/s, ~165 kg fuel
$$\text{capture: } e = 0.940,\; r_p^\text{alt} = 84{,}614\,\text{km}$$
06 Federici-Style Perturbations
The clean-physics 100% rate is the easy half of robustness. The harder question is what survives when the simulator stops being a clean ephemeris integrator. Roberto Federici's Deep Learning-Based Spacecraft Optimal Guidance dissertation defines a five-channel noise model for exactly this scenario with navigation sensor noise, thruster magnitude noise, thruster pointing noise, unmodelled-force impulses, and stochastic missed-thrust events. The Hohmann env already had these wired in while the Mars NASA env had been ignoring its cfg.perturb entirely.
All five channels are now plumbed through MarsNasaEnv:
The Mars defaults are roughly 10× milder than the Hohmann defaults. A first attempt at "v34_robust" used the Hohmann-scale perturbations on Mars and the result was a 100% failure rate: a 5% thruster magnitude error on the 3.5 km/s TMI burn translates to about 175 m/s of $\Delta v$ uncertainty, which by the time the spacecraft reaches Mars compounds to roughly 1.2 Mkm of SOI miss. The oracle then cannot recover the trajectory inside its Phase 1 $\Delta v$ budget. Gentler defaults at the Mars scale let the oracle do its job over the 240-day cruise.
Sweeping the oracle alone across $\{0\times, 1\times, 5\times, 10\times\}$ the Mars defaults gives capture rates of $100\%$, $90\%$, $80\%$, $70\%$ across 20 seeds each. At the realistic $1\times$ scale, the oracle drops to 90%. At Federici's full strength ($10\times$) it drops to 70%. The Mars defaults are the regime our launch windows actually face.
07 v34f: Perturbation-Robust Capture
With perturbations on, v33 alone in the hierarchy collapses from 100% to 25% capture. The drift accumulated through 240 days of cruise puts the spacecraft well off the cached spawn waypoint at hand-off, and v33's training distribution is too narrow to recover. The obvious move is to widen that training distribution, which means training a new capture specialist under the same noise the deployment will see.
Three earlier attempts to do this failed. v34c fine-tuned v33's anchor with 50 km/axis spawn-state noise and collapsed to a coast policy. v34d turned perturbations on without spawn noise and collapsed to the same coast policy. v34e used a narrower 5 km/axis spawn noise on v32 instead of v33, and the basin barely widened (10% capture, deployment-shifted by an average of 230k km from training spawn). This shows that fine-tuning from v32 or v33 anchors with anything resembling realistic spawn-distribution width destroys the policy faster than the new gradient can rebuild it.
v34f changes the initialisation. Instead of resuming from a previous PPO checkpoint, it warm-starts from a behaviour-cloning baseline trained on a Phase-2-heavy oracle dataset (bc_policy_v5_p2heavy.pt). The BC loss landscape is flatter than v32's or v33's sharp local optimum, so PPO has room to find a broader basin without first having to escape an existing one:
python main_train.py --env mars_nasa \ --bc-init bc_checkpoints/bc_policy_v5_p2heavy.pt \ --cruise-offset-random-choices-s "0,604800,2592000,3456000" \ --cruise-offset-random-weights "1,3,1,3" \ --spawn-pos-noise-km 30000 --spawn-vel-noise-ms 20 \ --spawn-mass-kg-override 252 --epoch-weights "1,0,0,0" \ --lr 1e-4 --ent-coef 0.002 --log-std-init -2.0 \ --clamp-log-std --clamp-log-std-min -2.5 --clamp-log-std-max -0.3 \ --timesteps 1_000_000 --n-envs 8
The 30,000 km / 20 m/s spawn-state noise covers the actual deployment drift distribution (mean $\sim$230k km cumulative miss, with the long tail captured by the noise's 3$\sigma$ wing). 5 km was too narrow, 50 km from a sharper anchor was too wide, 30 km from a BC baseline lands in the sweet spot.
Save Best Model Early
Evaluation reward peaks around step 200,000 (mean 985, std 524 across episodes, with the large variance reflecting captured-vs-not-captured mixing) and then degrades back toward a coast policy by 1M steps (eval reward 528). The standard SB3 best-checkpoint callback caught the peak and the late degradation didn't overwrite it.
Result
Over 50 perturbed episodes on window 0:
v34f under perturbations not only matches the clean-physics rate of the oracle alone, it produces a markedly tighter capture orbit: mean eccentricity 0.59 versus the oracle's 0.86, mean periapsis altitude 46,000 km versus the oracle's 102,000 km. The PPO uses the extra information in its observation (Mars-relative state, B-plane error) to commit a sharper MOI brake than the oracle's open-loop Lambert solution.
v32 and v33 are sharp local optima of the Phase-2 reward surface, fitted to narrow spawn distributions. A gradient step in any direction that helps generalisation also moves the policy away from one of those optima, and the reward signal under exploration noise is dominated by the immediate loss of capture probability. PPO retreats. BC v5 sits in a flatter region of the same loss landscape with smaller second derivatives, so PPO can shift the policy along the generalisation direction without paying as much in the local capture rate.
The model is window-0 only. The same training recipe with --epoch-weights set differently is the natural starting point for the remaining three launch windows, but the proof-of-concept for the project's headline goal is complete with a learned policy delivers an Earth-to-Mars transfer under realistic deployment noise.