A reverse curriculum, a Phase 2 reward overhaul, the first 5/5 deterministic Mars captures, multi-offset training that scaled to 20/20, and the wall that stopped extension cold.
Last week's nasa_v2 plateaued at evaluation reward 75 after 10M steps with zero captures, traced to a 4D log_std runaway plus a curriculum that never advanced past Stage 1. The fix list was a per-environment ent_coef = 0.002, a CLI flag for initial log_std, and a --reset-log-std escape hatch for resumes.
nasa_v3 trained for another 5M steps from nasa_v2's best checkpoint with ent_coef=0.002 and log_std reset to $-2.0$. Eval reward climbed cleanly to about 200, then stalled without captures. The mean policy steered the spacecraft to a closest approach of around 250 Mkm, well inside Mars's sphere of influence (577 Mkm), but it never produced a sustained anti-velocity burn at Mars periapsis. Several follow-up runs (nasa_v4 through nasa_v22) cycled through tweaks to alignment bonuses, distance shaping, fuel weights, and BC dataset composition. All of them plateaued in the same place: somewhere between $+250$ and $+450$ evaluation reward, with zero captures across roughly 60M cumulative training steps.
That much wasted compute against the same plateau is a sign that the gradient that PPO is climbing does not lead to capture. There is a partial-brake harvesting path worth $+400$, and there is a full-capture path worth over $+1500$. The first is reachable from random exploration; the second is not. This week's work tries to close that gap two different ways.
A reverse curriculum trains the last phase of a problem first. For Mars, the last phase is the brake at periapsis: a multi-hundred-m/s anti-velocity burn that converts the hyperbolic approach into a bound orbit. If the agent could be dropped at Mars's sphere of influence with a credible incoming hyperbola already in hand, capture would be the only thing it ever had to learn.
The oracle already constructs that approach state every time it runs, because Phase 0 (TMI) and Phase 1 (cruise + TCMs) come before Phase 2 in the same episode. So scripts/precompute_soi_entry.py runs the oracle once per launch window, snapshots the heliocentric $(r, v)$ at the moment the spacecraft crosses Mars's SOI, and dumps the result to bc_checkpoints/soi_entry_states.npz. The script also samples 18 cruise waypoints earlier in the trajectory (from 1 hour back through 120 days), so the env can spawn at any chosen time-before-SOI.
InterplanetaryConfigV2.start_at_soi_edge flips on the new spawn path. When set, reset() looks up the cached waypoint matching the chosen launch window and cruise offset, writes those numbers into BSK as the initial state, and sets self._phase = PHASE_MOI (or PHASE_CRUISE if the offset is non-zero). The episode budget is shared with the full-trajectory case (440 steps), but with no Phase 0 or early Phase 1 to spend it on, the agent gets $\sim$200 Phase-2 steps inside the SOI to commit to a brake.
An earlier attempt placed the spacecraft on an analytic two-body Mars hyperbola with a tunable B-plane offset. Sun-perturbed dynamics during real cruise put the spacecraft on a different geometry from any clean two-body flyby, so the analytic spawn was off-distribution from the moment Phase 2 began. Caching the oracle's actual SOI state instead of synthesising one fixed it.
Reverse curriculum changes the spawn distribution but says nothing about whether the reward surface inside Phase 2 has a path to capture. Inspecting the v17–v22 plateau showed it does not. The agent's locally optimal behaviour at SOI edge is a brief partial brake that nudges the trajectory closer to Mars, harvests the distance-shaping bonus, and lets the periapsis pass without committing the rest of the dv. Several reward terms had to be added or rebalanced to make a full capture strictly better than the harvest path.
The Phase 0 alignment bonus pays the agent for firing in the prograde direction relative to Earth's velocity. The Phase 2 analogue rewards firing anti-velocity relative to Mars:
This is the gesture an MOI burn requires. With the quadratic profile, mid-alignment burns earn very little, so an agent fishing for partial credit gets nothing; a committed periapsis brake earns hundreds.
The boolean capture criterion (bound, latched for 3 steps) is a step function on a continuous quantity. Adding a per-step shaping term on the underlying energy turns it into a smooth gradient:
A brake reduces $\varepsilon$, so $\Delta\varepsilon < 0$ and the agent gets positive reward proportional to the energy reduction. An earlier saturating-tanh form gave near-zero gradient at the hyperbolic baseline because $\varepsilon / E_\text{scale} \approx 2.5$ saturates the tanh derivative to about $0.07$. The linear form gives proportional signal at any $\varepsilon$. Telescoping is preserved (sum equals $-w_\varepsilon \cdot (\varepsilon_\text{final} - \varepsilon_\text{init}) / E_\text{scale}$), so an oscillating policy that pumps and brakes the same energy gets nothing.
The boolean criterion only fires once per episode at the moment of capture; until that moment, the Gaussian-kernel proximity score $q_\text{moi} \in [0, 1]$ (last week's terminal grading) carries no per-step weight. A tiny per-step contribution turns it into a continuous shaping signal:
The gating on $e < 1$ is what makes this safe to add. The agent earns no continuous reward by getting "almost captured"; it only earns once it has actually bent the hyperbola into an ellipse. Across $\sim$50 to 100 in-SOI steps, this sums to $+250$ to $+500$ of reward for the post-capture coast.
Last week the B-plane shaping term was off entirely because of two interacting bugs. A saturating tanh potential plus a lazy first-entry initialisation lets it come back online:
The total contribution is bounded by $2 w_b = 400$, so a divergent trajectory cannot accumulate the $-50000$ penalty that wrecked nasa_v1. The lazy-init removes the spurious one-shot delta the first time the spacecraft enters the evaluation region.
A committed MOI brake risks both crashing into Mars (too much dv) and slipping back to heliocentric (too little). The old penalty magnitudes dwarfed the energy reward, so any committed brake was net-negative in expectation under exploration noise. The success bonus and quality scale go up by 2.5× in aggregate; the failure penalties drop to roughly the same magnitude as a single bad partial brake. Earth re-impact stays at $-500$ because that's a Phase 0 anomaly and has nothing to do with the brake-attempt risk profile.
With the new reward surface and SOI-edge spawning, training proceeded in two stages. v23 ran with a relatively wide exploration setting (ent_coef = 0.005, log_std_init = -1.0, no clamp) for 4M steps from BC initialisation. Eval reward bounced between 600 and 900 with deterministic captures still at zero, but rollout reward (which includes stochastic action noise) showed individual episodes occasionally crossing the 1500 mark. Some action samples were threading the capture; the deterministic mean at the centre of the action distribution was not.
v24 forces the issue. It resumes from v23's best checkpoint and tightens the action distribution: ent_coef = 0.0005, --reset-log-std -1.5, and a hard log_std clamp of $[-2.5, -0.5]$. The clamp is the new piece. Without it, gradient updates that nudge log_std outside the box just keep going; with it, every update gets the freedom to move the mean inside a sigma window of $0.08$ to $0.61$. The constrained policy can no longer hide capture in stochastic outliers because the outliers don't exist anymore.
The first eval after the resume showed a single capture (eval reward 1763 at step 400k). Subsequent evals slipped back to the 700–900 range, but the best_model checkpoint preserved the 400k peak. Re-running deterministic evaluation on that checkpoint gave 5/5 captures across 5 random seeds, all on launch window 0 (2024 OCT 15), with mean reward $+1762$ and mean closest-approach to Mars under 1 Mkm (versus a 577 Mkm SOI). The model is saved at successful_models/v24_window0_capture.zip.
Decomposing a capture episode by reward source clarifies which terms actually drove the policy:
The energy term and the dense quality term together contribute about as much as the terminal success bonus does, which is the right ratio. If terminal reward dominated, exploration would never reach it; if shaping dominated, the policy would farm shaping without capturing.
v24 captures only when spawned exactly at the SOI edge. Spawning even 7 days earlier and giving the agent the same Phase 2 budget produces zero captures. Several rounds of sequential per-offset training (v25 through v30) crystallised separate policies for different cruise offsets, but each new offset training run forgot the previous one (v28 was 5/5 at T-30d but 0/5 at T=0; v30 was 5/5 at T-40d but 0/5 at T-30d). A different approach was needed.
cruise_offset_random_choices_s and cruise_offset_random_weights let one PPO run train against multiple spawn points simultaneously. Every reset() picks an offset by weighted random sampling, so a single network sees all four spawn distributions and is required to handle each. v31 used uniform weights ($\{1, 1, 1, 1\}$) and plateaued at 10/20 captures across $\{T = 0,\, T - 7d,\, T - 30d,\, T - 40d\}$: window 0 and T-30d only. The two failing offsets (T-7d and T-40d) are at adjacent positions in offset space, suggesting the policy got stuck in a local mode that couldn't satisfy them.
v32 keeps the same spawn distribution but oversamples the failing offsets via $\{1, 3, 1, 3\}$ weights. The extra exposure to T-7d and T-40d perturbs the policy enough to escape the v31 attractor:
20 out of 20 deterministic captures from a single network at 1.35M steps, on launch window 0, across all four spawn points. The model lives at successful_models/v32_window0_multioffset_0to40d_5x5_5x5.zip and represents the best Mars policy this project has produced.
The plan from there was to extend v32: more cruise offsets, more launch windows, eventually a single policy that handles the full Earth-to-Mars run. Five attempts followed, and all five failed in different ways.
v33 added T-50d and T-60d to the spawn pool with $\{1, 1, 1, 1, 2, 2\}$ weights so the new offsets dominated. v34 was a gentler version with only T-45d added at weight 1. Both runs destabilised v32's anchor: eval reward bounced between 700 and $-142$ with no recovery, and the saved best_model stayed pinned at v32's checkpoint without ever progressing past it. The agent forgets the inner offsets faster than it learns the outer ones.
v35 and v35b loaded v32 and turned on launch-window randomisation with $\{1, 2, 2, 2\}$ weights (oversampling the unseen windows 1, 2, 3). The eval reward locked at 125.66, which is exactly the truncate-no-capture-no-crash terminal value. Window 0's capture behaviour was destroyed in the first million steps, while no capture behaviour for windows 1-3 ever appeared. The policy collapsed to a coast.
v36 isolated window 1 (2026-11-11) with weights $\{0, 1, 0, 0\}$. The first eval at 100k steps produced a single capture (reward $+2331$), suggesting the geometry was reachable. Subsequent evals dropped to negative reward by 1.5M steps. v32's window-0 weights are too crystallised to generalise; the gradient that would push them toward window 1 also pushes them away from window 0, and there is no path that gets close to both.
v37 sidestepped v32 entirely. A new BC checkpoint (bc_policy_v5_p2heavy.pt) was trained on a Phase-2-heavy oracle dataset and used to initialise PPO with all four launch windows enabled. Eval reward started at $-907$ and degraded to $-2589$ over 1.15M steps. The BC v5 baseline alone is poor (the dataset is too narrow), and PPO from a weak baseline degrades further rather than improving.
PPO retraining from the v32 anchor breaks v32 faster than it learns anything new. PPO from BC v5 has a baseline reward too low to climb from. Either the policy crystallises and refuses to budge, or it never crystallises in the first place. None of the four extension paths pushed evaluation reward above v32's baseline at any point.
The full Earth-to-Mars problem has been solved twice already this project. The oracle does Phase 0 + 1 + 2 at 97.5% across all four launch windows. v32 does Phase 2 plus the last 40 days of cruise at 100% on window 0. What is missing is one PPO policy that does both. The plan for week 10 sidesteps that requirement: compose the two solvers behind a single predict interface, dispatching by phase.
if env._phase == PHASE_PRE_TMI: return oracle()if env._phase == PHASE_CRUISE and t_to_soi > 40d: return oracle()if env._phase == PHASE_CRUISE: return v32.predict(obs)return v32.predict(obs)
The hand-off boundary is exactly $T - 40$ days, the outermost cruise spawn point v32 was trained against. bc_checkpoints/soi_entry_states.npz was produced by running this same oracle, so the oracle's natural trajectory state at $T - 40d$ matches v32's training distribution by construction. There is no out-of-distribution stretch where the controller has to bridge between two unrelated regimes.
The PBRS state lives entirely on the env (_ref_dev_prev, _bplane_err_prev, _mars_energy_prev) and updates every step regardless of which solver supplied the action. Telescoping is preserved across the hand-off without any controller-side bookkeeping. The interface is the two-line SB3 contract every other validate script uses (predict(obs, deterministic=True) → (action, None)), so the controller drops into main_validate.py and main_montecarlo.py unchanged.
Both pieces are individually 100% on clean physics, so anything below $\sim$19/20 captures on clean window 0 will be a wiring bug rather than a design flaw. The gating diagnostics: print Δr and Δv at the hand-off step against the cached cruise_r_helio[0, idx_40d], expect sub-km / sub-cm/s under clean physics. Mean Phase 0 dv against the oracle's reported TMI dv (3.5–4.3 km/s for window 0). Mean $q_\text{moi}$ against v32's training distribution. Optional perturbed run: 20 seeds, expected 85–95% capture rate, anything below treated as a known limitation rather than a regression.
Week 10's deliverable is the hierarchical controller running end-to-end on window 0, clean physics, with a full Earth-to-Mars trajectory plot and a Monte Carlo distribution of capture quality. Two failure modes worth watching for: drift in v32's training distribution due to small ephemeris differences between the precomputed cache and the live SPICE state at evaluation time, and any unexpected sensitivity of the PBRS terms to the hand-off timing.
Beyond that, three open paths from this week's failures are worth keeping on the list. A bigger network ($[512, 512]$ instead of $[256, 256]$) might fit a multi-window policy where $[256, 256]$ over-specialises. An explicit window-id observation component would let a single network condition on which window it's running, replacing implicit feature learning with a direct switch. Per-window v32 specialists plus an ensemble dispatcher would extend the "compose, don't retrain" idea from across-phase to across-window. The hierarchical controller is the fastest route to a working full transfer; whichever of those three the next iteration explores depends on what the validation reveals.
The interesting question for next week is whether the stitched controller is enough to call it solved, and whether anything qualitatively new shows up at the 40-day hand-off boundary that the individual training of either piece would never have surfaced.