Finding the invaders, and knowing when not to trust the map
This maps riparian woody vegetation and its invasive share from Landsat / Sentinel-2 time-series and puts a Random Forest head-to-head with Ai2's OlmoEarth foundation model — a real, leave-one-reach-out field test of where the FM earns its keep and where it ties. It also shows the tests that made two tempting results fall apart. And the grounded agent that answers your questions at the end runs on Ai2's OLMo: OlmoEarth sees, OLMo speaks.
Built end-to-end on the Allen Institute for AI's open models: OlmoEarth (Earth-observation foundation model) delineates the vegetation, and OLMo (open LLM) grounds every answer in the documents. An independent test and an application of Ai2's models — not a benchmark they ran themselves.
Extent is solved. The contribution is the honest invasive layer.
Basin-scale riparian extent was already mapped for this watershed (CO-RIP, Woodward 2018, κ 0.80). Reproducing it is the control, not the result. What doesn't exist yet is a wall-to-wall, native-vs-invasive product that is honest about its own uncertainty — which species, how much, and over what stretch of the record the answer is even trustworthy. That is the target, and this page is the evidence for how far it currently reaches.
A bounding box becomes a phenology cube, then a decision
The signal that separates tamarisk from native cottonwood is phenological — they leaf out and senesce on different schedules, with a distinct short-wave-infrared water signature. So a single image is the wrong input. Each reach becomes a 12-month median-mosaic cube, identically composited, and both models see the full spectro-temporal vector — not a greenness index.
Why not just NDVI?
Because for this problem, greenness is nearly blind. Trained to separate invasive from native on the NDVI index alone, AUC ≈ 0.50 — a coin flip. The separability lives in the SWIR / red-edge bands across the season: tamarisk holds green later into the fall than cottonwood.
23% of the mapped corridor is the wrong plant
Green is the predicted riparian-woody corridor (7.6 km²); red is the invasive tamarisk / Russian-olive share within it (1.7 km²). Toggle either layer. This is the real prediction, on satellite imagery.
On rivers they tie. One arroyo decides it.
The real test isn't in-sample accuracy — it's leave-one-reach-out transfer: train on some reaches, predict a reach the model has never seen. A per-pixel Random Forest, pooled across four morphologically diverse reaches, transfers well to unseen river corridors. But it has exactly one hole — the lone arroyo, the only example of its kind. That gap is precisely OlmoEarth's predicted opening: spatial context on under-represented morphology. This is the honest field test of whether the FM earns it.
All three on the ground, over the labeled reach: the foundation model (green) tracks the corridor; NMRipMap truth (outline) is reality; and the Random Forest (orange), transferred to this arroyo, fires only in the wetter eastern end — the part whose spectra resemble the river reaches it trained on — and misses the western dry wash that FM and truth both capture. That's genuine under-prediction (~0.1% of pixels; verified against a fully-covered cube, not a data gap), and it's the 0.557-vs-0.889 gap made spatial: to the eye, the FM is simply more accurate. Both model layers are full-extent leave-one-reach-out predictions.
The beetle that didn't break the classifier
We predicted a today-trained tamarisk classifier would invert before the tamarisk beetle arrived (~2004–07), when the greenness signal flips. It didn't. And a pre-registered negative control — Russian olive, which the beetle doesn't touch — moved seven times more than the tamarisk "signal," proving the cross-era noise is too large to resolve a beetle effect at all.
| taxon vs native (AUROC) | 2020 | 2015 | 2000 pre-beetle | Δ |
|---|---|---|---|---|
| tamarisk (signal) | 0.849 | 0.813 | 0.862 | +0.013 |
| Russian olive (control) | 0.891 | 0.767 | 0.553 | −0.338 |
Declaring the control in advance is what let it veto the result instead of us over-reading a 0.013 wiggle. The honest read: a beetle-robust discriminator, on a sample too small and too cross-sensor-confounded to say more.
The "5× growth" that was an artifact
The first pass back through the Landsat archive looked like a clean ~5× rise in invasive extent since 1990. Every robustness check dissolved it. The pre-2000 numbers swing with the compositing recipe — no normalization fixes it — because 1990 is pure Landsat-5 TM, spectrally too distant from the training year to reconcile.
See it directly. Drag the year: the map shows the model's actual invasive prediction for each epoch, over the same corridor (green reference). Watch the flag in the corner — from ~2000 on the estimate is stable; slide to 1990 and it turns amber, because that pure Landsat-5 map moves with the compositing recipe. This isn't a growth animation — it's a demonstration of where the record stops being trustworthy.
The method is the deliverable
What separates this from a demo is that the failure modes were hunted, not hidden. Four practices do the work:
AI-assisted research that catches its own errors
This was built fast, with heavy AI assistance — which is exactly where research quietly goes wrong. Hallucinated code is the easy failure; a compiler catches it. The dangerous ones are semantic: they compile, pass the tests, and read beautifully.
AI-assisted research fails by producing work that is fast, fluent, plausible, and wrong — and the wrongness is invisible precisely because the output looks finished. Exhortation doesn't fix this. Gates do.
| Failure mode | Caught by |
|---|---|
| Hallucinated API call | the compiler |
| Wrong logic | a unit test |
| A retracted result still published as fact | nothing |
| A model scored against 45%-wrong labels | nothing |
| A novelty claim already falsified by a 2018 paper | nothing |
So honesty here is mechanized, not exhorted. Retired numbers, withdrawn claims and orphaned docs are caught by drift gates that run identically on a laptop and in CI. The finding, stated plainly in the method write-up: every rule that was merely documented eventually drifted; every rule that was mechanized held.
How it's built
The same discipline runs through the stack — a reproducible EO pipeline, a typed spatial store, two model tracks, a services API, and the grounded agent, each behind its own gate.
The agent below is part of the same discipline: no source, no claim. Grounding is enforced backend-side, so every sentence it returns is pinned to a retrieved document you can open.
Ask the agent — it answers from the documents, and cites them
The findings above, the methods, and the underlying watershed literature are indexed in a hybrid-retrieval RAG system (dense + BM25). Ask anything: the answer is generated by OLMo, Ai2's open language model, grounded in the retrieved sources with clickable citations, and a mention of a reach flies the map to it. No source, no claim — so the same Ai2 stack that maps the vegetation also explains it.