Model Validation Demo¶

This notebook demonstrates TerraFlow's model validation workflow, including:

• Spatial block cross-validation with buffer zones (Roberts et al. 2017)
• Cohen's kappa against a reference dataset
• Moran's I on score residuals
• Validation block in report.json

In [ ]:

from pathlib import Path
from terraflow.validation import run_validation

config_path = Path("../examples/demo_config.yml")
report_path = run_validation(config_path)
print(f"Updated report: {report_path}")

from pathlib import Path from terraflow.validation import run_validation config_path = Path("../examples/demo_config.yml") report_path = run_validation(config_path) print(f"Updated report: {report_path}")

In [ ]:

import json

with open(report_path) as f:
    report = json.load(f)

val = report.get("validation", {})
for key, value in val.items():
    print(f"{key}: {value}")

import json with open(report_path) as f: report = json.load(f) val = report.get("validation", {}) for key, value in val.items(): print(f"{key}: {value}")

Interpreting Results¶

Mean fold accuracy: Measures how consistently the suitability labels hold across spatial blocks. Because TerraFlow's model has no free parameters (scores are deterministic given config), fold accuracy reflects spatial label distribution consistency rather than model generalization.

Cohen's kappa: Measures agreement between TerraFlow's classification and the reference dataset, corrected for chance. Values: <0 = less than chance, 0 = chance, 0.2-0.4 = fair, 0.4-0.6 = moderate, 0.6-0.8 = substantial, 0.8-1.0 = near-perfect.

Moran's I: Measures spatial autocorrelation in score residuals. Positive values indicate spatial clustering of similar residuals; values near 0 indicate spatial randomness.

Kriging LOOCV RMSE: Per-variable leave-one-out cross-validation error for kriging interpolation, computed during the pipeline run (not by validation).

In [ ]:

loocv = report.get("kriging_loocv")
if loocv:
    print("Kriging LOOCV RMSE:")
    for var, rmse in loocv.items():
        print(f"  {var}: {rmse:.6f}")
else:
    print("No kriging LOOCV data (kriging was not used in this run)")

loocv = report.get("kriging_loocv") if loocv: print("Kriging LOOCV RMSE:") for var, rmse in loocv.items(): print(f" {var}: {rmse:.6f}") else: print("No kriging LOOCV data (kriging was not used in this run)")

References¶

• Roberts, D.R. et al. (2017). Cross-validation strategies for data with temporal, spatial, hierarchical, or phylogenetic structure. Ecography, 40(8), 913-929.
• Cliff, A.D. & Ord, J.K. (1981). Spatial Processes: Models and Applications. Pion.