Sensitivity Analysis Demo¶

This notebook demonstrates TerraFlow's sensitivity analysis workflow:

• Sobol' indices (S1, ST): quantify how much each ModelParams weight independently and in total drives score variance.
• Morris elementary effects (μ*, σ): fast screening to rank parameters by influence without requiring a full pipeline run.

Sensitivity analysis helps answer: if I perturb w_v vs w_t, which one changes my suitability scores most?

In [1]:

import json
import tempfile
import textwrap
from pathlib import Path

from terraflow.sensitivity import run_sensitivity

# ── Self-contained synthetic config (no real data files needed) ──────────
tmp = Path(tempfile.mkdtemp())
config_path = tmp / "config.yml"
config_path.write_text(textwrap.dedent(f"""\
    raster_path: /nonexistent/raster.tif
    climate_csv: /nonexistent/climate.csv
    output_dir: {tmp}/outputs
    roi:
      xmin: -100.1
      ymin: 39.9
      xmax: -99.9
      ymax: 40.1
    model_params:
      v_min: 0.0
      v_max: 25.0
      t_min: 0.0
      t_max: 40.0
      r_min: 0.0
      r_max: 300.0
      w_v: 0.4
      w_t: 0.3
      w_r: 0.3
    sensitivity:
      w_v:
        low: 0.2
        high: 0.5
      w_t:
        low: 0.2
        high: 0.5
      w_r:
        low: 0.1
        high: 0.4
      n_samples: 64
      method: both
"""))
print(f"Temp config: {config_path}")

import json import tempfile import textwrap from pathlib import Path from terraflow.sensitivity import run_sensitivity # ── Self-contained synthetic config (no real data files needed) ────────── tmp = Path(tempfile.mkdtemp()) config_path = tmp / "config.yml" config_path.write_text(textwrap.dedent(f"""\ raster_path: /nonexistent/raster.tif climate_csv: /nonexistent/climate.csv output_dir: {tmp}/outputs roi: xmin: -100.1 ymin: 39.9 xmax: -99.9 ymax: 40.1 model_params: v_min: 0.0 v_max: 25.0 t_min: 0.0 t_max: 40.0 r_min: 0.0 r_max: 300.0 w_v: 0.4 w_t: 0.3 w_r: 0.3 sensitivity: w_v: low: 0.2 high: 0.5 w_t: low: 0.2 high: 0.5 w_r: low: 0.1 high: 0.4 n_samples: 64 method: both """)) print(f"Temp config: {config_path}")

Temp config: /var/folders/h_/_bjpnb894kgf18frynwf6bww0000gn/T/tmp1w7f44qa/config.yml

Config setup¶

The config must include a sensitivity: block. The examples/demo_config.yml already has one — point to it here, or pass any config that includes a sensitivity: section.

In [2]:

report_path = run_sensitivity(config_path)
print(f"Sensitivity report written to: {report_path}")

report_path = run_sensitivity(config_path) print(f"Sensitivity report written to: {report_path}")

INFO:terraflow:Running Sobol' analysis with N=64 (total evaluations: 512)

                            Sobol' Sensitivity Indices                            
┏━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Rank ┃ Parameter ┃ S1 (first-order) ┃ S1 95% CI ┃ ST (total-order) ┃ ST 95% CI ┃
┡━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ 1    │ w_r       │ 0.3392           │ ±0.2278   │ 0.3332           │ ±0.1198   │
│ 2    │ w_v       │ 0.3273           │ ±0.1675   │ 0.3292           │ ±0.1181   │
│ 3    │ w_t       │ 0.3358           │ ±0.2196   │ 0.3291           │ ±0.1215   │
└──────┴───────────┴──────────────────┴───────────┴──────────────────┴───────────┘

INFO:terraflow:Running Morris analysis with 6 trajectories (total evaluations: 24)

                                  Morris Elementary Effects                                   
┏━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓
┃ Rank ┃ Parameter ┃ mu* (mean abs effect) ┃ mu* 95% CI ┃ mu (mean effect) ┃ sigma (std dev) ┃
┡━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩
│ 1    │ w_r       │ 0.1500                │ ±0.0000    │ 0.1500           │ 0.0000          │
│ 2    │ w_t       │ 0.1500                │ ±0.0000    │ 0.1500           │ 0.0000          │
│ 3    │ w_v       │ 0.1500                │ ±0.0000    │ 0.1500           │ 0.0000          │
└──────┴───────────┴───────────────────────┴────────────┴──────────────────┴─────────────────┘

INFO:terraflow:Sensitivity report written to /var/folders/h_/_bjpnb894kgf18frynwf6bww0000gn/T/tmp1w7f44qa/outputs/sensitivity_report.json

Sensitivity report written to: /var/folders/h_/_bjpnb894kgf18frynwf6bww0000gn/T/tmp1w7f44qa/outputs/sensitivity_report.json

Inspect the results¶

In [3]:

with open(report_path) as f:
    report = json.load(f)

print("Keys in sensitivity_report.json:", list(report.keys()))

with open(report_path) as f: report = json.load(f) print("Keys in sensitivity_report.json:", list(report.keys()))

Keys in sensitivity_report.json: ['schema_version', 'method', 'n_samples', 'parameters', 'bounds', 'timestamp', 'sobol', 'morris']

In [4]:

sobol = report.get("sobol")
if sobol:
    params = list(sobol["S1"].keys())
    print("Sobol' indices (S1 = first-order, ST = total-order):")
    print(f"{'Parameter':<12} {'S1':>8} {'ST':>8}")
    print("-" * 30)
    for param in params:
        s1 = sobol["S1"].get(param, float("nan"))
        st = sobol["ST"].get(param, float("nan"))
        print(f"{param:<12} {s1:>8.4f} {st:>8.4f}")
else:
    print("No Sobol' results (method was not 'sobol' or 'both')")

sobol = report.get("sobol") if sobol: params = list(sobol["S1"].keys()) print("Sobol' indices (S1 = first-order, ST = total-order):") print(f"{'Parameter':<12} {'S1':>8} {'ST':>8}") print("-" * 30) for param in params: s1 = sobol["S1"].get(param, float("nan")) st = sobol["ST"].get(param, float("nan")) print(f"{param:<12} {s1:>8.4f} {st:>8.4f}") else: print("No Sobol' results (method was not 'sobol' or 'both')")

Sobol' indices (S1 = first-order, ST = total-order):
Parameter          S1       ST
------------------------------
w_v            0.3273   0.3292
w_t            0.3358   0.3291
w_r            0.3392   0.3332

In [5]:

morris = report.get("morris")
if morris:
    params = list(morris["mu_star"].keys())
    print("Morris elementary effects (mu_star = mean |effect|, sigma = std):")
    print(f"{'Parameter':<12} {'mu_star':>10} {'sigma':>8}")
    print("-" * 32)
    for param in params:
        mu = morris["mu_star"].get(param, float("nan"))
        sig = morris["sigma"].get(param, float("nan"))
        print(f"{param:<12} {mu:>10.4f} {sig:>8.4f}")
else:
    print("No Morris results (method was not 'morris' or 'both')")

morris = report.get("morris") if morris: params = list(morris["mu_star"].keys()) print("Morris elementary effects (mu_star = mean |effect|, sigma = std):") print(f"{'Parameter':<12} {'mu_star':>10} {'sigma':>8}") print("-" * 32) for param in params: mu = morris["mu_star"].get(param, float("nan")) sig = morris["sigma"].get(param, float("nan")) print(f"{param:<12} {mu:>10.4f} {sig:>8.4f}") else: print("No Morris results (method was not 'morris' or 'both')")

Morris elementary effects (mu_star = mean |effect|, sigma = std):
Parameter       mu_star    sigma
--------------------------------
w_v              0.1500   0.0000
w_t              0.1500   0.0000
w_r              0.1500   0.0000

Interpreting results¶

Sobol' S1 (first-order index): The fraction of output variance explained by a single parameter alone. High S1 → parameter is individually important.

Sobol' ST (total-order index): Variance fraction including all interactions. If ST >> S1 for a parameter, it interacts strongly with others.

Morris μ* (mean absolute elementary effect): Overall sensitivity rank. Higher μ* → more influential parameter.

Morris σ: Standard deviation of elementary effects. High σ relative to μ* suggests non-linear behaviour or interactions.

Practical guidance: Parameters with low ST can often be fixed at their default values without loss of model fidelity.

References¶

• Saltelli, A. et al. (2010). Variance based sensitivity analysis of model output. Design and estimator for the total sensitivity index. Comput. Phys. Commun., 181(2), 259–270.
• Morris, M.D. (1991). Factorial sampling plans for preliminary computational experiments. Technometrics, 33(2), 161–174.
• Herman, J. & Usher, W. (2017). SALib: An open-source Python library for sensitivity analysis. J. Open Source Softw., 2(9), 97.