Exercise: measuring a segmentation

Duration ~35 min  ·  Session Day 2, Python notebooks

You will use relevant metrics to settle the question the last exercise left open: which segmentation was actually best?

Data: data/bbbc020/ field jw-24h 3_c1. This field is used deliberately: it is one dataset where every cell was outlined by the annotator. Scoring against a partly-annotated field punishes a method for finding cells that are really there.

import numpy as np
import tifffile
import matplotlib.pyplot as plt
import os; os.environ["DISPLAY"] = ":1"

from skimage.measure import label
from skimage.color import label2rgb
from skimage.filters import threshold_otsu, gaussian
from skimage.morphology import remove_small_objects
from scipy.ndimage import binary_fill_holes, distance_transform_edt
from skimage.segmentation import watershed

from course import DATA, show

image = tifffile.imread(DATA / "bbbc020" / "images" / "jw-24h 3_c1.tif")
truth = tifffile.imread(DATA / "bbbc020" / "gt" / "jw-24h 3_c1 [1].tif")
print("annotated cells:", len(np.unique(truth)))

Task 1: build two segmentations to compare

A classical pipeline and Cellpose, both on this field.

Hint

The classical pipeline is the one from 02_image_processing: smooth → threshold → fill holes → remove small objects → watershed seeded from distance > t. For Cellpose use models.Cellpose(model_type="cyto", gpu=False).

from cellpose import models
from skimage.color import rgb2gray

# --- your turn ---
smoothed = ...
mask = ...          # TODO: threshold, fill holes, remove small objects
distance = ...
classical = ...     # TODO: watershed seeded from a distance cut-off

cellpose_labels, _, _, _ = ...   # TODO: run Cellpose

results = {"classical": classical, "Cellpose": cellpose_labels}
for name, r in results.items():
    print(f"{name:10s}: {r.max():3d} objects   (annotated: {len(np.unique(truth))})")

Task 2: score them on pixels

Print IoU and Dice for both.

Hint

Your functions already accept label images, because they convert with > 0.

# --- your turn ---
print(f"{'method':12s} {'IoU':>7s} {'Dice':>7s} {'objects':>9s}")
for name, result in results.items():
    print(...)   # TODO

Task 3: score them on objects

Now the part the pixel scores cannot do. Complete match_objects, which pairs each predicted object with the ground-truth object it overlaps most and accepts the pairing when their IoU clears min_iou.

Hint 1, which ground-truth objects can possibly match?

Only those that overlap the prediction at all: np.unique(truth[predicted]), skipping 0.

Hint 2: one prediction per truth object

Keep a set of already-matched ground-truth ids so two predictions cannot both claim the same cell.

Hint 3: the counts

false_positives = prediction.max() - true_positives and false_negatives = truth.max() - true_positives.

def match_objects(prediction, truth, min_iou=0.5):
    """Pair predicted objects with ground-truth objects. Returns (TP, FP, FN)."""
    matched_truth = set()
    true_positives = 0

    for p in range(1, prediction.max() + 1):
        predicted = prediction == p
        if not predicted.any():
            continue

        # --- your turn ---
        best_score, best_id = 0.0, None
        for t in ...:               # TODO: truth ids overlapping this prediction
            ...                      # TODO: keep the best-scoring one
        
        if ...:                      # TODO: accept if it clears min_iou and is unclaimed
            true_positives += 1
            matched_truth.add(best_id)

    false_positives = prediction.max() - true_positives
    false_negatives = truth.max() - true_positives
    return true_positives, false_positives, false_negatives

def scores(tp, fp, fn):
    precision = tp / (tp + fp) if tp + fp else 0.0
    recall = tp / (tp + fn) if tp + fn else 0.0
    f1 = 2 * precision * recall / (precision + recall) if precision + recall else 0.0
    return precision, recall, f1


# --- your turn ---
for name, result in results.items():
    tp, fp, fn = ...        # TODO
    p, r, f1 = scores(tp, fp, fn)
    print(...)              # TODO: one row per method

Let’s analyze the table.

• Do the pixel scores and the object scores rank the two methods the same way?

• If they disagree, which would you report, and why does the answer depend on what you are trying to measure?

(edit this cell to add your answer)

Task 4: how strict should the matching be?

Recompute F1 at several matching thresholds and plot it.

Hint

match_objects already takes min_iou. Loop over [0.3, 0.5, 0.7, 0.9], collect the F1 values, and use plt.plot.

# --- your turn ---
thresholds = [0.3, 0.5, 0.7, 0.9]

plt.figure(figsize=(7, 4))
for name, result in results.items():
    f1s = ...    # TODO: F1 at each threshold
    plt.plot(...)
plt.xlabel("matching IoU threshold"); plt.ylabel("F1")
plt.ylim(0, 1); plt.legend(); plt.show()

The plot shows that every curve falls as the threshold rises.

• What would it mean if one method’s curve fell much faster than another’s?

(edit this cell to add your answer)