Add qc metrics #66

davcrom · 2025-03-27T12:19:27Z

some new metrics added, some update, now with ruff format fixes

- revert n_unique_samples to number instead of fraction

- now pass t to detect_spikes rather than y

- takes sum of all points that are above the expected maximum value

- if KDE fails, always use global median

- now based on fixing repeated sampling of the same channel

grg2rsr · 2025-05-09T11:54:37Z

src/iblphotometry/metrics.py

+    dts = np.diff(t)
+    dt = np.median(dts)
+    n_violations = np.sum(np.abs(dts - dt) > atol)
+    ## TODO: make ibllib wrappers to convert metrics to QC vals


this is indeed a huge todo. The structure that I see for this, is a separate definition of the ranges for the metric and the corresponding label. I would split this away from the function definition as we might change the interpretation for the metric, while the definition of it is standalone, to be decided. I also want to study the QC system for the ephys more first to see what works well there and what doesn't

kick dt violations

grg2rsr · 2025-05-09T11:55:01Z

src/iblphotometry/metrics.py

+#     return bool(even_check & odd_check)
+
+
+def n_early_samples(A: pd.DataFrame | pd.Series, dt_tol: float = 0.001) -> int:


please add docstring and definition what is an early sample

move to processing

grg2rsr · 2025-05-09T11:55:17Z

src/iblphotometry/metrics.py

+    return find_early_samples(A, dt_tol=dt_tol).sum()
+
+
+def n_repeated_samples(


please add docstring

grg2rsr · 2025-05-09T11:59:00Z

src/iblphotometry/metrics.py

    return P[1] - P[0]


+def deviance(


add docstring
unused argument w_len, also looks like you are using w_len for samples. In sliding operations, I followed a syntax as such

y, t = F.values, F.index.values fs = 1 / np.median(np.diff(t)) if fs is None else fs w_size = int(w_len * fs)

grg2rsr · 2025-05-09T12:00:28Z

src/iblphotometry/metrics.py

+    return np.median(np.abs(a - np.median(a)) / np.median(a))
+
+
+def sliding_deviance(


If possible I would argue for not having "sliding" variants of metrics but using a framework defined elsewhere such as sliding operations to use for evaluating metrics in a sliding manner.

grg2rsr · 2025-05-09T12:01:04Z

src/iblphotometry/metrics.py

+    return np.mean(x) + np.std(x) * np.sqrt(2 * np.log(len(x)))
+
+
+def n_expmax_violations(A: pd.Series | np.ndarray) -> int:


please docstring

merge to expmax_violations

grg2rsr · 2025-05-09T12:01:16Z

src/iblphotometry/metrics.py

+    return sum(np.abs(a) > exp_max)
+
+
+def expmax_violation(A: pd.Series | np.ndarray) -> float:


please docstring

grg2rsr · 2025-05-09T12:02:02Z

src/iblphotometry/metrics.py

    return reg.popt[1]


+def bleaching_amp(A: pd.Series | np.ndarray) -> float:


I am not sure if this is a meaningful metric as we don't do a proper quantitative measurement of photon count or similar

grg2rsr · 2025-05-09T12:02:38Z

src/iblphotometry/metrics.py



-def low_freq_power_ratio(A: pd.Series, f_cutoff: float = 3.18) -> float:
+def response_variability_ratio(


grg2rsr · 2025-05-09T12:02:45Z

src/iblphotometry/metrics.py

+    return (responses).mean(axis=0).var() / (responses).var(axis=0).mean()
+
+
+def response_magnitude(A: pd.Series, events: np.ndarray, window: tuple = (0, 1)):


leftovers from pynapple, fixme

grg2rsr · 2025-05-09T12:04:42Z

src/iblphotometry/processing.py

+    return dt - A.index.diff() > dt_tol
+
+
+def _fill_missing_channel_names(A: np.ndarray) -> np.ndarray:


ultimately I would argue these functions should better live in preprocessing
we need to define the differences where preprocessing ends and processing starts, but I think fixing signal issues that come from FP3002 bugs feel like they are in the domain of preprocessing

grg2rsr · 2025-05-09T12:06:13Z

src/iblphotometry/processing.py


-    return pd.Series(y, index=t)
+
+def find_repeated_samples(


docstring me pls

code review this

grg2rsr · 2025-05-09T12:06:27Z

src/iblphotometry/processing.py

+    return repeated_sample_mask
+
+
+def fix_repeated_sampling(


docstring me pls

grg2rsr · 2025-05-09T12:08:11Z

src/iblphotometry/qc.py

    return pd.Series(m, index=t[inds + int(w_size / 2)])


+def _eval_metric_sliding(


For this we should probably sit down together and you need to walk me through the proposed changes in this function.

grg2rsr

left some comments, we should sit together and discuss some of your proposed changes and see how we structure the qc looking forward.

davcrom added 9 commits January 17, 2025 17:50

Add PSTH based metrics

c8f03bd

Allow flexible input dtype

0eae5b4

Update n_unique_samples docstring

8ebb16d

Normalize n_unique_samples by total samples

a9750f7

Spike detection for both positive and negative jumps

8d4e5cc

Add check for dead signals

34ca19e

Allow AR(n) models in ar_score

b2782eb

Revert to original spike detection

80a3166

ruff format

9f7c092

davcrom requested a review from grg2rsr March 27, 2025 12:25

davcrom and others added 20 commits April 4, 2025 17:21

Add dt_violations metric

168445c

Add interleaved acquisition metric

aff9a50

Add sliding_deviance metric

058e887

Add f_unique_samples

e09eed3

- revert n_unique_samples to number instead of fraction

Fix bug in remove spikes

e0e332c

- now pass t to detect_spikes rather than y

Duplicate n_spikes for dt and dy

787b07b

Suggest simple nan fill method with local median

51fa9c0

Add sliding_expmax_violation

5f519b9

- takes sum of all points that are above the expected maximum value

Add photobleaching_amp (WIP)

c4ddc8a

ruff format

3c1fd7b

merge not present conflict?

c07651f

Remove always True conditional

9109b38

- if KDE fails, always use global median

Fix bug in n_spikes_dt

287b11f

Make expmax_violation safe for no outliers case

db07a5a

Fix merge conflict

f1ab68a

Housekeeping

b596663

Add separate functions for dy and dt spike detection

8d41f34

Replace global median with local median in remove_spikes

6cd8811

Update detection of "spikes" caused by early sampling

fa58dfa

- now based on fixing repeated sampling of the same channel

Add deviance metric

d5222ac

davcrom added 8 commits April 18, 2025 12:26

Update expmax violation metrics

23d9790

Bugfix in model fitting

ae5c8cb

WIP: bleaching_amp metric

63b1f41

Update low_freq_power_ratio to accept dt as kwarg

1f11178

Handle edge cases with too few data points in ar_score

81d6c0b

WIP: add two methods for sliding_robust_zscore

a024876

Major update to qc_series sliding metric handling

1d7d9dc

Ruff format

50995f9

grg2rsr changed the base branch from main to develop May 6, 2025 15:32

grg2rsr marked this pull request as draft May 6, 2025 15:32

grg2rsr reviewed May 9, 2025

View reviewed changes

src/iblphotometry/metrics.py

return find_early_samples(A, dt_tol=dt_tol).sum()

def n_repeated_samples(

Copy link

Contributor

grg2rsr May 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please add docstring

grg2rsr reviewed May 9, 2025

View reviewed changes

src/iblphotometry/processing.py

return repeated_sample_mask

def fix_repeated_sampling(

Copy link

Contributor

grg2rsr May 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

docstring me pls

grg2rsr reviewed May 9, 2025

View reviewed changes

grg2rsr requested changes May 9, 2025

View reviewed changes

		# return bool(even_check & odd_check)


		def n_early_samples(A: pd.DataFrame \| pd.Series, dt_tol: float = 0.001) -> int:

		return find_early_samples(A, dt_tol=dt_tol).sum()


		def n_repeated_samples(

		return np.median(np.abs(a - np.median(a)) / np.median(a))


		def sliding_deviance(

		return np.mean(x) + np.std(x) * np.sqrt(2 * np.log(len(x)))


		def n_expmax_violations(A: pd.Series \| np.ndarray) -> int:

		return sum(np.abs(a) > exp_max)


		def expmax_violation(A: pd.Series \| np.ndarray) -> float:

		return reg.popt[1]


		def bleaching_amp(A: pd.Series \| np.ndarray) -> float:



		def low_freq_power_ratio(A: pd.Series, f_cutoff: float = 3.18) -> float:
		def response_variability_ratio(

		return (responses).mean(axis=0).var() / (responses).var(axis=0).mean()


		def response_magnitude(A: pd.Series, events: np.ndarray, window: tuple = (0, 1)):

		return dt - A.index.diff() > dt_tol


		def _fill_missing_channel_names(A: np.ndarray) -> np.ndarray:

		return pd.Series(m, index=t[inds + int(w_size / 2)])


		def _eval_metric_sliding(

Add qc metrics #66

Are you sure you want to change the base?

Add qc metrics #66

Uh oh!

Conversation

davcrom commented Mar 27, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

grg2rsr left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants