1
$\begingroup$

Are there generalizations of common hypothesis tests (e.g. t-test, mann-whitney) that can take into account different confidences in the sample measurements?

For example, if I have two sets of measurements:

A = [1, 5, 2, 6, 8, 9, 2]

B = [2, 7, 1, 6, 9, 7, 6]

and a given set of sample-specific measurement errors (different from the sampling error):

e_A = [0.2, 0.3, 0.1, 0.4, 0.1, 0.2, 0.3]

e_B = [0.1, 0.1, 0.1, 0.3, 0.2, 0.2, 0.1]

(these can be a standard deviation or the width of a confidence interval around the measurement, e.g. the true measure of $A[i]$ is inside the interval $[A[i]-e_A[i], A[i]+e_A[i]]$ with some probability $p$)

Is there a way to account for the heterogeneity in confidence/uncertainty concerning each measurement when running such a hypothesis test?

Similarly, are there hypothesis tests that can work with intervals (i.e. instead of a measurement $a$, we are testing sets of intervals $[a_1, a_2]$ that contain the true measurement with some probability)?

I have done some review of the literature and other posts but haven't quite found anything that would address this type of setup. Any help would be greatly appreciated!

Improve this question
$\endgroup$
1
  • $\begingroup$ Yes, you could consider an error-in-variables model. The t-test is a special case of general linear models. You can also have error-in-variables for Mann-Whitney U. $\endgroup$ Commented Jan 19, 2023 at 16:09

1 Answer 1

Reset to default
2
$\begingroup$

Error-In-Measurements

You could consider an error-in-variables model. The t-test is a special case of general linear models. You can also have error-in-variables for Mann-Whitney U.

Here is a toy example of simulating the error-in-variables assuming a normal distribution with a diagonal covariance.

import matplotlib.pyplot as plt
import numpy as np
from scipy import stats

# Parameters
A = [1, 5, 2, 6, 8, 9, 2]
B = [2, 7, 1, 6, 9, 7, 6]

e_A = [0.2, 0.3, 0.1, 0.4, 0.1, 0.2, 0.3]
e_B = [0.1, 0.1, 0.1, 0.3, 0.2, 0.2, 0.1]


# Resample statistic with error
k = 10000
resample_A = np.random.multivariate_normal(A, np.diag(e_A), size=k)
resample_B = np.random.multivariate_normal(B, np.diag(e_B), size=k)

t_results = stats.ttest_ind(resample_A, resample_B, axis=1)
U_results = stats.mannwhitneyu(resample_A, resample_B, axis=1)

# Plot results
fig, axes = plt.subplots(2,2)

axes[0][0].set_title('t-Score')
axes[0][0].hist(t_results[0])

axes[0][1].set_title('p-value (t-test)')
axes[0][1].hist(t_results[1])

axes[1][0].set_title('U-Score')
axes[1][0].hist(U_results[0])

axes[1][1].set_title('p-value (U-test)')
axes[1][1].hist(U_results[1])

plt.tight_layout()
plt.show()

enter image description here

Interval Arithmetic

As for interval arithmetic, you'll have to decide how turn those measurement error terms into intervals. Once you have intervals, you just go through the same computations as you would with scalars but being mindful of when you are performing non-monotonic functions (mainly the squaring in computing the standard deviation).

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thank you for the helpful response. From my understanding, error-in-variable models don't usually take in explicit errors for each observation but rather some parametric assumptions of the error distribution in the independent variables (i.e. normality, covariance with dependent variable, etc). Is this correct and can they still be adapted to the case above where we have known errors for each observation? $\endgroup$ Commented Jan 19, 2023 at 22:19
  • $\begingroup$ Also, in the example that you have shown with repeated sampling from the normal distributions to obtain a distribution of p-values, is there an intuitive interpretation of what these p-value distributions would mean? For example, if I disregard the uncertainties, I can get a p-value $p_1$ that tells me the probability of as or more extreme events under the null. For the distribution of p-values if we were to summarize it with the mean or median p-value (say $p_2$), is there a similar interpretation? Thanks! $\endgroup$ Commented Jan 19, 2023 at 22:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.