4
$\begingroup$

There are statistical test suites (below) that are commonly used to determine whether a sequence appears to be (pseudo) random. Some of these test suites have a few tests (ent/ent3000), whilst others have over a 100 (NIST STS). All of them assume the same null hypothesis that the samples are from an independent and uniform distribution.

And these tests are pretty serious. They're used in national security cryptography, your banks' security and in casinos transacting billions of dollars.

Non of them apply a Bonferroni correction to the critical $\alpha$ value. Why not?

diehard

dieharder

pracrand

ent

ent3000

NIST STS

Test U01

Some others...

Improve this question
$\endgroup$
1
  • $\begingroup$ It's unnecessary. Part of the test definition is the sample size. Using the required minimum size limits the decision risk. The problem with relying on Bonferroni here is that it doesn't really apply: these tests all use the same stream of random values and so are completely interdependent in a way that's difficult to characterize. One could use Bonferroni as a conservative check and that's probably not a bad idea if a very large number of tests are run on the same stream. $\endgroup$ Commented Dec 3, 2023 at 15:48

2 Answers 2

Reset to default
8
$\begingroup$

Adjustments for multiplicity or comparisons are not often helpful, in my opinion, and in the context of your question they are a very bad way to protect against false positive errors. As the datasets are computer-generated you can simply obtain a second set of data and re-test them. (Or divide a large dataset into two before testing.) The probability of a false positive occurring for the same comparison in both runs of the testing is very small.

Improve this answer
$\endgroup$
4
$\begingroup$

I am no expert on RNGs, but I can think of two reasons off the top of my head:

First:

Remember that if you decrease the chance of a type 1 error, you increase the chance of a type 2 error. For testing RNGs, a type 2 error seems much worse.

Null hypothesis: This RNG is OK under the criterion of the particular test.

Type 1 error: We reject the null when it's true. That is, we reject a good RNG. Type 2 error: We fail to reject the null when it's false. That is, we use a bad RNG.

Type 1 error means the programmers have more work to do. Type 2 error means our banking records are not secure.

Second, and more general, the whole question of whether we need to do any correction for multiple comparisons is one in which (to quote Jacob Cohen) "reasonable people can differ". I would say that, if the tests being applied to the RNGs are really different from each other, that would argue against correction. Also, if the plan of testing is specified ahead of time, that would also argue against correction. I think the corrections are most needed when we are in a field where we know very little and are doing a whole lot of testing of hypotheses. One such is the area that Fisher did most of his work in: Testing fertilizers and other products in agriculture (especially back when Fisher was working). That was more a of a "let's try this, it might work!" atmosphere. And there, the importance of the two types of error was quite different.

Improve this answer
$\endgroup$
3
  • 1
    $\begingroup$ This list of generic comments overlooks the circumstances surrounding RNG testing that differentiate it from most other applications of NHST: namely, samples of arbitrarily large size are available (and typically enormous samples are used). $\endgroup$ Commented Dec 3, 2023 at 15:49
  • $\begingroup$ @whuber Yes, this answer shows dieharder sucking on nearly 250 GB of samples. $\endgroup$ Commented Dec 3, 2023 at 23:32
  • 1
    $\begingroup$ I also want to add that all pseudo-RNGs (which is what you mean by "RNG" here) are "bad" in the sense that they will fail at least one test of randomness when given a sufficiently large dataset. One isn't trying to establish whether a pseudo-RNG is not uniformly iid, but that it's good enough for an intended purpose. $\endgroup$ Commented Dec 4, 2023 at 14:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.