2
$\begingroup$

I am trying to grapple with the following problem.

I have an application that develops empirical distributions. In essence, I end up with a histogram of equally spaced $x$ values, with both a $max$ and a $min$, and the probability for each bucket. What I need to do is 'compress' it for storage (there may be 1,000 to 10,000 buckets) so that the distribution can be recreated with reasonable accuracy.

The obvious choices are:

  1. find a Taylor series approximation

  2. find a Fourier transform approximation

  3. find a moment-generating approximation

Obviously with #1 one only needs convergences between the $max$ and $min$. With #2, one would model the distribution as periodic, with one period occurring between $min(x)$ and $max(x)$. In both cases it is obvious how to recover the distribution based on the very definition of the Taylor series and Fourier transform.

My question is this: if one went the route of #3, what is the 'inversion' formula that recovers a generalized functional representation of a distribution based on the first n integral moments being known?

Improve this question
$\endgroup$
3
  • 2
    $\begingroup$ I'm not sure why you have to compress it; with equally-spaced $x$ values, all you need to store is the overall $\min$, $\delta$, and count of number of values ($n$) - two floats and an int. Then you need to store $n$ integers, maybe even short integers, so your total storage space is about $4n$ bytes - only 40 KB with $n=10000$, give or take a little. This doesn't seem particularly onerous. $\endgroup$ Commented Dec 24, 2017 at 2:12
  • 1
    $\begingroup$ Consider other options. For instance, you could fit a spline to the actual frequencies (not the relative frequencies) using a Poisson GLM and then normalize it. That would likely require just a handful of coefficients and would give an easily computed formula for the log density. $\endgroup$ Commented Dec 24, 2017 at 17:19
  • $\begingroup$ See also stats.stackexchange.com/questions/141652/… $\endgroup$ Commented Jan 8, 2020 at 13:55

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.