New FWCI and MRPE Formulations Under Big Data
Let us conceptualize a big dataset arriving from a source of interest, for example, a bank, stock market, credit card company, or Internet provider, or with data on the status of health, education, nutrition, wealth distribution, housing, income, expenditure, loan default of a population. To be specific, we concentrate on income distribution of a random variable X with its d.f. F(-) defined on the support (0,oo). The dataset (or the population) is supposedly very large and hence we decide to observe independent vectors of observations (Хф..., Х,д.) at-a-time at the zth instance where к is "large" in itself, but held fixed, and then let i run through 1,..., r.
We may add that Mahalanobis (1940) introduced the novel idea of interpenetrating subsampling carried out in the context of large-scale survey designs. What we are proposing is not interpenetrating subsampling in the sense of Mahalanobis (1940). This conforms reasonably well with today's principle of what may be called "divide and conquer" or "split and conquer" in big data science (Chen and Xie, 2014).
Since we will be incorporating sequential inference strategies, it may not be entirely out of place to include a bit of interesting history which connects this area to Mahalanobis's (1940) ground-breaking research. The founding father of sequential analysis, Abraham Wald, wrote in the introduction of his classic text: "...The occasional practice of designing a large-scale experiment in successive stages may be regarded as a forerunner of sequential analysis... A very interesting example of this type is the series of sample censuses of area of jute in Bengal carried out under the direction of P. C. Mahalanobis..(Wald, 1947, Introduction, p. 2).
The Foundation and Structure
We focus on large k, certainly >2, and comparatively small r, thinking about the choice of “k" which may be reasonable as we come face to face with big data scenarios. Our total sample size is going to be n = rk where r will denote the number of 1'-tuples that will be recorded. Chattopadhyay and De (2014, 2016) and De and Chattopadhyay (2017) had fixed к = 1 and r = n.
But, that may not be very realistic or logistically convenient under big data scenarios. Malinovsky and Zacks (2018) and Mukhopadhyay and Wang (2020) incorporated group sequential sampling or sequential sampling in groups for parametric estimation. Their motivational pursuits were distinct from ours.
In the spirit of (11.2)—(11.3), we form an estimator of Gf as follows:
where Х,д is the sample mean and Д,д is the sample GMD at the /th instance based on (Х,д,..., Х,д) alone, respectively determined as follows:
i = l,...,r. Clearly, we have 0 < G/д- < 1 with probability (w.p.) 1.
Now, the random variables Gj/s from (11.8) are i.i.d. sample statistics and we decide to work under a customary assumption:
We realize that G/д is a ratio of two U-statistics, and hence G/д is approximately normal (Hoeffding, 1948,1961; Xu, 2007) since к is supposedly large if Ер[Щд] < оо. Then, for large к, we may express:
where an expression of the asymptotic variance of - Gf), namely £2
is given by:
l
with
One may refer to Chattopadhyay and De (2016) and De and Chattopadhyay (2017) for critical details. By the way, the statistic V or V,, used in (11.4) or (11.7) was consistent for £2 or respectively for large n.
We ought to emphasize an important point: In the implementation of our proposed methodologies, we do not require an exact analytical expression in a closed form for either functional or £2.
Chattopadhyay and De (2016) and De and Chattopadhyay (2017) required an exact expression of!;} and they found it as in (11.12).
We reiterate that the estimator Gyt across i instances are i.i.d. and each is supposed to be remarkably accurate to estimate Gf since their common individual asymptotic variance will become /c_1£2 ss 0 with к large, i = ,r.
Then, our final estimator Gr^ of Gf based on combining the /с-tuples recorded across r independent instances is proposed in the form of a sample mean of very accurate i.i.d. sample statistics, as follows:
Next, we can claim:
with к held fixed, if Ef[X2] < oc. This follows from a standard central limit theorem (CLT).
At this point, an important question is: How do we optimally determine r, the number of /с-tuples necessary to observe? A response ought to depend upon a specific inference problem at hand taking into account preassigned accuracy measures that one sets prior to collection of data. So, in what follows, we take up that task as we revisit the FWCI and MRPE problems. Neither of these inference problems can be solved when r is kept fixed (Dantzig, 1940).
