Relating Random Variables and their Statistics to Inner Products

Let us consider a typical real-valued random variable $Y$ defined on a probability space $(\Omega,\mathcal{F},\mathbb{P})$.  Let $X$ be another such random variable, and assume $X,Y\in L^{2}(\mathcal{F}$).  We emphasize the $\sigma$-algebra $\mathcal{F}$ because this will be the structural element of our probability space subject to frequent change; in particular, the measure and sample space $\Omega$ will remain constant (we ignore the issues that arise with this assumption when additional random variables are added to our universe, and implicitly assume $\Omega$ and $\mathbb{P}$ have been properly extended in a way that is consistent with all other previous random variables in play).

The space $L^{2}(\mathcal{F})$ is a Hilbert space with norm
$$(X,Y)=\mathbb{E}[XY]=\int_{\Omega}X(\omega)Y(\omega)\;d\mathbb{P}(\omega).$$ The integral defining this inner product can be calculated as a Lebesgue integral on the range of the random vector $(X,Y)$ (in particular, an integral over $\mathbb{R}^{2}$) in the usual way by changing variables and using the appropriate push-forward (distribution) measures of $X$ and $Y$ (and densities if the distributions are absolutely continuous with respect to Lebesgue measure).

Measure Theoretic Approach to Linear Regression

In this post we discuss the basic theory of linear regression, mostly from the perspective of probability theory.  At the end we devote some time to statistical applications.  The reason for focusing on the probabilistic point of view is to make rigorous the actual use of linear regression in applied settings; furthermore, the approach clarifies what linear regression actually is (and isn't). Indeed, there seems to be wide gaps in understanding among practitioners, probably due to the ubiquitous use of the methodology and the need for less mathematically inclined people to use it.

Regardless of the perspective, we have a probability (sample) space $(\Omega,\mathcal{F},\mathbb{P})$ and real-valued $\mathcal{F}$-measurable random variables $X$ and $Y$ defined on $\Omega$.  What differentiates the probabilistic and statistical point of views are what we know about $X$ and $Y$.  In the former, we know their distributions, i.e. all of the statistical properties of $X$ and $Y$ on their (continuous) range of values.  In the latter case we have only samples $\{(x_{n}, y_{n})\}_{n\geq0}$ of $X$ and $Y$.  In regression theory, the goal is to express a relationship between $X$ and $Y$ and associated properties, a much easier task from the probabilistic point of view since we have a parametric (or at least fully determined) distributions.  From the statistical point of view, due to lack of a complete distribution, we have to make certain modeling, distributional, and even sampling assumptions in order to make useful inferences (without these assumptions, the inferences we would be able to make would be far too weak to serve any useful purpose).  We will discuss some of these assumptions that are typical in most regression applications later.  For now, we focus on $X$ and $Y$ with known distribution measures $\mu_{X}$ and $\mu_{Y}$.

Analyzing the Definition of Independence

One of the most fundamental concepts of probability theory is that of independence.  The concept is intuitive and captures the idea that two experiments are independent if the outcome of one does not affect the outcome of the other.  What we mean here by experiment is a measurable space $\Omega$ (called the sample space) consisting of points $\omega\in\Omega$ that represent all the possible outcomes of our experiment, and a $\sigma$-algebra $\mathcal{F}$ consisting of all possible combinations of outcomes (events) represented by subsets $A\in\Omega$.  Additionally, there is a measure $\mathbb{P}$ that assigns $1$ to the entire sample space $\Omega$ and that is countably additive on $\mathcal{F}$ in the sense that whenever $\{A_{n}\}_{n\geq0}$ is an at-most countable set of disjoint events of $\mathcal{F}$ we have $\mathbb{P}(\cup_{n} A_{n})=\sum_{n}\mathbb{P}(A_{n})$.  We then have the following formal definition of independence:

Gaussian Form of Geometric Brownian Motion

In numerical analysis, it can in some cases be more efficient to evaluate a function when it is expressed in a certain form.  For example, it is numerically more efficient to compute a second degree polynomial
$$P_{2}(x)=a_{2}x^{2}+a_{1}x+a_{0}$$
 when it is in the form
$$P(x)=x(a_{2}x+a_{1})+a_{0}.$$
That is because evaluating the former requires three multiplication operations and two addition operations, whereas the latter requires only two multiplication operations and two addition operations.  This way of evaluating polynomials numerically is known as Horner's Method, which describes the general formulation for a polynomial of degree $n$ and coefficients $\{a_{j}\}_{0\leq j\leq n}.$  A simple implementation of the algorithm in C can be found here.  In general, evaluation of a $n$ degree polynomial $P(x)$ will require $2n-1$ multiplication operations and $n$ addition operations using the naïve method, whereas it will require just $n$ multiplication operations and $n$ addition operations using Horner's Method.  Although from the point of view of algorithmic analysis both methods are equivalent (they both require $O(n)$ operations to compute $P(x)$), on a practical/implementation level, Horner's Method requires only $2n$ operations whereas the naïve approach requires $3n-1$ operations.  On the other hand, since the savings come from only the multiplication operations, the performance gain (unless each $a_{j}$ and $x$ are all integers) is somewhat tempered by the fact that for most computing hardware floating point multiplication is much faster than floating point addition (the opposite is generally true for integral arithmetic).  Note also that if $x^{2}$ is expressed explicitly as $x*x$, then a modern compiler will very likely optimize the naïve expression into something similar to Horner's Method; however, for high degree polynomials, expanding the powers with the multiplication operator like this is not feasible without producing very ugly, static, and difficult to maintain code - a power function of some sort is likely to be used and it is then significantly less likely the compiler will be able (or if it can, choose) to optimize the resulting code).

A Primer in Harmonic Analysis

I picked these problems from Modern Fourier Analysis Vol I - I think that they serve as a good primer for the basic techniques and theorems in harmonic analysis (a subject that I have recently started looking back into in order to deal with some of the techniques used when Levy processes in mathematical finance).

Problem I.  Fix $d\geq1$ and suppose $\psi:(0,\infty)\mapsto[0,\infty)$ is $C^{1}$, non-increasing, and $\int_{\mathbb{R}^{d}}\psi(|x|)\;dx\leq A<\infty.$  Define
$$[M_{\psi}f](x):=\sup_{0<r<\infty}\frac{1}{r^{d}}\int_{\mathbb{R}^{d}}|f(x-y)|\psi\left(\frac{|y|}{r}\right)\;dy$$
and show that $$[M_{\psi}f](x)\leq A[Mf](x)$$
where $M$ is the usual Hardy-Littelwood maximal function.

Solution.  We first observe that the translation invariance of the indicated estimate implies that it is sufficient to prove the case $x=0$ (this can be seen more explicitly by replacing $f$ by $\tau_{x}f$, where $\tau_{x}$ is the translation by $x$ operator, and applying the present case to be proven to see then that the estimate holds for all $x$).   For convenience let us define $\psi_{r}(|y|)=r^{-d}\psi(|y|/r)$.  The radial properties of the terms in the estimate suggest polar coordinates will be useful in dealing with the resultant integrals.  Let us recall that the polar coordinate formula implies as a consequence of itself that
$$\frac{d}{ds}\int_{B(0,s)}f(y)\;dy=\frac{d}{ds}\int_{0}^{s}dt\int_{\partial B(0,t)}f(\omega)\;dS(\omega)=\int_{\partial B(0,s)}f(\omega)\;dS(\omega)=s^{d-1}\int_{S^{d-1}}f(s\omega)\;dS(\omega).$$