As ICA (Independent Components Analysis) is one a large family of Blind Source Separation (BSS) methods, so let’s begin with a general introduction to BSS.
Mixing and Unmixing Signals
Central to the problem tackled by BSS methods is that of mixing and unmixing. We can think of the signal on which we wish to apply BSS as our signal mixture; it is a mixture of signals, which we can refer to source signals. These source signals, however, are unmixed signals; in other words, they are the signals that we wish to recover by applying the BSS method.
We can start to tackle the question of how to unmix our signal mixture by looking at what happens when we mix signals. Here, we are considering, in particular, the linear combination of source signals.
When we mix two source signals (e.g. speech signals) to yield several signal mixtures, we can consider that this leads to at least 3 principal effects, each of which can form the basis for unmixing:
- Loss of Independence: We can presume that our source signals are independent. But, as the two signals are shared between the signal mixtures, we cannot say that say that the signal mixtures are independent.
- Normality/Gaussianity: The distribution of signal mixtures is generally more gaussian than its constituent source signals.
- Complexity: In general, it is found that the complexity of a signal mixture is greater than or equal to that of the least complex constituent source signal. This, therefore, implies that by identifying the least complex signal from a set of signal mixtures, we can arrive at a source signal. However, do note that this works best when our source signals are very simple, such as pure sinusoidal tones.
Put together, these observations suggest that source signals can extracted from mixtures by identifying signals that maximise independence, that have a signal distribution approaching gaussian and that have low complexity. Sounds easy…
The Cocktail Party Problem
To look at the process of mixing in a more concrete manner, we can consider the cocktail party problem.
Our cocktail party (figure1) is animated by two speakers (S1 and S2), who unfortunately insist on speaking at the same time. Each of the speakers voice is picked up by two microphones, positioned at a certain distance from each speaker. Because each microphone receives a combination of both speakers’ voices, we call the signal picked up by the microphone as signal mixtures (M1 and M2). Besides illustrating signal mixing, this figure also illustrates an important property of ICA – namely that there should be at least as many mixture signals (microphones) as source signals (speakers).
If we consider that the relative contribution of each speaker to the input at each microphone is determined by the distance of each from each microphone as well as the relative loudness of their voices. Thus, we can say that the mixture signal at each microphone is a weighted combination the two source signals. In figure 2, a, b, c and d correspond to coefficients that determine the contribution of each source signal, yielding the following expression:
x 1 = aS1 + cS2 and x 2= bS1 + dS2
Immediately, we can appreciate that we could recover the source signals if we know the coefficients (a, b, c and d).
The above figure summarizes the source-mixture signal relationship as presented in our simplified cocktail party problem. You can also notice that the mixing coefficients are presented in matrix form (A). Matrix operations are central to linear algebra and, here, we can say that the mixing coefficients (a, b, c, d) linearly transform the source signals to mixture signals. Also, in solving ICA, it is very helpful to conceptualise the problem as linearly transforming a source space (space S) to a mixture space (space X).
Mixing Signals
So we have generated signal mixtures, in a linear manner, from a set of source signals. This means that each source signal can be recovered by recombining the signal mixtures. How the signal mixtures are recombined to recover the source signal is determined by a set of unmixing coefficients.
Importantly, the problem of finding the unmixing coefficients is the problem that is tackled by all blind source methods in general.
The above figure summarizes the mixing process: the mixing coefficients (a, b, c, d) linearly transform the source signals (space S) into the mixture signals (space X). Mathematically, this can be summarized as:
S →X :X=AS
…where A ∈ RpXM is the mixing coefficients matrix.
Note: In the above example we are dealing with temporal signals, signals,however, for ICA, in its general form, we do not have to take account of the time dimension as the ICA model assumes that the source and mixture signals are random variables, rather than temporal series.
So, some remarks concerning the mixing/unmixing process tackled by ICA or our Cocktail Party Problem:
- It involves separating a set of source signals from a set of mixture signals.
- To extract the source signals we can only use what we know and/or assume about our source signals and our mixture signals.
- We can refer to the source signals as latent variables, as we cannot observe them directly.
- Our mixture signals correspond to the observed data.
Unmixing
As the name suggest, the aim of unmixing is to reverse the effect of mixing. So, mathematically, if we have coefficients that determine the way in which our sources or latent variables are mixed, there should exist a set of coefficients that will also us to recover our source signals from the mixture signals; we can refer to these coefficients as our unmixing coefficients.
In the above equation, the unmixing coefficients are defined as :
If we remember, from above, our mixing matrix, which was defined as:
And, if we posit that the process of unmixing is the inverse of mixing, we could hypothesis the following :
…that we find W by computing the inverse of A…
But, unfortunately, we generally don’t know A, so things are not so simple.