# EM Algorithm on linear model clusters in R

Let’s together explore another technique that it has several application which includes parameter estimation on mixtures models or hidden markov models, data clustering, and discovering hidden variables - EM Algorithm applications. To grasp the power of EM-Algorithm let’s considered an always familiar and simple linear models, and try to discover from which model was the points originated (clustering) with an iterative approach.

This is a crosspost from EM with mix of two linear models, with some personal changes to the original code.

The algorithm basically solve the following problem: given a set of observations and prior knowledge of how many distributions generated the observations, is it possible to estimate from which observation belongs to which distribution?.

## Generate random data

Create a function that generates observation based on a linear model with some gaussian noise:

To use and plot:

Now the image bellow just by simple sight two different linear pattern emerge, so the EM algorithm is used to find the hyperparameters that generated these distribution automatically.

## Pseudo code of the EM-Algorithm

### Step 1: Initialize random parameters

Since here, there is prior knowledge of 2 distributions the parameters that need to be generated are:

\[f_1(x) = \beta_0 + \beta_1X\] \[f_2(x) = \beta_1 + \beta_2X\]Assuming some uniform distribution to randomly initialize these \(\beta_n\) we have:

### Step 2: Calculate each observation probabilities

With the initialization parameters, let’s proceed to calculate the weights of each observation assuming a gaussian noise is present:

Now is a matter to use these calculated weight to solve using Weighted Least Square

As a side note is important to now that the solve function will calculate the inverse of the formula inside it..

We can use both the e.step and m.step functions iteratively to calculate the parameters continuously until it converged, or it reached a max of iterations.