Introduction to Markov Chains

An introduction to stochastic processes with a focus on Markov chains

There are numerous instances in financial modelling where a policy or contract moves between multiple states over time, with each state implying a different state of ‘health’. For example, suppose an individual has taken out a loan from the bank. For simplicity, consider only two states: performing, and not performing. If the individual is in the performing state, then they will make their monthly repayment. If the individual is in the non-performing state then they will not make their monthly repayment. In order for the bank to model the revenue they will make from this loan, it is necessary to model how the borrower will move between these two states over the duration of the loan. Markov chains are a common tool used for this.

Before we can dive into the Markov chain illustration, some key definitions are required. A stochastic process is a family or set of ordered random variables. We will use X(t) to denote the value of the stochastic process at time t. The state space is the set of values that the random variable X(t) can take. The state space and the time set (i.e. the values that t takes) can both either be discrete or continuous — and do not have to be either both discrete or both continuous. A sample path is a particular realisation of the stochastic process.

Generally, the random variable X(t) will depend on the earlier values of the stochastic process. A special case is stochastic processes that satisfy the Markov property — referred to as Markov processes. If a stochastic process satisfies the Markov property then the random variable X(t) will depend only on the previous value of the process. If a Markov process has both a discrete state space and a discrete time set then it is referred to as a Markov chain.

Returning to our example, we can now define the setup more rigorously. Suppose an individual borrows £100 from the bank and agrees to repay the loan (with interest of course!) in 12 equal monthly instalments of £10. In each month the borrower is in one of two states, denoted H and D for performing and non-performing respectively as we defined initially. Suppose the probability of a performing borrower transitioning to a non-performing borrower is 0.5% each month, and the probability of transitioning from non-performing to performing is 0.25% each month. Since these probabilities do no vary over time, we say the chain is time-homogenous. This Markov chain can be illustrated in the following diagram and matrix:

Two state transition diagram

Two state transition matrix

There are a few properties of this Markov chain that are worth mentioning. Firstly, this chain can be described as being irreducible because each state can be reached from any other state. Secondly, the chain is aperiodic (or has a period of 1) because a return to either state is possible in 1 step — i.e. for each state there is no number d > 1 such that return to that state is only possible in a number of steps that is a multiple of d. The stationary distribution of a Markov chain describes its long-term behaviour, specifically the proportion of time that the chain is expected to spend in each state in the long run. Since the Markov chain is irreducible and has a finite state space, it has a unique stationary probability distribution. These properties can all be extracted using the markovchain package in R:

#creating a markov chain object
mc = new("markovchain", transitionMatrix = P, states = states, name = "LoanHealth") 

#extracting the properties of the chain
is.irreducible(mc)
period(mc)

#stationary distribution
stationary_dist = steadyStates(mc)

Let our random variable X(t) denote the state the borrower is in at time t, where t takes on values in the set {0,1,2,…,12}. We will set X(0) = D. We can model the banks profit (or loss) on this loan using the Markov chain. We could simulate a sample path of X(t) from t=1 to t=12, but that would only give us the profit the bank makes for that particular sample path. To determine the banks expected profit, we could proceed as follows: for each time step determine the probability that the borrow is in state H, given that the borrower is in state H at time t=0. For each time step this probability is the first entry of the transition matrix Pᵗ, i.e. the initial transition matrix raised to the power of t. This approach is equivalent to beginning with the row vector representing the starting distribution and computing the dot product of the starting position and the transition matrix P to produce the distribution at time t=1, and then taking the dot product of the distribution at time 1 and the transition matrix P to produce the distribution at time 2, and so on. Although tedious to compute by hand, this is straight forward in R.

Using the second approach, the expected revenue the bank can expect to make on this loan is £116.21 , so their expected profit on the £100 loan (assuming no costs) is £16.21, or 16.21%. This is less than the 20% interest they charge on the loan because each month there is a small probability that the borrower does not make that month’s repayment.

#setting up the loan information
loan_amount = 100
repayment = c(10, 0)
prior_dist = c(1,0)
expected_repayment = numeric(12)

#calculating the expected profit
for (i in 1:12){
  new_dist = prior_dist %*% P
  expected_repayment[i] = sum(new_dist * repayment)
  prior_dist = new_dist
}

expected_revenue = sum(expected_repayment)
expected_profit = expected_revenue - loan_amount

Another aspect of the Markov chain that would be of interest to the bank would be its long term behaviour. If the bank were to lend money on a much longer term, for instance a 20 year mortgage loan, it is valuable to know what percentage of months it can be expected that the borrower will be in the performing state as this will inform the banks revenue expectations. As we noted above, this chain has a unique stationary distribution. Finding this stationary distribution requires solving the following system of equations, which we solve below:

As you would have noticed from the section on the properties of this Markov chain, the station distribution is easily computed in R, which will be particularly useful for chains with more than two states.

The concepts discussed here can be extended to model more complex dynamics, such as several states, state-dependent repayment amounts, and time-dependent transition probabilities.

The repository with the R code to reproduce the diagram and calculations can be found here: https://github.com/ValentineChisango/A212-CS2