Markov Decision Process

Tweet about this on TwitterShare on LinkedInShare on FacebookShare on Google+Share on Reddit

Now that we know about Markov chain, let’s focus on a slightly different process: the Markov Decision Process.

This process is quite similar to a Markov chain but adds more concept into it: Actions and Rewards. Having a reward means that it’s possible to learn which action yield the best rewards. This type of learning is also known as reinforcement learning.

In this post we’re going to see what exactly is a Markov decision process and how to solve it in an optimal way. Continue reading “Markov Decision Process”

Hidden Markov Model

Tweet about this on TwitterShare on LinkedInShare on FacebookShare on Google+Share on Reddit

Last time we talk about what is a Markov chain. However there is one big limitation:

A Markov chain implies that we can directly observe the state of the process. (e.g the number of people in the queue).

Many times we can only access an indirect representation or noisy measure of the state of the system. (e.g. we know the noisy GPS coordinates of a robot but we want to know it’s real position).

Trellis diagram

In this post we’re going to focus on the second point and see how to deal with HMMs. In fact HMM can be useful every time that we don’t have direct access to the system state. Let’s take some motivational examples first before we dig into the maths. Continue reading “Hidden Markov Model”

Markov chain

Tweet about this on TwitterShare on LinkedInShare on FacebookShare on Google+Share on Reddit

I’ve heard the term “Markov chain” a couple of times but never had a look at what it actually is. Probably because it sounded too impressive and complicated.

It turns out it’s not as bad as it sounds. In fact the whole idea is pretty intuitive and once you get a feeling of how things work it’s much easier to get your head around the mathematics.

Andrey Markov was a Russian mathematician who studied stochastic processes (a stochastic process is a random process) and specially systems that follow a suite of linked events. Markov found interesting results about discrete processes that form a chain. Continue reading “Markov chain”

The free monad without the boilerplate

Tweet about this on TwitterShare on LinkedInShare on FacebookShare on Google+Share on Reddit

The free monad is really neat for creating DSL as it allows to completely separate the business logic from the implementation.

It leaves a great freedom for the implementation choices and should you change your implementation you just need to rewrite your interpreter without changing any of the business logic.

That’s great! However after playing a little around with the free monad I was a bit skeptical about the amount of boilerplate required to plug everything together. Writing smart constructors and injectors is not the most trivial thing (depending on how your team is familiar with functional programming).

To conclude this serie on the free monad we’ll have a look at some solutions to remove this boilerplate code, especially freasy and freek.
Continue reading “The free monad without the boilerplate”

The free monad in (almost) real life

Tweet about this on TwitterShare on LinkedInShare on FacebookShare on Google+Share on Reddit

Now that we have a fairly good understanding of what the free monad is and how it works. Let’s see what it looks like to write a real program based on this concept. This time we won’t implement Free ourselves but use the cats library for that. If you don’t know all the theory behind the free monad and just want to see how to use it then read on as this post builds an application from scratch.

In this post we’ll write simple DSLs, lift them into the free monad and even combine them into one program. There is some boiler plate around but I want to get a feeling of what it looks like to combine more DSLs and see how we can architecture a whole application around the free monad. Continue reading “The free monad in (almost) real life”

The Free monad explained – part 2

Tweet about this on TwitterShare on LinkedInShare on FacebookShare on Google+Share on Reddit

Remember in part 1 we implemented a little program to interact with the user. We nicely separated the business logic from the implementation. And as good as it is we now need to add more functionality to our application and this exactly what we’re going to cover here.

The idea is to define a new language for user authentication and then combine it with the user interaction language to build a more complex program.
Continue reading “The Free monad explained – part 2”

The Free monad explained – part 1

Tweet about this on TwitterShare on LinkedInShare on FacebookShare on Google+Share on Reddit

My next series of blog posts will be on the Free monad. ┬áDon’t be scared by the name, it’s not so difficult to grasp it (if explains correctly) and your code could greatly improve.

Why ? Because it allows a clear separation between the program definition (i.e. the business logic) and its implementation. That means if one day you decide to change your implementation (e.g change your persistent storage) you just need to re-write your interpreter – leaving the business logic untouched.
Continue reading “The Free monad explained – part 1”

Distributed training of neural networks

Tweet about this on TwitterShare on LinkedInShare on FacebookShare on Google+Share on Reddit

Distributed training of neural networks is something I’ve always wanted to try but couldn’t find much information about it. It seems most people train their models on a single machine.

In fact it makes sense because training on a single machine is much more efficient than distributed training. Distributed training incurs additional cost and is therefore slower than training on a single machine so it must be reserved only for cases where the neural network or the data (or both) don’t fit on a single machine.
Continue reading “Distributed training of neural networks”