Neural network hardware considerations

This post will present some principles to be considered when choosing the hardware that will run some neural net computation, either for training models or making prediction using an existing model (i.e. inference).

Need more cores

Usually CPU are considered to not perform well enough when it comes to neural network computing and they are now outperformed by GPUs.

CPUs run faster then GPU but they are not design to perform many parallel operations simultaneously which is precisely what GPU are made for.
Continue reading “Neural network hardware considerations”

Kafka streams

Stream computing is one of the hot topic at the moment. It’s not just hype but actually a more generic abstraction that unifies the classical request/response processing with batch processing.

Stream paradigm

The request/response is a 1-1 scheme: 1 request gives 1 response. On the other hand the batch processing is an all-all scheme: all requests are processed at once and gives all response back.

Stream processing lies in between where some requests gives some responses. Depending on how you configure the stream processing you lie closer to one end than the other.
Continue reading “Kafka streams”

Introduction to Alluxio

Continuing my tour of the Spark ecosystem today’s focus will be on Alluxio, a distributed storage system that integrates nicely with many compute engines – including Spark.

What is Alluxio ?

The official definition of Alluxio is (or at least that’s how one of its author presents it):

Alluxio is an open source memory speed virtual distributed storage

Let’s see what each of these terms actually means:
Continue reading “Introduction to Alluxio”

Neural network implementation guidelines

Today to conclude my series on neural network I am going to write down some guidelines and methodology for developing, testing and debugging a neural network.

As we will see (or as you already experienced) implementing a neural network is tricky and there is often a thin line between failure and success – between something that works great and something making absurd predictions.

The number of parameters we need to adjust is just great: from choosing the right algorithm, to tuning the model hyper-parameters, to improving the data, ….

In fact we need a good methodology and a solid understanding of how our model works and what is the impact of each of its parameters.
Continue reading “Neural network implementation guidelines”

Apache Spark data structures

Apache Spark is a computation engine for large scale data processing. Over the past few months a couple of new data structures have been available. In this post I am going to review each data structure trying to highlight their forces and weaknesses.

I also compares how to express a basic word count example using each data structure.
Continue reading “Apache Spark data structures”

Recurrent Neural Network

After introducing the convolutional neural networks I continue my serie on neural networks with another kind of specialised network: the recurrent neural network.

Principle

The recurrent neural network is a kind of neural network that specialises in sequential input data.

With traditional neural network sequential data (e.g. time series) are split into fixed-sized windows and only the data points inside the window can influence the outcome at time t.

With recurrent neural network the network can remember data points much further in the past than a typical window size.
Continue reading “Recurrent Neural Network”

Convolutional Neural Network

Inspiration

Convolutional Neural Networks are a kind of network inspired by the cats’ visual cortex.

A cat visual cortex is made of 2 distinct type of cells:

  • simple cells which specializes into edge detection.
  • complex cells with larger receptive field which are sensitive to a small region of the visual field and are less sensitive to the exact position of edges.

Convolutional neural network are inspired by the latter type of cells. Each neuron is sensitive to a small region of the input data and less to a specific position of a pattern.
Continue reading “Convolutional Neural Network”

Keras – Tensorflow and Theano abstraction

As we’ve seen in the Tensorflow introduction having access to the computation  is a powerful feature. We can define any operation we’d like and tensor flow (or Theano) will compute the gradient and perform the optimisation for us. That’s great!

However if you always define the same kind of operation you’ll eventually find this approach a bit tedious. This is where we need a higher level of abstraction that allows us to define our neural net in terms of layer and not in terms of operations.
Continue reading “Keras – Tensorflow and Theano abstraction”

Neural network design

Today I continue my neural network post series with some considerations on neural network implementation.

So far we covered what is a neural network and how it works but we are still left with numerous choices regarding its design.

How many layers should we use, how many units (neurons) in each layer, which activation functions, which cost function, … ? There are so many questions and choices to make that it has bothered me for quite some time now.

If you scroll the web you may find some advice on these questions. But this is it – you can only get advice as there is no clear answers. It’s just trial and errors so you’d better try for yourself and see how different designs perform on your problem.
Continue reading “Neural network design”