As many might think, gRPC doesn’t stand for “google Remote Procedure Call” but is a recursive acronym meaning “gRPC Remote Procedure Call”. I don’t know if you buy it but the truth is that is was originally developed by Google and then open-sourced.
If you’ve been in the IT for a while RPC doesn’t necessarily bring back happy memories. On the JVM it all started with RMI in the 90s. RMI was inspired by CORBA and suffered from a lack of interoperability as both the client and the server had to be implemented in Java. RMI was also particularly slow as Java serialisation is not a very efficient protocol.
Later in the 2000s came XML based RPC with XML-RPC and especially SOAP. Both of these formats address the interoperability as it no longer matters how the client/server are implemented. They only need to speak XML. However XML is still not an efficient protocol and communications remain slow.
SOAP provides an interesting definition language (WSDL – Web Service Definition Language) that can be used to generate service implementations.
gRPC addresses all these drawbacks. By default, it uses protobuf (Protocol buffers) for the service definitions and as its serialisation mechanism, which allows it to interoperate with many different languages while providing an efficient serialisation protocol. gRPC also takes advantage of HTTP/2 to add streaming capabilities.
Unfortunately Scala is not in the list! … but we have scalaPB (and sbt-protoc) to save the day! Continue reading “gRPC in Scala”
Many people see Kafka as a messaging system but in reality it’s more than that. It’s a distributed streaming platform. While it can be used as a traditional messaging platform it also means that it’s more complex.
In this post we’ll introduce the main concepts present in Kafka and see how they can be used to build different application from the traditional publish/subscribe all the way up to streaming applications. Continue reading “Kafka concepts and common patterns”
Today’s focus is on scalameta. In this introduction post we’re going to see how to create a macro annotation to generate protobuf formats for case classes.
The idea is to be able to serialise any case classes to protobuf just by adding a
@PBSerializable annotation to the case class declaration.
Then behind the scene the macro will generate implicit formats in the companion object. These implicit formats can then be used to serialise the case class to/from protobuf binary format.
This is quite similar to Json formats of play-json.
In this post we’re going to cover the main principles of scalameta and how to apply them to create our own macros. Continue reading “Generating protobuf formats with scala.meta macros”
Protocol Buffer (aka Protobuf) is an efficient and fast way to serialise data into a binary format. It is much more compact than Java serialisation or any text-based format (Json, XML, CSV, …).
Protobuf is schema based – it needs a description (in a .proto file) of the data structures to be serialised/deserialised.
On the JVM, protoc (the Protobuf compiler) reads the .proto description files and generates corresponding classes.
For Scala there is a very good sbt plugin “scalaPB” that follows the same process and generates case classes corresponding to the .proto files definitions.
The .proto files are an easy way to describe a protocol between 2 components (e.g. services). However there are some cases (e.g. writing to persistent storage) where the .proto files definition are just unnecessary and add superfluous complexity. (Who likes to read auto-generated code?).
In such cases it would be much easier to serialise an object directly into protobuf (using its class definition as a schema). Afterall this is what the protobuf java binding does: it serialises (auto-generated) java classes into protobuf binary format.
To that matter, let me introduce – PBDirect – a scala library to directly encode scala objects into protobuf. Continue reading “PBDirect – Protobuf without the .proto files”
Streams processing have been around for a while and encompasses a great number of applications:
- HTTP servers handling stream of incoming HTTP requests
- Message streams: Twitter hose, user posts, …
- Time-series messaging: stream from IoT sensors
- Database querying: result set contains a stream of record
Most interestingly reactive streams have gain traction over the past few years. They bring back-pressure into the game in order to avoid having the destination stream over flooded by messages from the source stream.
This post focuses on AkkaStream, a reactive stream implementation based on Akka actors. Unlike actors which are untyped, AkkaStreams provides type safety at every stage of the stream pipeline and also comes with a nice and fluent API. However the documentation is sometimes lacking or not easy to search when someone needs to implement common patterns. This post tries to cover the most common ones in a clear and concise way. Continue reading “Akka Streams patterns”
The actor model allows us to write complex distributed applications by containing the mutable state inside an actor boundary. However with Akka this state is not persistent. If the actor dies and then restarts all its state is lost.
To address this problem Akka provides the Akka Persistence framework. Akka Persistence is an effective way to persist an actor state but it’s integration needs to be well thought as it can greatly impact your application design. It fits nicely with the actor model and distributed system design – but is quite different from what a “more classic” application looks like.
In this post I am going to gloss over the different components of Akka Persistence and see how they influence the design choices. I’ll also try to cover some of the common pitfalls to avoid when building a distributed application with Akka Persistence.
Although Akka Persistence allows you to plug in various storage backends in this post I mainly discuss using the Cassandra backend. Continue reading “Akka persistence”
Last time we’ve seen how to deal with future inside an actor. However some times you just need to wait for a future to finish before processing the next message.
Continue reading “Synchrone computation in Akka actor”
Future and Actor are two different paradigms to express concurrent computations. Both of them are perfectly valid abstractions. However one must be careful when it comes to mixing them up.
Continue reading “Futures in actor”
People regularly complains about the lack of type safety within Akka actors. After 2 rather unsuccessful attempts (using byte-code generation at runtime for the first and java proxies for the second) the third attempt seems much more promising.
Let’s start with regular non-typed actors to implement a very basic toy example and then move on towards a fully typed actor.
Continue reading “The return of the typed Actor”
Stream computing is one of the hot topic at the moment. It’s not just hype but actually a more generic abstraction that unifies the classical request/response processing with batch processing.
The request/response is a 1-1 scheme: 1 request gives 1 response. On the other hand the batch processing is an all-all scheme: all requests are processed at once and gives all response back.
Stream processing lies in between where some requests gives some responses. Depending on how you configure the stream processing you lie closer to one end than the other.
Continue reading “Kafka streams”