As promised in my previous post we’re going to explore to internal of Fluent and how it uses Shapeless and implicit resolution to transform case classes.
Fluent started as an experiment (and still is), the code is rather small (about 300 lines of code) and yet I am still impressed by the variety of cases it can handle.
Before working with Shapeless I’ve often heard that is pure magic and I got the impression that most people (including me) don’t really know how it works. It turns out that the principles used in Shapeless are not really difficult to understand – especially if you read the well-written Type Astronaut’s guide to Shapeless.
Understanding how Shapeless works doesn’t mean it’s easy to work with. Actually Shapeless makes a heavy use of implicits and working with implicits is hard. Remember that implicits resolution is performed at compile time so when it fails, there is nothing to debug, no log messages or stack trace. We are just left with rather blunt messages like could not find implicit value for parameter ...
In this post I am going to explain the concept used in Fluent, the problem I faced during implementation and hopefully by the end of the post, you’ll know enough to understand and edit the code (Pull requests welcomed!).
Principle
At the heart of Fluent is a type class: Transformer.
trait Transformer[A, B] {
def apply(a: A): B
}
That’s quite simple, a Transformer[A, B] is something that can transform an instance of A into an instance of B.
Then we have an implicit class that adds a method transformTo[B] to any instance A for which it exists an implicit Transformer[A, B] who can turn A into B.
implicit class TransformerOps[A](a: A) {
def transformTo[B](implicit transform: Transformer[A, B]): B =
transform(a)
}
That really is the heart of Fluent. The remaining of the code uses Shapeless to automatically derive Transformer type class instances.
Deriving type class instances
As explained in this previous post we can use Shapeless’s Generic to convert case classes into/from HLists.
import shapeless._ case class Circle(x: Double, y: Double, radius: Double, colour: String) val circle: Circle = Circle(1.0, 2.0, 3.0, "Blue") val repr: Double :: Double :: Double :: String :: HNil = Generic[Circle].to(circle) println(repr) // prints: 1.0 :: 2.0 :: 3.0 :: Blue :: HNil
That’s a good start but we need the field names as well otherwise there is no way to pick the right value between x, y and radius as they’re all of type Double.
To this matter Shapeless provides us with Singleton types. Let’s take an example: 1 is an Int so we can consider it as a subtype of Int which is just the value 1.
def onlyOne(one: Witness.`1`.T) = one onlyOne(1) // works onlyOne(2) // doesn't compile (type mismatch)
We can do the same with strings – "Hello" is a subtype of String.
def onlyHello(hello: Witness.`"Hello"`.T) = hello
onlyHello("Hello") // works
onlyHello("Hi") // doesn't compile (type mismatch)
That’s great we can use singleton types to represent our field names at the type level. What’s even better is that Shapeless does it for us if we use LabelledGeneric instead of Generic.
val labelledRepr = LabelledGeneric[Circle].to(circle) println(labelledRepr) // prints: 1.0 :: 2.0 :: 3.0 :: Blue :: HNil
Wait! LabelledRepr looks exactly the same as repr. That’s right but its type looks different:
Double with KeyTag[Symbol with Tagged[String("x")],Double] ::
Double with KeyTag[Symbol with Tagged[String("y")],Double] ::
Double with KeyTag[Symbol with Tagged[String("radius")],Double] ::
String with KeyTag[Symbol with Tagged[String("colour")],String] ::
HNil
Basically it’s still a HList but with refined types. Each refined type is called a Record.
Now that we know which is which we can go back to our generic derivation of Transformer type class instances.
The easiest case is that given a Transformer[A, BRepr] and a LabelledGeneric.Aux[B, BRepr] we can easily make a Transformer[A, B].
implicit def genericTransformer[A, B, BRepr <: HList](implicit
generic: LabelledGeneric.Aux[B, BRepr],
transform: Lazy[Transformer[A, BRepr]]
): Transformer[A, B] = new Transformer[A, B] {
override def apply(a: A): B = generic.from(transform.value(a))
}
That’s one step in the right direction, but we still need a Transformer[A, BRepr]. BRepr is an HList so we can construct it recursively. The stop case is easy, it’s the one who turns an A into an HNil.
implicit def hnilTransformer[A]: Transformer[A, HNil] = new Transformer[A, HNil] {
override def apply(a: A): HNil = HNil
}
Now the recursive case. Basically we need a Transformer for the tail of the HList and a Transformer for the head.
implicit def hlistTransformer[A, ARepr <: HList, F, K, V, T <: HList](implicit
generic: LabelledGeneric.Aux[A, ARepr],
selector: Selector.Aux[ARepr, K, F],
transformHead: Transformer[F, V],
transformTail: Transformer[A, T]
): Transformer[A, FieldType[K, V] :: T] = new Transformer[A, FieldType[K, V] :: T] {
override def apply(a: A): ::[FieldType[K, V], T] = {
val h = transformHead(selector(generic.to(a)))
field[K](h) :: transformTail(a)
}
}
It looks complicated, so let’s break it down. First we need to turn A into an HList. That’s the job of the LabelledGeneric.Aux[A, ARepr].
Once we have ARepr we need to pick the right field from the List. That’s why we need a Selector.
Once we have picked the right field we need to transform it into the type expected in BRepr. In our example we’re going to turn a Double into a Double so the identity transformer will do.
At this point we’re ready to compute the head of BRepr. Remember it needs to be a Record which we can easily construct : transformHead(selector(generic.to(a))) gives the expected head of type V which we then refine into a Record by calling field[K](h).
Once we have our new head we just need to append the tail that we compute directly using our tail transformer.
The only missing bit is the identity transformer that I mentioned above:
implicit def identityTransformer[A]: Transformer[A, A] = new Transformer[A, A] {
override def apply(a: A): A = a
}
Example
Now that we’re all setup let’s turn our Circle into a Point.
case class Point(x: Double, y: Double) val point = circle.transformTo[Point]
If we print every step of the execution, this is what we get:
- IdentityTransformer: 1.0 -> 1.0 - IdentityTransformer: 2.0 -> 2.0 - HNilTransformer: Circle(1.0,2.0,3.0,Blue) -> HNil - HListTransformer: Circle(1.0,2.0,3.0,Blue) -> 2.0 :: HNil - HListTransformer: Circle(1.0,2.0,3.0,Blue) -> 1.0 :: 2.0 :: HNil - GenericTransformer: Circle(1.0,2.0,3.0,Blue) -> Point(1.0,2.0)
As you can see we use all the transformers we have defined above. It’s interesting to see how the HList of B representation gets constructed recursively.
Common pitfalls
Ambiguous implicits
The remaining of the Fluent code is just more implicit transformers. However as I added more implicit definition I ended up the following error `ambiguous implicit values`.
It happens when there is more than one implicit of a given type in scope. I encountered this problem when I added support for user-defined function. The initial implementation used to turn the implicit function into a Transformer
implicit def customTransformer[A, B](implicit f: A => B): Transformer[A, B] = new Transformer[A, B] {
override def apply(a: A): B = f(a)
}
and then transform A into B‘s HList with the following:
implicit def hlistGlobalTransformer[A, K, V, T <: HList](implicit
transformHead: Transformer[A, V],
transformTail: Transformer[A, T]
): Transformer[A, FieldType[K, V] :: T] = new Transformer[A, FieldType[K, V] :: T] {
override def apply(a: A): ::[FieldType[K, V], T] =
field[K](transformHead(a)) :: transformTail(a)
}
The problem is that now both hlistTransformer and hlistGloablTransformer conflicts with each other. But what I really wanted to do was to use hlistGlobalTransformer only when the user provides an implicit function and default to the hlistTransformer when hlistGlobalTransformer is not applicable.
The problem is that hlistTransformer is always applicable so if hlistGlobalTransformer is also applicable we end up with the ambiguous implicit values error.
We need some sort of priority rules between the possible implicit values. To solve this issue we have to turn to the Scala implicit resolution rules. It turns out that the compiler looks first into the current instance definition and then into its parent definition (i.e. the class or traits that the current object extends).
We can then move the hlistTransformer into a lower priority trait which the Transformer companion object extends.
trait LowPriorityImplicits {
implicit def hlistTransformer[A, ARepr <: HList, F, K, V, T <: HList](implicit
generic: LabelledGeneric.Aux[A, ARepr],
selector: Selector.Aux[ARepr, K, F],
transformHead: Transformer[F, V],
transformTail: Transformer[A, T]
): Transformer[A, FieldType[K, V] :: T] = new Transformer[A, FieldType[K, V] :: T] {
override def apply(a: A): ::[FieldType[K, V], T] = {
val h = transformHead(selector(generic.to(a)))
field[K](h) :: transformTail(a)
}
}
}
object Transformer extends LowPriorityImplicits {
implicit def hlistGlobalTransformer[A, K, V, T <: HList](implicit
transformHead: Transformer[A, V],
transformTail: Transformer[A, T]
): Transformer[A, FieldType[K, V] :: T] = new Transformer[A, FieldType[K, V] :: T] {
override def apply(a: A): ::[FieldType[K, V], T] =
field[K](transformHead(a)) :: transformTail(a)
}
// more implicits here ...
}
This is actually working quite nicely and used in multiple places in Shapeless.
Note: that in the current implementation the hlistGlobalTransformer has been moved into a low priority trait and user defined functions are supported with higher-priority by the hlistCustomTransformer who takes the user’s implicit function as a parameter.
Diverging implicits
When using recursive implicit resolution it’s common to encounter the `diverging implicit expansion for type …`. Fortunately Shapeless provide us with the Lazy macro where all we have to do is wrapped our type into a Lazy construct and use value to get the wrapped value.
This is what is done in the genericTransformer:
implicit def genericTransformer[A, B, BRepr <: HList](implicit
generic: LabelledGeneric.Aux[B, BRepr],
transform: Lazy[Transformer[A, BRepr]]
): Transformer[A, B] = new Transformer[A, B] {
override def apply(a: A): B = generic.from(transform.value(a))
}
Cats
Cats was introduced to deal with repeated or optional fields. It is used for the functorTransformer.
This transformer can turn any F[A] into an F[B] given that F is a functor (we can map over it) and a transformer from A to B is available.
implicit def functorTransformer[F[_], A, B](implicit
functor: Functor[F],
transform: Transformer[A, B]
): Transformer[F[A], F[B]] = new Transformer[F[A], F[B]] {
override def apply(a: F[A]): F[B] = functor.map(a)(transform.apply)
}
That’s why you then need to import cats.instances.all._ when you use Fluent. You can of course be more specific and import only the cats instances you need.
DeepHLister
The DeepHLister comes in handy when transforming nested case classes into a flatter one. Let’s consider this use case:
case class Cylinder(origin: Point, radius: Double, height: Double, colour: String)
If we want to turn a Cylinder into a Circle we need to get the x and y inside the Point.
If we use a LabelledGeneric to get the List for the Cylinder we get something like:
Point with KeyTag[Symbol with Tagged[String("origin")],Point] ::
Double with KeyTag[Symbol with Tagged[String("radius")],Double] ::
Double with KeyTag[Symbol with Tagged[String("height")],Double] ::
String with KeyTag[Symbol with Tagged[String("colour")],String] ::
HNil
There is no x and y available here. For that we need to further expand Point into
Double with KeyTag[Symbol with Tagged[String("x")],Double] ::
Double with KeyTag[Symbol with Tagged[String("y")],Double] ::
HNil
but we’d like to get a “deep” HList for our Cylinder in order to get something like
Double with KeyTag[Symbol with Tagged[String("x")],Double] ::
Double with KeyTag[Symbol with Tagged[String("y")],Double] ::
Double with KeyTag[Symbol with Tagged[String("radius")],Double] ::
Double with KeyTag[Symbol with Tagged[String("height")],Double] ::
String with KeyTag[Symbol with Tagged[String("colour")],String] ::
HNil
This is the job of our DeepHLister. It is inspired by this example and adapted to work with LabelledGeneric and flatten the resulting HList using hlist’s Prepend operation.
Conclusion
This is a rather long post but hopefully it gives a pretty good understanding of Fluent implementation and how it uses Shapeless’s generic programming to derive Transformer type class instances.
Remember that Fluent is still very experimental and won’t cover all possible cases. If you find yourself in such situation you should be able to define your own implicit function or even submit a PR to handle your use case. Any contribution is welcomed!