Five reasons why you should use Scala on Android

Last week I've introduced the basic building blocks of a Scala Android App and talked only very short about the motivation. This time I will give you five catchy reasons why you should not go mainstream and use plain old boring Java but give Scala a favour on Android.

1. Escape the callback hell with Future's

On Android you have to spin off a thread for everything that needs some time until it's finished. You can't do this on the main thread, because then you would block the responsiveness of your app. This work could be e.g. a call to a REST server to fetch or post some data.
AsyncTask from the Android library helps Android Java developers with this problem. But it uses callbacks as soon as the task completes. This doesn't sound so bad if you only do one thing, but just think about dependent async calls: First fetch user data, then fetch the avatar whose link has been contained in the returned user data. Now you have to nest callback handlers which really leads to ugly code. If you're not convinced read "Callbacks are the modern gotos".

Fortunately functional languages like Scala have a very nice solution for this problem and they are called Monad's. In our concrete case of async execution the Monad to use is Future. You just have to return a Future of your return type instead of returning it directly. The nice thing is now that you can combine Future's very easily using a for comprehension:
With that mechanism you may combine several inter dependent async calls and they nearly look as simple as synchronous code.

2. Use concise functions instead of verbose anonymous listener classes

Android's UI code makes heavy use of the Listener concept like several other Java based UI frameworks like Swing. Every java developer knows the extreme amount of boilerplate code that is needed to write an anonymous listener class:
(Yes I know, this is already Scala, but it still uses the standard android listener)

You may now simply define a function that wraps the ugly listener concept or simply use Scaloid which already supplies you with one:
If you prefer not to use Scaloid you just have to use the "Pimp my library" Pattern of Scala and enrich every Android View with a method that accepts an "on click" handler function:
Now you simply import the implicit class and you may use e.g. button.onClick() like shown above.

3. Concise Scala code vs verbose Java boilerplate

Java is still ultra verbose and you as a developer need powerful IDE's to generate hundreds of boring code lines like this:
In Scala the above is simply done with:
Case classes generate everything for you: equals, hashCode, getter's, toString. No setter's since Scala case classes teach you to favour immutability. But Case class have clever copy methods if you need to create an altered version of an object.

4. Use compile time checked dependency injection with the cake pattern

You don't need Spring's Dependency Injection facilities or Guice with Scala. You'll have something more powerful without the need to use an external library. Just learn and use the Cake pattern and you are able to use Dependency injection in your Android code. And the best: It's statically checked by the compiler and not at runtime like in Spring or Guice.

5. Get rid of if-cascades with Pattern matching

Checking conditions of an object in Java often leads to several if-cascades. And if you also want to extract some data out of the object you'll have to write a lot of code. Scala has it's swiss army knife for that: Pattern Matching. Here's an example:

With pattern matching you have the possibility to decompose an object into it's parts.
Case classes already provide automatic support for pattern matching, but you may use pattern matching even for java classes or classes for which you don't have the source code by using Scala's extractor's.

Scala on Android: Motivation, Building and useful libraries

Today I'll start a Scala on Android blog series.

This knowledge is mainly taken from the research and the work I've done for my current project. At the beginning there were big discussions if we should do plain old Java or use Scala for the app. Fortunately we decided together to go down the Scala road. See this Interview for more insight on the decision (not very technical and in german)

Here's the upcoming (it's Beta at the time of this writing) Play Store listing for the App.
We develop this app with Scala 2.11 for Android 4.0.3 and above.

Main Motivation

Scala is a beautiful language. I come from Java land, and it was a real eye opener to work with Scala and learn day by day more of the beauty of functional programming. So I definitely wanted to write as much Scala code and as less Java code as possible, even on Android.

But there's one really convincing feature in Scala that makes your life on Android so much easier and less painful: Futures instead of AsyncTask's Callback Handler approach.

With callback handler's you'll get soon in a spaghetti code situation. If you use Scala Future's you may easily combine them with their combinators, for-comprehensions or the new async/await macros. Just write some code with both concepts and you will see the difference.

For a more detailed  view on the motivation, see my follow up post.

I'll now start with the basic building blocks of the App.

Build system

One of the biggest questions when you start is: "Which build system should I use?" As I'm not a big fan of SBT I tried to look around and found one for my favourite build system gradle: gradle-android-scala-plugin. But unfortunately the author doesn't recommend it to use it in production yet.

So I tried out the Android SDK Plugin for SBT and it turned out to be very well. It is actively developed, the author is very responsive and helpful, and the Intellij integration via the SBT-Idea plugin is working well enough.

Scala and the Dexer limit

Compiled Java code for android is not shipped as .class files but in a format called dex. The problem is that currently one dex file allows only 65k method entries, and there are some big java android apps out there that already hit this limit. (e.g. the facebook app).

So even if you have a normal sized app, if you write it in Scala the scala library itself will reach this limit. But every Android developer knows ProGuard: A tool which inspects java bytecode and removes unused code and optionally obfuscates it. 

So if you use Scala for your app, using ProGuard is mandatory, even in the development stage, because it strips away a lot of the unused methods and fields from the Scala library.
These tools are very well integrated in the Android SDK Plugin for SBT, where you also have the possibility to customize a Proguard cache to save a lot of time during your development build cycles.

Libraries

 

REST client 

As a rest client we use the Spring Android REST Template from Spring Android. Not much to say here, it just works well. (In my Java days I was a big Spring fan.) We don't use the standard spring json parsing message converters. See next chapter.

Json Parsing

For my own (and first) Android app in Java I had used GSon for Json Parsing. It's very small and easy to use but unfortunately (and not surprisingly) it does not well support the Scala types we all love to use: Options, Lists etc.
So my next evaluation was to use Jackson together with the Jackson Scala Module. At Gutefrage.net we use it on the Scala server side, but it showed up that it is very cumbersome to use on Android. The main reason is that it heavily relies on reflection and if you look at how Proguard works, you'll get a feeling why this will cause trouble. You'll have to add several "-keep" rules to tell Proguard not to remove code that Jackson needs later on to build up your object graph from json, and a lot of more stuff. I would not recommend it.
So the final library in my journey was spray-json. It is a native Scala library and you define your json mapping as an external protocol (in Scala).

This has several advantages:

• No annotation and no reflection magic, all compile time, so no need to configure Proguard in any special way
  Type-class based (de)serialization of custom objects (no reflection, no intrusion)
• Annotations (e.g. used in Jackson) allow one representation, external protocol allows multiple
• Define a protocol for classes w/o source
• Simple and lightweight (Jar Size compared to Jackson)
• Plugging in of fast low level parsers simple

As it turned out the standard json parser of spray-json isn't the fastest, but it's very easy to plug in a really fast one. We use Jawn.

Libraries for making Android less painful

We use Scaloid, it has several nice features e.g.

• turning the usual android callback handlers (e.g. onClickListener) into a functional equivalent
• a Scala DSL for building and reusing layouts
• a lot of Rich* classes for the Android functionality and implicit parameters reducing a lot of boilerplate code 
• and a lot more

At some early point of the app we decided however not to use the Scala DSL as an alternative for the android layout xml's, but Scaloid allows you to make your choice.

Another UI DSL Library which might be worth to have a look at is Macroid.


The next upcoming post will talk about generalizing the error handling while talking to API servers.

Keep your clients simple with real REST and HATEOAS

The last months we were moving some backend services from an old legacy architecture to a new scala based service architecture, already having an iPhone app that uses the old services.

The legacy services exposed their API via HTTP and JSON in a format that has been copied from Facebook (and Google). For example look at the URL for getting questions with user information who created the questions:

/question?fields=question.title,user,user.nickname,....

The facebook idea is highly flexible, no doubt. Allowing you to tailor down the response the exact way you want it. You get exactly what you want and you're able to build clients that use the API to get new relations without the need to change the server side.

As we tried to re-write these services in Scala but staying backward compatible, we ran into the downsides of this approach:

Downside 1: Infinite number of URL query parameter variations

It was quite a hard job to find out which URL variations our current sole and own (!) client (iPhone App) uses from this API, because the above approach allowed so many possible variations of the URL. And sure, as we finally switched we broke the App because we've missed one call.

Downside 2: The client has to have explicit knowledge how to build URL's

The facebook approach builds an implicit contract between clients and the API for how to use the API. If the API URL format changes you'll now have to worry about supporting old clients using the old URL format.

Downside 3: Infinite number of response formats 

The flexible URL format as proposed by Facebook leads to responses being as dynamic as the URL parameters, so it was hard to find out which sets of responses are used by the client and have to be re-implemented to keep backward compatibility.

Alternative solution

We're currently writing a brand new Android App in Scala for sure, stay tuned for upcoming infos) and we don't want to go down the same error prone way as the months before. So for the new services (as a replacement for the old legacy one) we try to learn from the above problems.

Use fixed set of responses

We've introduced shapes for resources (like /users or /questions). These shapes are T-Shirt sizes: S, M, L, XL. The different shapes return you a different amount of associated data for your resource. For example M for a question would return a question with some inlined user information. We're thinking about allowing shapes for associated resources as well. E.g.

/questions?shape=M&user.shape=S

(To be honest, I'm not really a fan of inlining associated resources (even with the above shape way) because it contradicts the REST principles, but we want to reduce the amount of requests for the mobile clients here.)

With the introduction of shapes you have a reasonable amount of flexibility to customize the amount of data in your response and the query costs on the server side but you now have a stable set of response formats you have to support in the future.

This shape idea adresses downside 1 and 3 from the above list, but what about clients having to know how to build URL's?

Use resource links

Our current task was to implement paging for a specific resource stream. Up to now the old and also the new services used an approach also inspired by Facebook (and a lot more):

/questions?limit20&offset=100

We encountered problems when paging here, because a lot of new questions are created every second, so using limit and offset is no fixed but a moving window and the paged responses are eventually including already returned questions in another page. With this approach the client has to filter the paged responses for already returned responses. This is a really heavy burden for a client as you can imagine.

Also the client again has to know the logic how to build the paging URL as seen above. If you want to change that format in the future, you'll have big problems, because there are a lot of clients out there that use your old format.

So we decided to move on to an alternate solution:

We move the logic how to get the previous or next paged response from the client back to the server. The server as he returns paged responses now simply returns two resource links with the relation name 'previous' and 'next'. All the client has to do to get a new paged response before or after the current one is: Follow one of the links! This is just the same idea as a human uses the WWW. You don't worry about how the URL looks like on a page that you read, you just click the link and your browser follows...

The benefits of using resource links here are:

The server encodes the state for the paging inside the relation links, so this is what the HATEOAS (Hypermedia As The Enginge Of Application State) principle of REST is all about: The state for paging is now encoded in the returned links.

Now that the state is inside the links you have more advantages:

• If you decide to use timestamps for paging instead of Id's (as above), you simply change it on the server side. You don't brake your clients because your clients just follow the links
• The client doesn't anymore need to know how to build paging URL's, you can eliminate that logic from your client
• As you can see we also changed to now use beforeId or afterId params together with limit to eliminate duplicates in different pages (which happened as we use the offset param). But as we moved the pagination links generation to the server, the client doesn't have to have the logic for finding the last or first id of a page to get a new one

Validation in Scala Part 1: The return type candidates

Introduction

In every project you need validation of user input, scala projects are no exception. In Java some people use Bean Validation to annotate their objects and have them validated by a validator instance.
So some months ago we also used Java Bean Validation in our Finagle based Scala project.
Unfortunately this wasn't a good choice for our distributed service architecture. We're using several services to do validations and we had to wrap these calls in Futures and had to "await" them inside our custom BeanValidations which causes threads to be blocked. Not a very reactive way for today!

Also the BeanValidation API is somehow not so nice to use, e.g. it forces you to annotate classes and we needed more flexibility than we expected, so to make a long story short: We moved away from it.

Now I've tried to move to a more functional and flexible solution, inspired by excellent posts like http://blog.lunatech.com/2012/03/02/validation-scala

What to use?

Keep it simple, use simple validation methods

We're back using simple validation methods like

  def validateUserRegistration(request: UserRegistration)

So the interesting part here is the return type to be used for returning either a success or a failure(s).

Here's my current choice list

• Option
• \/ (Scalaz Either)
• Validation (also from Scalaz)
• ValidationNel (a specialized Validation)
  

We will now look into them in a little bit more detail.

Option

Option is the simplest solution, it may contain either a None (representing e.g. something like "NotFound) or a Some(x) for the Success case. You may be tempted to use a Some(error) and a None for expressing "No failure" but I always regard this a little bit of irritating. Later in the conversion section we'll see that I'm not alone with this.
The nice thing is that Option is a Monad, you may use it in a for comprehensions like this:
You'll get either a Some if every step contains a Some or a None if any of the Options used in the for comprehensions are a None.

 

\/ (Scalaz's Either implementation)

As I already explained in another post we make use of Scalaz Either (written as \/ ), instead of the plain Scala Either, because you can combine them with other Monads (like Twitter futures) using Monad Transformers.

Besides that an \/ instance is either a Left (per convention) the bad case or a Right (per convention) the good case.
You create them like this:
Little bit funny at first that with ".left" and ".right" you'll specify the other type (right type for a ".left", left type for a ".right"), but it makes sense because you have to help the compiler with the non-present type, doesn't it?

Validation

Validation is pretty much like EitherZ but Success and Failure is not a convention any more but is expressed in the subtypes: Failure and Success.
Here's an example creating them:

ValidationNel

"Nel" stands for NonEmptyList, as the scaladoc says: A singly-linked list that is guaranteed to be non-empty. Why is this important?
Imagine the following:

  def registerUser(name: String,...): Validation[List[RegisterUserFailures], User]

The returned “List” of failures may be possibly empty with this signature. This is irritating, because what does this mean? So Scalaz helps us with a simple but nice trick:

Conversions between the above types

The nice thing that you may choose whatever best fitting type of the above for you validation method. If you later on compose several validation methods which use different return types of the above you may easily convert between them and use the best fitting type in your aggregation method:

The "fold" from Option to \/ is one of several ways to "enrich" the missing Left information in an Option. If you go back from \/ to Option with "toOption" the Left information from a -\/ is simply dropped and "mapped" to a None. So like I said above: "None" of an Option is usually considered an error case (without detailed information). The toOption method of \/ has the same viewpoint here.

In Part 2 of this blog we'll have a look at fail-fast strategy for validation or aggregating several validation failures.

Using Functors or How to use implicit conversions inside containers

Since I work with Scala in my current project, using implicit conversions is quite nice to do the usual mapping stuff 'behind the scenes', for example to map between the external representation of a time value (in thrift) and the internal one using Joda DateTime

Using this just requires it to bring it in scope, e.g. with an import:

Some weeks ago a colleague asked me why the Scala compiler is so stupid not to apply a implicit conversion in the following case:

But how should the compiler now how to convert an Option[JodaDateTime] to Option[ThriftDateTime]? Yes, there is a map() method in Option, but this is just some kind of usual convention, and it would be really dangerous if the scala compiler would automatically use a method map() if present.

At this time I just suggested to explicitly map or use an implicit conversion on Option:
But this is really cumbersome and you have to do it also for lists or any other container type, even if they all have a common map method with the same semantic.

Functors

It turns out that there is a functional class which describes the above situation, they are called functors.
A functor is simply a trait which specifies to supply a map() method with adheres to the functor laws.

The above functor definition is directly taken from Scalaz 7.
Using it for an implicit "container" conversion of our two classes means:


With that the conversions work in all "containers" for which scalaz defines functors:

..and if you want you can write your own functors for your classes.

Using Futures together with Either or My way to scalaz

Intro

We're using Twitter's Finagle stack in my current scala project. Finagle is Future based and the Future concept makes usage of exceptions easy. Using Exceptions for "infrastructural" problems (like when the network or the database is down) is perfectly ok, in Java you would use RuntimeExceptions for this which don't have to show up in the method signature. In Scala there's no difference any more between checked exceptions and runtime exceptions. In Java checked exceptions are often used for "business" exceptions, i.e. exceptions that steer the way your business logic works. For example, some months ago in Java projects I've often written API methods like this:

So the possible return types of this method is either a ArticleDetails object containing the information about the article or an Exception saying that the article could not be found.

Pros

• Method signature tells you everything about the return types

Cons

• Two different ways of returning information
• return new ArticleDetails(...)
• throw new ArticleNotFoundException(...) 
• This also means two different styles of handling this information at the caller side

Scala

In Scala there are no checked exceptions any more, so migrating the above java interface to Scala and still use an exception for the failure case would result in the following:
The method signature doesn't tell you anymore that the method could produce a separate exception output. You could use scaladoc or @throws to document this, but the compiler doesn't check this and this probably leads (after some evolution cycles of your class) to a method signature that doesn't document anymore what it is returning in case of expected business flow failures.

Either to the rescue

In Scala there's a more functional way to say what the method is returning: Either

The above method either returns an ArticleNotFound or an ArticleDetails object.
An Either has a left side, which is used per convention for the failure case or a right side, which represents the "right", the success case.

Using Futures 

As I said we're using Twitter Futures for our code, so if we're doing some blocking method call, we do it async and return a Future[RETURN_OBJECT] instead of the simple method return object to keep the current thread from being blocked. So the above method blocks because of a call to MySQL and would therefore look like this:

For Comprehensions

So far everything looks good, and it seems pretty easy to get away from business exceptions. But imagine now a second method which returns an articleId for an unique article name:

If someone has only the unique name and wants to retrieve the article details I would naively use the following for comprehension and would expect Left's (Failures) to fail fast and "bubble" out of the for comprehension:

Unfortunately the above doesn't compile. What we need is a way to compose the Future with Either. At this point I was very lost and asked for help in this stackoverflow question. Fortunately the help came at the Scala User Group Munich where Lars Hupel helped me a lot with this.

Monad Transformers or "My way to scalaz"

The way out of this problem is a Monad transformer. A simple mental model for  a Monad in Scala is a class which implements map, flatMap and withFilter and can therefore be used in a for comprehension.
Either and Futures are Monads and Monads don't compose out of the box together. There's even not a guarentee that mondas can be combined.

My first problem was now that scala.util.Either is not even a Monad. Only it's right or left protection is a Monad! Which makes it a little bit cumbersome to use. You'll have to say always .right in for comprehensions.

As I found out that scalaz has also an Either called \/ which is right biased and not neutral like scala.util.Either. The best: It has a transformer class EitherT to combine it with other Monads.

Converting the above example to \/ leads us to:

Pretty similar to the solution with Scala's Either, but without .right's, because of it's right biasing like in Haskell. To use EitherT with Twitter Futures we now only need to define the following implicit vals:

Then we can use EitherT together with Twitter Futures in our for comprehension:
The run method turns the resulting EitherT back to a Future[\/]. A EitherT in this case is an EitherT[Future, ArticleNotFound.type, ArticleDetails], so this could be used as alternate return type. Then you can omit the run call.

Here's the final complete example:

If a left happens on one of the calls in the for comprehension, it bubbles out in fail fast strategy. If there are only Right's than the article is yielded.

Slick in Scala: Combining multiple fields in one mapping case class field

Yesterday I've had a problem with an existing database schema. In one table there were several Boolean columns each expressing the existence of an user role.

Since we use Slick for MySQL access in our services, I wanted to combine these several columns in one single (enumeration) field of the mapping case class:
This can be achieved by providing your own constructor and extractor functions for User objects to Slick using the <> function on the * projection. Something like this:
Be sure that your own constructor and extractor functions have the apply/unapply style, e.g. extractUser should return an Option of a tuple, if it does not (extractUser returns a tuple instead of an option of a tuple) you get a hard-to-read error message from the compiler:

overloaded method value <> with alternatives:
  [R(in method <>)...
    userModeratorMajor ~ partner ~ premiumPartner ~ corporatePaid <> (constructUser _, extractUser _)

Thanks Mr. Vogt from Typesafe for the hint.