I’ve been exploring functional programming with Scala and its ecosystem for the past few months.
In this post, I’ll highlight some of the features of the language that enable intuitive creation of functional code for distributed systems and data operations.
Higher Order Functions
As per the official documentation, Functions are first class objects in Scala, which means that they can:
- Take another function as an argument, or …
- Return a function
An example of a function taking another function as an argument is the map() function in Scala's standard collections library.
val examplelist: List[Int] = List(2,9,8,14)
examplelist.map(x=> x * 2) // anonymous function as argument
When working with standard Scala collections, it’s also very intuitive to chain operators, especially with the infix notation. In the small code example below, I’m defining a list of numbers from 1 to 20, filtering on even numbers and then summing them up.
(1 to 20).toList filter (_%2 == 0) reduce (_ + _)
The _ is the wildcard operator - in the case of maps and filters, it refers to the value in the collection.
Recursion
The recommended way to do operations on all the items in a collection is to use the operators map, flatMap, or reduce.
In case those operators don’t meet a use case’s requirements, it’s very useful to write a tail-recursive function to operate on all the items in a collection.
The code example below shows a tail-recursive function definition to compute the factorial of a number.
import scala.annotation.tailrec
@tailrec
// Factorial Function Implementation that uses Tail Recursion
def factorial(in_x: Double, prodsofar: Double = 1.0): Double = {
if (in_x==0) prodsofar
else factorial(in_x-1, prodsofar*in_x)
}
factorial(5)
In Scala, a tail-recursive function as shown above can be optimised by the compiler (using the @tailrec annotation above) to occupy just 1 stack frame, so there's no chance of a stackoverflow error even for many levels of recursion. This is possible out-of-the-box, without any need for frameworks or plugins.
As mentioned above, the recommended way is to use to collections operators (such as reduce, etc.). As a demo of the ease of use of the collection's APIs, the above factorial function can also be implemented by the 1-liner below:
(1 to 5).toList reduce (_*_)
To conceptually understand reduce, check out this great link!
(Also do check out the explanations of foldLeft, foldRight, map, flatMap to understand some commonly used data operations!)
Case Classes
Case classes can be instantiated very easily with no boiler plate code, such as the example below.
case class BusinessTransaction(
sourceaccountid: Long,
targetaccountid: Long,
amount: Long
)
// create some transactions now to demo case classes
// I lend my friend
val 1_xaction = BusinessTransaction(112333L, 998882L, 20L)
// My friend pays me back
val 2_xaction = BusinessTransaction(998882L, 112333L, 20L)
Just 1 case class .. line above does the following useful things:
- Defines the 3 immutable values sourceaccountid, targetaccountid and amount
- Defines get methods to access the constructor arguments (eg: 1_xaction.amount)
While the ease of use is great, case classes are the recommended way to store immutable data instances in Scala. For example, in a big data application, each line of a large datafile can be modelled by a case class and stored.
An example of the use of a case class to store data is here.
In the linked example, the function rawPostings models each line of the datafile as an instance of case class Posting. It will eventually return a dataset of type RDD[Posting].
Pattern Matching
In Scala, objects such as case classes, regular classes, and collections can be decomposed through pattern matching.
Essentially, this means that you can use pattern matching to:
- Decompose an object’s type (example below)
- Get the head of a collection (such as a List or a Seq)
The code example below shows how to use pattern matching to decompose a Seq.
val seq1: Seq[Int] = Seq(1,3,4,5,5)
seq1 match {
case x::y => println(s"The first element in the sequence is ${x}")
case Nil => println("The sequence is empty")
}
The cons operator (::) creates a list made of the head ( x) and the rest of the list (called the tail, y).
Companion Objects
In OOP, a static variable is sometimes used in a class to store state or property across multiple instantiated objects.
However, there is no static keyword in Scala. Instead, what we use are Companion Objects aka Singleton Objects. A Companion Object is defined using the object keyword and has the exact same name as its accompanying class.
The companion objects can define immutable values, which can then be referenced by methods in the class.
There are 2 common patterns to use companion objects in Scala:
- As a factory method
- To provide functionality that is common to the class (i.e. static function in Java)
// The 'val specie' straightaway defines an immmutable class parameter
abstract class Animal(val specie: String)
{
import Animal._
// Common Behaviour to be mixed-in to Canine/Feline classes
def getConnectionParameters: String = Animal.connectionParameter
}
object Animal
{
// .apply() is the factory method
def apply(specie: String): Animal = specie match {
case "dog" => new Canine(specie)
case "cat" => new Feline(specie)
}
val connectionParameter:String = System.getProperty("user.dir")
}
class Canine(override val specie: String) extends Animal(specie)
{
override def toString: String = s"Canine of specie ${specie}"
}
class Feline(override val specie: String) extends Animal(specie)
{
override def toString: String = s"Feline of specie ${specie}"
}
// syntactic sugar, where we don't have to say new Animal
val doggy = Animal("dog")
val kitty = Animal("cat")
doggy.getConnectionParameters
Options
Most application code checks for Null/None types. Null types are handled a little differently in Scala — the construct used is called an Option. This is best demonstrated with an example.
val customermap: Map[Int, String] = Map(
11-> "CustomerA", 22->"CustomerB", 33->"CustomerC"
)
customermap.get(11) // Map's get() returns an Option[String] customermap.get(11).get // Option's get returns the String customermap.get(999).get // Will throw a NoSuchElementException customermap.get(999).getOrElse(0) // Will return a 0 instead of throwing an exception
In a language like Python, if None: checks would be quite common throughout the codebase. In Java, there would be try-catch blocks that would handle thrown exceptions. Option s allow for focusing on the logic flow with minimal diversions for type or exception checks.
A standard way of using Options in Scala is for your custom functions to return Option[String] (or Int, Long, etc.). Let's look at the Map structure's get() function signature:
def get(key: A): Option[B]
One (intuitive) way to use this is to chain it with the getOrElse() function as shown below:
// Map of IDs vs Names
val customermap: Map[Int, String] = Map(
11-> "CustomerA", 22->"CustomerB", 33->"CustomerC"
)
customermap.get(11).getOrElse("No customer found for the provided ID")
A very useful way of using Options is together with a collection operator like flatMap that directly handles the types for you transparently.
// Map of IDs vs Names
val customermap: Map[Int, String] = Map(
11-> "CustomerA", 22->"CustomerB", 33->"CustomerC"
)
val listofids: List[Int] = List(11,22,33,99)
listofids flatMap (id=> customermap.get(id)) //flatMap magic
And that’s it from me! My next excursion is to explore concurrent systems with Akka and the Actor model. Look out for a future post, where I’ll share my learnings on that topic (and it’s relationship to Scala’s approach to functional programming).
Originally published at http://github.com.
