Scala

CASSANDRA + SPARk 1 of 2

A while ago I wrote about using Apache Spark, which is a great tool. I have been using Cassandra for a bit at work now, so thought it might be nice to revisit that artilcle and talk through how to use Spark with Cassandra.

Here is the 1st part of that ; http://www.codeproject.com/Articles/1073158/Apache-Spark-Cassandra-of

Advertisements
Scala

Scala : multi project sbt setup

A while ago I wrote a post about how to use SBT (Scala Build Tool):

https://sachabarbs.wordpress.com/2015/10/13/sbt-wheres-my-nuget/

In that post I showed simple usages of SBT. Thing is that was not really that realistic, so I wanted to have a go at a more real world example of this. One where we might have multiple projects, say like this:

 

SachasSBTDemo-App which depends on 2 sub projects

  • SachasSBTDemo-Server
  • SachasSBTDemo-Common

So how do we go about doing this with SBT?

There are 5 main steps to do this. Which we look at in turn.

SBT Directory Structure

The first that we need to do is create a new project folder (if you are from Visual Studio / .NET background think of this as the solution folder) called “project”

In here we will create 2 files

build.properties which just lists the version of SBT we will use. It looks like this

sbt.version=0.13.8

SachaSBTDemo.scala is what I have called the other file, but you can call it what you like. Here is the contents of that file, this is the main SBT file that governs how it all hangs together. I will be explaining each of these parts as we go.

  import sbt._
  import Keys._

object BuildSettings {


  val buildOrganization = "sas"
  val buildVersion      = "1.0"
  val buildScalaVersion = "2.11.5"

  val buildSettings = Defaults.defaultSettings ++ Seq (
    organization := buildOrganization,
    version      := buildVersion,
    scalaVersion := buildScalaVersion
  )
}


object Dependencies {
  val jacksonjson = "org.codehaus.jackson" % "jackson-core-lgpl" % "1.7.2"
  val scalatest = "org.scalatest" % "scalatest_2.9.0" % "1.4.1" % "test"
}


object SachasSBTDemo extends Build {

  import Dependencies._
  import BuildSettings._

  // Sub-project specific dependencies
  val commonDeps = Seq (
     jacksonjson,
     scalatest
  )

  val serverDeps = Seq (
     scalatest
  )


  lazy val demoApp = Project (
    "SachasSBTDemo-App",
    file ("SachasSBTDemo-App"),
    settings = buildSettings
  )
  //build these projects when main App project gets built
  .aggregate(common, server)
  .dependsOn(common, server)

  lazy val common = Project (
    "common",
    file ("SachasSBTDemo-Common"),
    settings = buildSettings ++ Seq (libraryDependencies ++= commonDeps)
  )

  lazy val server = Project (
    "server",
    file ("SachasSBTDemo-Server"),
    settings = buildSettings ++ Seq (libraryDependencies ++= serverDeps)
  ) dependsOn (common)
  
}

 

Projects

In order to have separate project we need to use the Project item from the SBT library JARs. A minimal Project setup will tell SBT where to create the new Project. Here is an example of a Project, where the folder we expect SBT to create will be called “SachasSBTDemo-App”.

lazy val demoApp = Project (
    "SachasSBTDemo-App",
    file ("SachasSBTDemo-App"),
    settings = buildSettings
  )

Project Dependencies

We can also specify Project dependencies using “dependsOn” which takes a Seq of other projects that this Project depends on.

That means that when we apply an action to the Project that is depended on, the Project that has the dependency will also have the action applied.

lazy val demoApp = Project (
    "SachasSBTDemo-App",
    file ("SachasSBTDemo-App"),
    settings = buildSettings
  )
  //build these projects when main App project gets built
  .aggregate(common, server)
  .dependsOn(common, server)

Project Aggregation

We can also specify Project aggregates results from other projects, using “aggregate” which takes a Seq of other projects that this Project aggregates.

What “aggregate” means is that whenever we apply an action on the aggregating Project we should also see the same action applied to the aggregated Projects.

lazy val demoApp = Project (
    "SachasSBTDemo-App",
    file ("SachasSBTDemo-App"),
    settings = buildSettings
  )
  //build these projects when main App project gets built
  .aggregate(common, server)
  .dependsOn(common, server)

Library Dependencies

Just like the simple post I did before, we still need to bring in our JAR files using SBT. But this time we come up with a nicer way to manage them. We simply wrap them all up in a simple object, and then use the object to satisfy the various dependencies of the Projects. Much neater.

import sbt._
import Keys._


object Dependencies {
  val jacksonjson = "org.codehaus.jackson" % "jackson-core-lgpl" % "1.7.2"
  val scalatest = "org.scalatest" % "scalatest_2.9.0" % "1.4.1" % "test"
}

  // Sub-project specific dependencies
  val serverDeps = Seq (
     scalatest
  )

  .....
  .....
  lazy val server = Project (
    "server",
    file ("SachasSBTDemo-Server"),
    //bring in the library dependencies
    settings = buildSettings ++ Seq (libraryDependencies ++= serverDeps)
  ) dependsOn (common)

The Finished Product

The final product once run through SBT should be something like this if viewed in IntelliJ IDEA:

image

 

Or like on the file system

image

If you want to grab my source files, they are available here at GitHub : https://github.com/sachabarber/SBT_MultiProject_Demo

MVP

MVP For 2016

Well I just got the email from the big house. I am an MVP for 2016 for “Visual Studio and Development Technologies”. This will be the 9th time I have been awarded the MVP award. Neato

Interesting thing is I spent 1/2 of last year working with the JVM / open source (few Apache projects) which I started to blog about. Whilst I spent the other 1/2 on .NET, so I honestly did not think I would be receiving the MVP award this time around, so it was a nice surprise. Awards are always nice.

That said I have never tried to “GET” the MVP award. It is nice when you get recognized for your efforts, but right now I am really enjoying the open source stuff, and I will be continuing to work with that for sure. I will ALWAYS have time for .NET, I love it. My current role has me spending 50% of my time in .NET land, and the other 50% in Scala and open source, so I am a happy camper right now.

I will continue to blog about stuff that I personally find interesting, and if that is .NET / Scala / open source stuff, so be it. Hopefully that will cover .NET and the other stuff of interest that I am digging lately. Only time will tell.

 

Anyway thanks Microsoft, and thanks to all the readers of my blog. Happy new year to you all