MADCAP IDEA 11 : FINISHING THE ‘VIEW RATING’ PAGE

Last Time

Last time we looked at creating an reactive Kafka publisher that would push out a Rating from a REST call (which will eventually come from the completion of a job).  The last post also used some akka streams, and how we can use a back off supervisor. In a nutshell this post will build on the Rating stream/Kafka KTable storage we have set up to date. Where we will actually create the React/Play framework endpoint to wire up the “View Rating” page with the Rating data that has been pushed through the stream processing so far.

 

PreAmble

Just as a reminder this is part of my ongoing set of posts which I talk about here :

https://sachabarbs.wordpress.com/2017/05/01/madcap-idea/, where we will be building up to a point where we have a full app using lots of different stuff, such as these

 

  • WebPack
  • React.js
  • React Router
  • TypeScript
  • Babel.js
  • Akka
  • Scala
  • Play (Scala Http Stack)
  • MySql
  • SBT
  • Kafka
  • Kafka Streams

Ok so now that we have the introductions out of the way, lets crack on with what we want to cover in this post.

 

Where is the code?

As usual the code is on GitHub here : https://github.com/sachabarber/MadCapIdea

 

What Is This Post All About?

As stated above this post largely boils down to the following things

  • Create an endpoint (Façade over existing Kafka stream Akka Http endpoint) to expose the previously submitted Rating data
  • Make the “View Rating” page work with the retrieved data

 

Play Back End Changes

This section shows the changes to the play backend API code

 

Rating Endpoint Façade

As we stated 2 posts ago we created an Akka Http REST endpoint to serve up the combined Rating(s) that have been pushed through the Kafka stream processing rating topic. However we have this Play framework API which we use for all other REST endpoints. So I have chose to create a façade endpoint in the Play backend that will simply call out to the existing Akka Http endpoint. Keeping all the traffic in one place is a nice thing if you ask me. So lets look at this play code to do this

 

New Route

We obviously need a new route, which is as follows:

 

GET  /rating/byemail                          controllers.RatingController.ratingByEmail()

 

Controller Action

To serve this new route we need a new Action in the RatingController. This is shown below:

 

package controllers

import javax.inject.Inject

import Actors.Rating.RatingProducerActor
import Entities.RatingJsonFormatters._
import Entities._
import akka.actor.{ActorSystem, OneForOneStrategy, Props, SupervisorStrategy}
import akka.pattern.{Backoff, BackoffSupervisor}
import akka.stream.{ActorMaterializer, ActorMaterializerSettings, Supervision}
import play.api.libs.json._
import play.api.libs.json.Json
import play.api.libs.json.Format
import play.api.libs.json.JsSuccess
import play.api.libs.json.Writes
import play.api.libs.ws._
import play.api.mvc.{Action, Controller}
import utils.{Errors, Settings}

import scala.concurrent.{ExecutionContext, Future}
import scala.util.Random
import scala.concurrent.duration._

class RatingController @Inject()
(
  implicit actorSystem: ActorSystem,
  ec: ExecutionContext,
  ws: WSClient
) extends Controller
{

  def ratingByEmail = Action.async { request =>

    val email = request.getQueryString("email")
    email match {
      case Some(emailAddress) => {
        val url = s"http://${Settings.ratingRestApiHostName}:${Settings.ratingRestApiPort}/ratingByEmail?email=${emailAddress}"
        ws.url(url).get().map {
          response => (response.json).validate[List[Rating]]
        }.map(x => Ok(Json.toJson(x.get)))
      }
      case None => {
        Future.successful(BadRequest(
          "ratingByEmail endpoint MUST be supplied with a non empty 'email' query string value"))
      }
    }
  }
}

 

The main thing to note here is:

  • We use the play ws (web services) library to issues a GET request against the existing Akka Http endpoint. Thus creating our façade.
  • We are still using Future to make it nice an async

 

 

React Front End Changes

This section shows the changes to the React frontend code

 

React “View Rating” page

This is the final results for the “View Rating” react page. I think its all fairly self explanatory. I guess the only bit that really of any note is that we use lodash _.sumBy(..) to do the summing up of the Ratings for this user to create an overall rating.The rest is standard jQuery/react stuff.

 

import * as React from "react";
import * as ReactDOM from "react-dom";
import * as _ from "lodash";
import { OkDialog } from "./components/OkDialog";
import 'bootstrap/dist/css/bootstrap.css';
import
{
    Well,
    Grid,
    Row,
    Col,
    Label,
    ButtonInput
} from "react-bootstrap";

import { AuthService } from "./services/AuthService";

import { hashHistory  } from 'react-router';



class Rating {
    fromEmail: string
    toEmail: string
    score: number

    constructor(fromEmail, toEmail, score) {
        this.fromEmail = fromEmail;
        this.toEmail = toEmail;
        this.score = score;
    }
}


export interface ViewRatingState {
    ratings: Array<Rating>;
    overallRating: number;
    okDialogOpen: boolean;
    okDialogKey: number;
    okDialogHeaderText: string;
    okDialogBodyText: string;
    wasSuccessful: boolean;
}


export class ViewRating extends React.Component<undefined, ViewRatingState> {

    private _authService: AuthService;

    constructor(props: any) {
        super(props);
        this._authService = props.route.authService;
        if (!this._authService.isAuthenticated()) {
            hashHistory.push('/');
        }
        this.state = {
            overallRating: 0,
            ratings: Array(),
            okDialogHeaderText: '',
            okDialogBodyText: '',
            okDialogOpen: false,
            okDialogKey: 0,
            wasSuccessful: false
        };
    }   


    loadRatingsFromServer = () => {

        var self = this;
        var currentUserEmail = this._authService.userEmail();

        $.ajax({
            type: 'GET',
            url: 'rating/byemail?email=' + currentUserEmail,
            contentType: "application/json; charset=utf-8",
            dataType: 'json'
        })
        .done(function (jdata, textStatus, jqXHR) {

            console.log("result of GET rating/byemail");
            console.log(jqXHR.responseText);
            let ratingsObtained = JSON.parse(jqXHR.responseText);
            self.setState(
                {
                    overallRating: _.sumBy(ratingsObtained, 'score'),
                    ratings: ratingsObtained
                });
        })
        .fail(function (jqXHR, textStatus, errorThrown) {
            self.setState(
                {
                    okDialogHeaderText: 'Error',
                    okDialogBodyText: 'Could not load Ratings',
                    okDialogOpen: true,
                    okDialogKey: Math.random()
                });
        });
        
    }

    componentDidMount() {
        this.loadRatingsFromServer();
    }

    render() {

        var rowComponents = this.generateRows();

        return (
            <Well className="outer-well">
                    <Grid>
                        <Row className="show-grid">
                            <Col xs={6} md={6}>
                                <div>
                                <h4>YOUR OVERALL RATING <Label>{this.state.overallRating}</Label></h4>
                                </div>
                            </Col>
                        </Row>
                        <Row className="show-grid">
                            <Col xs={10} md={6}>
                                <h6>The finer details of your ratings are shown below</h6>
                            </Col>
                        </Row>
                        <Row className="show-grid">
                            <Col xs={10} md={6}>
                                <div className="table-responsive">
                                    <table className="table table-striped table-bordered table-condensed factTable">
                                        <thead>
                                            <tr>
                                                <th>Rated By</th>
                                                <th>Rating Given</th>
                                            </tr>
                                        </thead>
                                        <tbody>
                                            {rowComponents} 
                                        </tbody>
                                    </table>
                                </div>
                            </Col>
                        </Row>
                        <Row className="show-grid">
                            <span>
                                <OkDialog
                                    open= {this.state.okDialogOpen}
                                    okCallBack= {this._okDialogCallBack}
                                    headerText={this.state.okDialogHeaderText}
                                    bodyText={this.state.okDialogBodyText}
                                    key={this.state.okDialogKey}/>
                            </span>
                        </Row>
                    </Grid>
            </Well>
        )
    }

    _okDialogCallBack = () => {
        this.setState(
            {
                okDialogOpen: false
            });
    }

    generateRows = () => {
        return this.state.ratings.map(function (item) {
            return  <tr key={item.fromEmail}>
                        <td>{item.fromEmail}</td>
                        <td>{item.score}</td>
                    </tr>;

        });
    } 
}

 

And with that the “View Rating” page is actually done. This was a short post for a change, which is nice.

 

 

Conclusion

The previous set of posts, have made this one very easy to do. Its just standard React/REST/Play stuff. So this one has been fairly easy to do.

 

Next Time

The more interesting stuff is still to come where we push out new jobs onto a new Kafka stream topic.

Advertisements

MADCAP IDEA 10 : PLAY FRAMEWORK REACTIVE KAFKA PRODUCER

Last Time

So last time we walk through the Rating Kafka streams architecture and also showed how we can query the local state stores. I also stated that the standard KafkaProducer that was used in the last post was more for demonstration purposes and long term, we would like to swap that out with a Play framework REST endpoint that allowed us to publish a message straight from our app to the Kafka rating stream processing

 

PreAmble

Just as a reminder this is part of my ongoing set of posts which I talk about here :

https://sachabarbs.wordpress.com/2017/05/01/madcap-idea/, where we will be building up to a point where we have a full app using lots of different stuff, such as these

 

  • WebPack
  • React.js
  • React Router
  • TypeScript
  • Babel.js
  • Akka
  • Scala
  • Play (Scala Http Stack)
  • MySql
  • SBT
  • Kafka
  • Kafka Streams

Ok so now that we have the introductions out of the way, lets crack on with what we want to cover in this post.

 

Where is the code?

As usual the code is on GitHub here : https://github.com/sachabarber/MadCapIdea

 

What Is This Post All About?

As stated in the last post “Kafka interactive queries” we used a standard KafkaProducer to simulate what would have come from the end user via the Play Framework API. This time we will build out the Play Framework side of things, to include the ability to produce “rating” objects that are consumed via the rating Kafka Stream processing topology introduced in the last post

 

SBT

So we already had an SBT file inside of the PlayBackEndApi project, but we need to expand that to include support for a couple of things

  • Reactive Kafka
  • Jackson JSON (play already comes with its own JSON support, but for the Kafka Serialization-DeSerialization (Serdes) I wanted to make sure it was the same as the Kafka Streams project

This means these additions to the built.sbt file

 

val configVersion = "1.0.1"

libraryDependencies ++= Seq(
  "org.reactivemongo" %% "play2-reactivemongo" % "0.11.12",
  "com.typesafe.akka" % "akka-stream-kafka_2.11" % "0.17",
  "org.skinny-framework.com.fasterxml.jackson.module" % "jackson-module-scala_2.11" % "2.8.4",
  "com.typesafe"        % "config" % configVersion
)

 

Topics

We also want to ensure that we are using the same topics as the stream processing topology so I have just replicated that class (in reality I should have made this stuff a common JAR, but meh)

 

package kafka.topics

object RatingsTopics {
    val RATING_SUBMIT_TOPIC = "rating-submit-topic"
    val RATING_OUTPUT_TOPIC = "rating-output-topic"
  }

 

Routing

As this is essentially a new route that would be called via the front end React app when a new Rating is given, we obviously need a new route/controller. The route is fairly simple which is as follows:

 

# Rating page
POST  /rating/submit/new                       controllers.RatingController.submitNewRating()

 

JSON

The new Rating route expects a Rating object to be provided as a POST in JSON. Here is the actual Rating object and play JSON handling for it

 

package Entities

import play.api.libs.json._
import play.api.libs.functional.syntax._

case class Rating(fromEmail: String, toEmail: String, score: Float)

object Rating {
  implicit val formatter = Json.format[Rating]
}

object RatingJsonFormatters {

  implicit val ratingWrites = new Writes[Rating] {
    def writes(rating: Rating) = Json.obj(
      "fromEmail" -> rating.fromEmail,
      "toEmail" -> rating.toEmail,
      "score" -> rating.score
    )
  }

  implicit val ratingReads: Reads[Rating] = (
      (JsPath \ "fromEmail").read[String] and
      (JsPath \ "toEmail").read[String] and
      ((JsPath \ "score").read[Float])
    )(Rating.apply _)
}

 

Controller

So now that we have a route lets turn our attention to the new RatingController. Which right now to just accept a new Rating just looks like this:

package controllers

import javax.inject.Inject

import Actors.Rating.RatingProducerActor
import Entities.RatingJsonFormatters._
import Entities._
import akka.actor.{ActorSystem, OneForOneStrategy, Props, SupervisorStrategy}
import akka.pattern.{Backoff, BackoffSupervisor}
import akka.stream.{ActorMaterializer, ActorMaterializerSettings, Supervision}
import play.api.libs.json._
import play.api.mvc.{Action, Controller}
import utils.Errors

import scala.concurrent.{ExecutionContext, Future}
import scala.util.Random
import scala.concurrent.duration._

class RatingController @Inject()
(
  implicit actorSystem: ActorSystem,
  ec: ExecutionContext
) extends Controller
{

  //Error handling for streams
  //http://doc.akka.io/docs/akka/2.5.2/scala/stream/stream-error.html
  val decider: Supervision.Decider = {
    case _                      => Supervision.Restart
  }

  implicit val mat = ActorMaterializer(
    ActorMaterializerSettings(actorSystem).withSupervisionStrategy(decider))
  val childRatingActorProps = Props(classOf[RatingProducerActor],mat,ec)
  val rand = new Random()
  val ratingSupervisorProps = BackoffSupervisor.props(
    Backoff.onStop(
      childRatingActorProps,
      childName = s"RatingProducerActor_${rand.nextInt()}",
      minBackoff = 3.seconds,
      maxBackoff = 30.seconds,
      randomFactor = 0.2
    ).withSupervisorStrategy(
      OneForOneStrategy() {
        case _ => SupervisorStrategy.Restart
      })
    )

  val ratingSupervisorActorRef = 
    actorSystem.actorOf(
      ratingSupervisorProps, 
      name = "ratingSupervisor"
    )

  def submitNewRating = Action.async(parse.json) { request =>
    Json.fromJson[Rating](request.body) match {
      case JsSuccess(newRating, _) => {
        ratingSupervisorActorRef ! newRating
        Future.successful(Ok(Json.toJson(
          newRating.copy(toEmail = newRating.toEmail.toUpperCase))))
      }
      case JsError(errors) =>
        Future.successful(BadRequest(
          "Could not build a Rating from the json provided. " +
          Errors.show(errors)))
    }
  }
}

 

The main points from the code above are

  • We use the standard Play framework JSON handling for the unmarshalling/marshalling to JSON
  • That controller route is async (see how it returns a Future[T]
  • That we will not process anything if the JSON is invalid
  • That the RatingController creates a supervisor actor that will supervise the creation of another actor namely a RatingProducerActor. It may look like this happens each time the RatingController is instantiated, which is true. However this only happens once as there is only one router in play, and the controller are created by the router. You can read more about this here : https://github.com/playframework/playframework/issues/4508#issuecomment-127820190. The short story is that the supervisor is created once, and the actor is supervises will be created using a BackOffSupervisor where the creation of the actor will be retried using an incremental back off strategy. We also use the OneForOneStrategy to ensure only the single failed child actor is effected by the supervisor.
  • That this controller is also responsible for creating a ActorMaterializer with a supervision strategy (more on this in the next section). The ActorMaterializer  is used to create actors within Akka Streams workflows.

 

 

RatingProducerActor

The final part of the pipeline for this post is obviously to be able to write a Rating to a Kafka topic, via a Kafka producer. As already stated I chose to use reactive a Reactive Kafka (akka streams Kafka producer which build upon Akka streams ideas, where we have Sinks/Sources/Flow/RunnableGraph all the good stuff. So here is the full code for the actor:

package Actors.Rating

import Entities.Rating
import Serialization.JSONSerde
import akka.Done
import akka.actor.{Actor, PoisonPill}
import akka.kafka.ProducerSettings
import akka.kafka.scaladsl.Producer
import akka.stream.{ActorMaterializer, KillSwitches}
import akka.stream.scaladsl.{Keep, MergeHub, Source}
import kafka.topics.RatingsTopics
import org.apache.kafka.clients.producer.ProducerRecord
import org.apache.kafka.common.serialization.{ByteArraySerializer, StringSerializer}
import utils.Settings

import scala.concurrent.ExecutionContext
import scala.util.{Failure, Success}


class RatingProducerActor(
    implicit materializer: ActorMaterializer,
    ec: ExecutionContext
) extends Actor {

  val jSONSerde = new JSONSerde[Rating]
  val ratingProducerSettings = ProducerSettings(
      context.system,
      new StringSerializer,
      new ByteArraySerializer)
    .withBootstrapServers(Settings.bootStrapServers)
    .withProperty(Settings.ACKS_CONFIG, "all")

  val ((mergeHubSink, killswitch), kafkaSourceFuture) =
    MergeHub.source[Rating](perProducerBufferSize = 16)
      .map(rating => {
        val ratingBytes = jSONSerde.serializer().serialize("", rating)
        (rating, ratingBytes)
      })
      .map { ratingWithBytes =>
        val (rating, ratingBytes) = ratingWithBytes
        new ProducerRecord[String, Array[Byte]](
          RatingsTopics.RATING_SUBMIT_TOPIC, rating.toEmail, ratingBytes)
      }
      .viaMat(KillSwitches.single)(Keep.both)
      .toMat(Producer.plainSink(ratingProducerSettings))(Keep.both)
      .run()

  kafkaSourceFuture.onComplete {
    case Success(value) => println(s"Got the callback, value = $value")
    case Failure(e) => self ! PoisonPill
  }

  override def postStop(): Unit = {
    println(s"RatingProducerActor seen 'Done'")
    killswitch.shutdown()    
    super.postStop()
  }

  override def receive: Receive = {
    case (rating: Rating) => {
      println(s"RatingProducerActor seen ${rating}")
      Source.single(rating).runWith(mergeHubSink)
    }
    case Done => {
      println(s"RatingProducerActor seen 'Done'")
      killswitch.shutdown()
      self ! PoisonPill
    }
  }
}

 

I’ll be honest there is a fair bit going on in that small chunk of code above so lets dissect it. What it happening exactly?

  • The most important point is that we simply use the actor as a vessel to host a reactive kafka akka stream RunnableGraph representing a Graph of MergeHub – > Reactive Kafka producer sink. This is completely fine and a normal thing to do. Discussing akka streams is out of scope for this post but if you want to know more you can read more on a previous post I did here : https://sachabarbs.wordpress.com/2016/12/13/akka-streams/ 
  • So we now know this actor hosts a stream, but the stream could fail, or the actor could fail. So what we want is if the actor fails the stream is stopped, and if the stream fails the actor is stopped. To do that we need to do a couple of thing
    • STREAM FAILING : Since the RunnableGraph can return a Future[T] we can hook a callback Success/Failure on that, and send a PoisonPill to the hosting actor. Then the supervisor actor we saw above would kick in and try and create a new instance of this actor. Another thing to note is that the stream hosted in this actor uses the ActorMaterializer that was supplied by the RatingController, where we provided a restart supervision strategy for the stream.
    • ACTOR FAILING : If the actor itself fails the Akka framework will call the postStop() method, at which point we want to shutdown the stream within this actor. So how can we shutdown the hosted stream? Well see in the middle of the stream setup there is this line .viaMat(KillSwitches.single)(Keep.both) this allows us to get a killswitch from the materialized values for the stream. Once we have a KillSwitch we can simply call its shutDown() method.
    • BELTS AND BRACES : I have also provided a way for the outside world to shutdown this actor and its hosted stream. This is via sending this actor a Done message. I have not put this in yet, but the hook is there to demonstrate how you could do this.
  • We can see that there is a MergeHub source which allows external code to push stuff through the MergeHub via the materialized Sink value from within the actor
  • We can also see that the Rating object that the actor sees is indeed pushed into the MergeHub materialized Sink via this actor, and then some transformation is done on it, to grab its raw bytes
  • We can see the final stage in the RunnableGraph is the  Reactive Kafka Producer.plainSink. Which would result in a message being pushed out to a Kafka topic from the hosted stream, pushed Rating object from this actor into the stream

And I think that is the main set of points about how this actor works

 

 

The End Result

Just to prove that this is all working here is a screen shot of the new RatingController http://localhost:9000/rating/submit/new endpoint being called with a JSON payload representing the Rating

 

image

 

And here is the Akka Http endpoint that queries the Kafka Stream state store(s)

 

http://localhost:8080/ratingByEmail?email=sacha@here.com this gives an empty list as we have NOT sent any Rating through for email “sacha@here.com” yet

 

image

 

http://localhost:8080/ratingByEmail?email=henry@there.com this gives 1 result which is consistent with the amount of Rating(s) I created

 

image

 

http://localhost:8080/ratingByEmail?email=henry@there.com this gives 3 result which is consistent with the amount of Rating(s) I created

 

image

 

So that’s cool this means that we have successfully integrated publishing of a JSON payload Rating object through Kafka to the the Kafka streams Rating processor…….Happy days

 

 

Conclusion

Straight after the last article I decided to Dockerize everything (a decision I have now reversed, due to the flay nature of Dockers dependsOn and it not truly waiting for the item depended on even when using “condition : server_healthy” and “healthcheck – test” etc et), and some code must have become corrupt, as stuff from the last post stopped working.

 

An example from the Docker-Compose docs being

 

version: '2.1'
services:
  web:
    build: .
    depends_on:
      db:
        condition: service_healthy
      redis:
        condition: service_started
  redis:
    image: redis
  db:
    image: redis
    healthcheck:
      test: "exit 0"

 

I love Docker but for highly complex setups I think you are better off using a Docker-Compose file but just not trying to bring it all up in one go. I may bring the Docker bits back into the fold for anyone that is reading this that wants to play around, but I will have to think about that closely.

Once I realized that my Docker campaign was doing more harm than good, and I reverted back to my extremely hand coded, but deterministic PowerShell startup script, I found that getting the Play Framework and a Reactive Kafka (akka streams Kafka producer up and running was quite simple, and it kind of worked like a charm first time. Yay

 

 

Next Time

Next time we should be able to making the entire rating view page work as we now have the following things

So we should quite easily be able to turn that data into a simple bootstrap table in the React portion of this application

MADCAP IDEA 9 : KAFKA STREAMS INTERACTIVE QUERIES

Last Time

 

So last time we came up with a sort of 1/2 way house type post that would pave the way for this one, where we examined several different types of REST frameworks for use with Scala. The requirements were that we would be able to use JSON and that there would be both a client and server side API for the chosen library. For http4s and Akka Http worked. I decided to go with Akka Http due to being more familiar with it.

So that examination of REST APIs has allowed this post to happen. In this post what we will be doing is looking at

  • How to install Kafka/Zookeeper and get them running on Windows
  • Walking through a KafkaProducer
  • Walking through a Kafka Streams processing node, and the duality of streams
  • Walking through Kafka Streams interactive queries

 

PreAmble

Just as a reminder this is part of my ongoing set of posts which I talk about here :

https://sachabarbs.wordpress.com/2017/05/01/madcap-idea/, where we will be building up to a point where we have a full app using lots of different stuff, such as these

 

  • WebPack
  • React.js
  • React Router
  • TypeScript
  • Babel.js
  • Akka
  • Scala
  • Play (Scala Http Stack)
  • MySql
  • SBT
  • Kafka
  • Kafka Streams

 

Ok so now that we have the introductions out of the way, lets crack on with what we want to cover in this post.

 

Where is the code?

As usual the code is on GitHub here : https://github.com/sachabarber/MadCapIdea

 

Before we start, what is this post all about?

 

Well I don’t know if you recall but we are attempting to create a uber simple uber type application where there are drivers/clients. A client can put out a new job, drivers bid for it. And at the end of a job they can rate each other, see the job completion/rating/view rating sections of this previous post : https://sachabarbs.wordpress.com/2017/06/27/madcap-idea-part-6-static-screen-design/

 

The ratings will be placed into streams and aggregated into permanent storage, and will be available for querying later to display in the react front end. The rating should be grouped by email, and also searchable via the use of an email.

 

So that is what we are attempting to cover in this post. However since this is the 1st time we have had to use Kafka, this post will also talk through what you have to go through to get Kafka setup on windows. In a subsequent post I will attempt to get EVERYTHING up and working (including the Play front end) in Docker containers, but for now we will assume a local install of Kafka, if nothing else its good to know how to set this up

 

How to install Kafka/Zookeeper and get them running on Windows

This section will talk you through how to get Kafka and get it working on Windows

 

Step 1 : Download Kafka

Grab Confluence Platform 3.3.0 Open Source : http://packages.confluent.io/archive/3.3/confluent-oss-3.3.0-2.11.zip

 

Step 2 : Update Dodgy BAT Files

The official Kafka windows BAT files don’t seem to work in the Confluence Platform 3.3.0 Open Source download. So replace the official [YOUR INSTALL PATH]\confluent-3.3.0\bin\windows BAT files with the ones found here : https://github.com/renukaradhya/confluentplatform/tree/master/bin/windows

 

Step 3 : Make Some Minor Changes To Log Locations etc etc

Kafka/Zookeeper as installed are setup for Linux, as such these paths won’t work on Windows. So we need to adjust that a bit. So lets do that now

 

  • Modify the [YOUR INSTALL PATH]\confluent-3.3.0\etc\kafka\zookeeper.properties file to change the dataDir to something like dataDir=c:/temp/zookeeper
  • Modify the [YOUR INSTALL PATH]\confluent-3.3.0\etc\kafka\server.properties file to uncomment the line delete.topic.enable=true

 

Step 4 : Running Zookeeper + Kafka + Creating Topics

Now that we have installed everything, it’s just a matter of running stuff up. Sadly before we can run Kafka we need to run Zookeeper, and before Kafka can send messages we need to ensure that the Kafka topics are created. Topics must exist before messages

 

Mmm that sounds like a fair bit of work. Well it is, so I decided to script this into a little PowerShell script. This script is available within the source code at https://github.com/sachabarber/MadCapIdea/tree/master/PowerShellProject/PowerShellProject where the script is called RunPipeline.ps1

 

So all we need to do is change to the directory that the RunPipeline.ps1 is in, and run it.

Obviously it is setup to my installation folders, you WILL have to change the variables at the top of the script if you want to run this on your own machine

 

Here is the contents of the RunPipeline.ps1 file

 

$global:mongoDbInstallationFolder = "C:\Program Files\MongoDB\Server\3.5\bin\"
$global:kafkaWindowsBatFolder = "C:\Apache\confluent-3.3.0\bin\windows\"
$global:kafkaTopics = 
	"rating-submit-topic",
	"rating-output-topic"
$global:ProcessesToKill = @()



function RunPipeLine() 
{
	WriteHeader "STOPPING PREVIOUS SERVICES"
	StopZookeeper
	StopKafka

	Start-Sleep -s 20
	
	WriteHeader "STARTING NEW SERVICE INSTANCES"
	StartZookeeper
	StartKafka
	
	Start-Sleep -s 20

	CreateKafkaTopics
    RunMongo

	WaitForKeyPress

	WriteHeader "KILLING PROCESSES CREATED BY SCRIPT"
	KillProcesses
}

function WriteHeader($text) 
{
	Write-Host "========================================`r`n"
	Write-Host "$text`r`n"
	Write-Host "========================================`r`n"
}


function StopZookeeper() {
	$zookeeperCommandLine = $global:kafkaWindowsBatFolder + "zookeeper-server-stop.bat"
	Write-Host "> Zookeeper Command Line : $zookeeperCommandLine`r`n"
    $global:ProcessesToKill += start-process $zookeeperCommandLine -WindowStyle Normal -PassThru
}

function StopKafka() {
	$kafkaServerCommandLine = $global:kafkaWindowsBatFolder + "kafka-server-stop.bat" 
	Write-Host "> Kafka Server Command Line : $kafkaServerCommandLine`r`n"
    $global:ProcessesToKill += start-process $kafkaServerCommandLine  -WindowStyle Normal -PassThru
}

function StartZookeeper() {
	$zookeeperCommandLine = $global:kafkaWindowsBatFolder + "zookeeper-server-start.bat"
	$arguments = $global:kafkaWindowsBatFolder + "..\..\etc\kafka\zookeeper.properties"
	Write-Host "> Zookeeper Command Line : $zookeeperCommandLine args: $arguments `r`n"
    $global:ProcessesToKill += start-process $zookeeperCommandLine $arguments -WindowStyle Normal -PassThru
}

function StartKafka() {
	$kafkaServerCommandLine = $global:kafkaWindowsBatFolder + "kafka-server-start.bat" 
	$arguments = $global:kafkaWindowsBatFolder + "..\..\etc\kafka\server.properties"
	Write-Host "> Kafka Server Command Line : $kafkaServerCommandLine args: $arguments `r`n"
    $global:ProcessesToKill += start-process $kafkaServerCommandLine $arguments -WindowStyle Normal -PassThru
}

function CreateKafkaTopics() 
{
	Foreach ($topic in $global:kafkaTopics )
	{
		$kafkaCommandLine = $global:kafkaWindowsBatFolder + "kafka-topics.bat"
		$arguments = "--zookeeper localhost:2181 --create  --replication-factor 1 --partitions 1 --topic $topic"
		Write-Host "> Create Kafka Topic Command Line : $kafkaCommandLine args: $arguments `r`n"
		$global:ProcessesToKill += start-process $kafkaCommandLine $arguments -WindowStyle Normal -PassThru
	}
}

function RunMongo() {
	$mongoexe = $global:mongoDbInstallationFolder + "mongod.exe"
	Write-Host "> Mongo Command Line : $mongoexe `r`n" 
	$global:ProcessesToKill += Start-Process -FilePath $mongoexe  -WindowStyle Normal -PassThru
}

function WaitForKeyPress
{
	Write-Host -NoNewLine "Press any key to continue....`r`n"
	[Console]::ReadKey()
}


function KillProcesses() 
{
	Foreach ($processToKill in $global:ProcessesToKill )
	{
		$name = $processToKill | Get-ChildItem -Name
		Write-Host "Killing Process : $name `r`n" 
		$processToKill | Stop-Process -Force
	}
}


# Kick of the entire pipeline
RunPipeLine

 

 

As you can see this script does a bit more than just run up Zookeeper and Kafka, it also create the topics and runs Mongo DB that is also required by the main Play application (remember we are using Reactive Mongo for the login/registration side of things)

 

So far I have not had many issues with the script. Though occasionally when you are trying out new code, I do tend to clear out all the Zookeeper/Kafka state so far, which for me is stored here

 

image

 

It just allows me to start with a clean slate as it were, you should need to do this that often

 

Walking through a KafkaProducer

So the Kafka Producer I present here will send a String key and a JSON Ranking object as a payload. Lets have a quick look at the Ranking object and how it gets turned to and from JSON before we look at the producer code

 

The Domain Entities

Here is what the domain entities looks like, it can be seen that these use the Spray formatters (part of Akka Http) Marshaller/UnMarshaller JSON support. This is not required by the producer but is required by the REST API, which we will look at later. The producer and Kafka streams code work with a different serialization abstraction, something known as SERDES, which as far as I know is only used as a term in Kafka. Its quite simple it stands for Serializer-Deserializer (Serdes)

 

Anyway here are the domain entities and Spray formatters that allow Akka (but not Kafka Streams, more on this later) to work

 

package Entities

import akka.http.scaladsl.marshallers.sprayjson.SprayJsonSupport._
import spray.json.DefaultJsonProtocol._

case class Ranking(fromEmail: String, toEmail: String, score: Float)
case class HostStoreInfo(host: String, port: Int, storeNames: List[String])

object AkkaHttpEntitiesJsonFormats {
  implicit val RankingFormat = jsonFormat3(Ranking)
  implicit val HostStoreInfoFormat = jsonFormat3(HostStoreInfo)
}

 

Serdes

So as we just described Kafka Streams actually cares not for the standard Akka Http/Spray JSON formatters, its not part of Kafka Streams after all. However Kafka Streams still has some of the same concerns where it needs to serialize and de-serialize data (like when it re-partitions (i.e. shuffles data)), so how does it realize that. Well it uses a weirdly name thing called a “serde”. There are MANY inbuilt “serde” types, but you can of course create your own to represent your “on the wire format”. I am using JSON, so this is what my generic “Serde” implementation looks like. I should point out that I owe a great many things to a chap called Jendrik whom I made contact with who has been doing some great stuff with Kafka, his blog has really helped me out : https://www.madewithtea.com/category/kafka-streams.html

 

Anyway here is my/his/yours/ours “Serde” code for JSON

 

import java.lang.reflect.{ParameterizedType, Type}
import java.util
import com.fasterxml.jackson.annotation.JsonInclude
import com.fasterxml.jackson.core.JsonParseException
import com.fasterxml.jackson.core.`type`.TypeReference
import com.fasterxml.jackson.databind.ObjectMapper
import com.fasterxml.jackson.databind.exc.{UnrecognizedPropertyException => UPE}
import com.fasterxml.jackson.module.scala.DefaultScalaModule
import org.apache.kafka.common.serialization.{Deserializer, Serde, Serializer}


package Serialization {

  object Json {

    type ParseException = JsonParseException
    type UnrecognizedPropertyException = UPE

    private val mapper = new ObjectMapper()
    mapper.registerModule(DefaultScalaModule)
    mapper.setSerializationInclusion(JsonInclude.Include.NON_NULL)

    private def typeReference[T: Manifest] = new TypeReference[T] {
      override def getType = typeFromManifest(manifest[T])
    }

    private def typeFromManifest(m: Manifest[_]): Type = {
      if (m.typeArguments.isEmpty) {
        m.runtimeClass
      }
      else new ParameterizedType {
        def getRawType = m.runtimeClass

        def getActualTypeArguments = m.typeArguments.map(typeFromManifest).toArray

        def getOwnerType = null
      }
    }

    object ByteArray {
      def encode(value: Any): Array[Byte] = mapper.writeValueAsBytes(value)

      def decode[T: Manifest](value: Array[Byte]): T =
        mapper.readValue(value, typeReference[T])
    }

  }

  /**
    * JSON serializer for JSON serde
    *
    * @tparam T
    */
  class JSONSerializer[T] extends Serializer[T] {
    override def configure(configs: util.Map[String, _], isKey: Boolean): Unit = ()

    override def serialize(topic: String, data: T): Array[Byte] =
      Json.ByteArray.encode(data)

    override def close(): Unit = ()
  }

  /**
    * JSON deserializer for JSON serde
    *
    * @tparam T
    */
  class JSONDeserializer[T >: Null <: Any : Manifest] extends Deserializer[T] {
    override def configure(configs: util.Map[String, _], isKey: Boolean): Unit = ()

    override def close(): Unit = ()

    override def deserialize(topic: String, data: Array[Byte]): T = {
      if (data == null) {
        return null
      } else {
        Json.ByteArray.decode[T](data)
      }
    }
  }

  /**
    * JSON serde for local state serialization
    *
    * @tparam T
    */
  class JSONSerde[T >: Null <: Any : Manifest] extends Serde[T] {
    override def deserializer(): Deserializer[T] = new JSONDeserializer[T]

    override def configure(configs: util.Map[String, _], isKey: Boolean): Unit = ()

    override def close(): Unit = ()

    override def serializer(): Serializer[T] = new JSONSerializer[T]
  }

}

 

And finally here is what the producer code looks like

 

As you can see it is actually fairly straight forward, its simple produces 10 messages (though you can uncomment the line in the code to make it endless) of String as a key and a Ranking as the message. The key would be who the ranking is destined for. These will then be aggregated by the streams processing node, and stored as a list against a Key (ie email)

 


package Processing.Ratings {

  import java.util.concurrent.TimeUnit

  import Entities.Ranking
  import Serialization.JSONSerde
  import Topics.RatingsTopics

  import scala.util.Random
  import org.apache.kafka.clients.producer.ProducerRecord
  import org.apache.kafka.clients.producer.KafkaProducer
  import org.apache.kafka.common.serialization.Serdes
  import Utils.Settings
  import org.apache.kafka.clients.producer.ProducerConfig

  object RatingsProducerApp extends App {

   run()

    private def run(): Unit = {

      val jSONSerde = new JSONSerde[Ranking]
      val random = new Random
      val producerProps = Settings.createBasicProducerProperties
      val rankingList = List(
        Ranking("jarden@here.com","sacha@here.com", 1.5f),
        Ranking("miro@here.com","mary@here.com", 1.5f),
        Ranking("anne@here.com","margeret@here.com", 3.5f),
        Ranking("frank@here.com","bert@here.com", 2.5f),
        Ranking("morgan@here.com","ruth@here.com", 1.5f))

      producerProps.put(ProducerConfig.ACKS_CONFIG, "all")

      System.out.println("Connecting to Kafka cluster via bootstrap servers " +
        s"${producerProps.getProperty(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG)}")

      // send a random string from List event every 100 milliseconds
      val rankingProducer = new KafkaProducer[String, Array[Byte]](
        producerProps, Serdes.String.serializer, Serdes.ByteArray.serializer)

      //while (true) {
      for (i <- 0 to 10) {
        val ranking = rankingList(random.nextInt(rankingList.size))
        val rankingBytes = jSONSerde.serializer().serialize("", ranking)
        System.out.println(s"Writing ranking ${ranking} to input topic ${RatingsTopics.RATING_SUBMIT_TOPIC}")
        rankingProducer.send(new ProducerRecord[String, Array[Byte]](
          RatingsTopics.RATING_SUBMIT_TOPIC, ranking.toEmail, rankingBytes))
        Thread.sleep(100)
      }

      Runtime.getRuntime.addShutdownHook(new Thread(() => {
        rankingProducer.close(10, TimeUnit.SECONDS)
      }))
    }
  }
}

 

 

 

Walking through a Kafka Streams processing node, and the duality of streams

Before we get started I just wanted to include a severak excerpts taken from the official Kafka docs : http://docs.confluent.io/current/streams/concepts.html#duality-of-streams-and-tables which talks about KStream and KTable objects (which are the stream and table objects inside Kafka streams)

 

When implementing stream processing use cases in practice, you typically need both streams and also databases. An example use case that is very common in practice is an e-commerce application that enriches an incoming stream of customer transactions with the latest customer information from a database table. In other words, streams are everywhere, but databases are everywhere, too.

Any stream processing technology must therefore provide first-class support for streams and tables. Kafka’s Streams API provides such functionality through its core abstractions for streams and tables, which we will talk about in a minute. Now, an interesting observation is that there is actually a close relationship between streams and tables, the so-called stream-table duality. And Kafka exploits this duality in many ways: for example, to make your applications elastic, to support fault-tolerant stateful processing, or to run interactive queries against your application’s latest processing results. And, beyond its internal usage, the Kafka Streams API also allows developers to exploit this duality in their own applications.

 

A simple form of a table is a collection of key-value pairs, also called a map or associative array. Such a table may look as follows:

 

    ../_images/streams-table-duality-01.jpg

     

    The stream-table duality describes the close relationship between streams and tables.

     

    • Stream as Table: A stream can be considered a changelog of a table, where each data record in the stream captures a state change of the table. A stream is thus a table in disguise, and it can be easily turned into a “real” table by replaying the changelog from beginning to end to reconstruct the table. Similarly, aggregating data records in a stream will return a table. For example, we could compute the total number of pageviews by user from an input stream of pageview events, and the result would be a table, with the table key being the user and the value being the corresponding pageview count.
    • Table as Stream: A table can be considered a snapshot, at a point in time, of the latest value for each key in a stream (a stream’s data records are key-value pairs). A table is thus a stream in disguise, and it can be easily turned into a “real” stream by iterating over each key-value entry in the table.

     

    Let’s illustrate this with an example. Imagine a table that tracks the total number of pageviews by user (first column of diagram below). Over time, whenever a new pageview event is processed, the state of the table is updated accordingly. Here, the state changes between different points in time – and different revisions of the table – can be represented as a changelog stream (second column).

     

    image

     

    Because of the stream-table duality, the same stream can be used to reconstruct the original table (third column):

     

    ../_images/streams-table-duality-03.jpg

     

    The same mechanism is used, for example, to replicate databases via change data capture (CDC) and, within Kafka Streams, to replicate its so-called state stores across machines for fault tolerance. The stream-table duality is such an important concept for stream processing applications in practice that Kafka Streams models it explicitly via the KStream and KTable abstractions, which we describe in the next sections.

     

     

    I would STRONLY urge you all to read the section of the official docs above, as it will really help you should you want to get into Kafka Streams.

     

    Anyway with all that in mind how does that relate to the use case we are trying to solve. So far we have a publisher that pushes out Rating objects, and as stated ideally we would like to query these across all  processor nodes. As such we should now know that this will involve a KStream and some sort of aggregation to an eventual KTable (where a state store will be used).

     

    Probably the easiest thing to do is to start with the code, which looks like this for the main stream processing code for the Rating section of then final app.

      import java.util.concurrent.TimeUnit
      import org.apache.kafka.common.serialization._
      import org.apache.kafka.streams._
      import org.apache.kafka.streams.kstream._
      import Entities.Ranking
      import Serialization.JSONSerde
      import Topics.RatingsTopics
      import Utils.Settings
      import Stores.StateStores
      import org.apache.kafka.streams.state.HostInfo
      import scala.concurrent.ExecutionContext
    
    
      package Processing.Ratings {
    
        class RankingByEmailInitializer extends Initializer[List[Ranking]] {
          override def apply(): List[Ranking] = List[Ranking]()
        }
    
        class RankingByEmailAggregator extends Aggregator[String, Ranking,List[Ranking]] {
          override def apply(aggKey: String, value: Ranking, aggregate: List[Ranking]) = {
            value :: aggregate
          }
        }
    
    
        object RatingStreamProcessingApp extends App {
    
          implicit val ec = ExecutionContext.global
    
          run()
    
          private def run() : Unit = {
            val stringSerde = Serdes.String
            val rankingSerde = new JSONSerde[Ranking]
            val listRankingSerde = new JSONSerde[List[Ranking]]
            val builder: KStreamBuilder = new KStreamBuilder
            val rankings = builder.stream(stringSerde, rankingSerde, RatingsTopics.RATING_SUBMIT_TOPIC)
    
            //aggrgate by (user email -> their rankings)
            val rankingTable = rankings.groupByKey(stringSerde,rankingSerde)
              .aggregate(
                new RankingByEmailInitializer(),
                new RankingByEmailAggregator(),
                listRankingSerde,
                StateStores.RANKINGS_BY_EMAIL_STORE
              )
    
            //useful debugging aid, print KTable contents
            rankingTable.toStream.print()
    
            val streams: KafkaStreams = new KafkaStreams(builder, Settings.createRatingStreamsProperties)
            val restEndpoint:HostInfo  = new HostInfo(Settings.restApiDefaultHostName, Settings.restApiDefaultPort)
            System.out.println(s"Connecting to Kafka cluster via bootstrap servers ${Settings.bootStrapServers}")
            System.out.println(s"REST endpoint at http://${restEndpoint.host}:${restEndpoint.port}")
    
            // Always (and unconditionally) clean local state prior to starting the processing topology.
            // We opt for this unconditional call here because this will make it easier for you to 
            // play around with the example when resetting the application for doing a re-run 
            // (via the Application Reset Tool,
            // http://docs.confluent.io/current/streams/developer-guide.html#application-reset-tool).
            //
            // The drawback of cleaning up local state prior is that your app must rebuilt its local 
            // state from scratch, which will take time and will require reading all the state-relevant 
            // data from the Kafka cluster over the network.
            // Thus in a production scenario you typically do not want to clean up always as we do 
            // here but rather only when it is truly needed, i.e., only under certain conditions 
            // (e.g., the presence of a command line flag for your app).
            // See `ApplicationResetExample.java` for a production-like example.
            streams.cleanUp();
            streams.start()
            val restService = new RatingRestService(streams, restEndpoint)
            restService.start()
    
            Runtime.getRuntime.addShutdownHook(new Thread(() => {
              streams.close(10, TimeUnit.SECONDS)
              restService.stop
            }))
    
            //return unit
            ()
          }
        }
      }
    

     

     

    Remember the idea is to get a Rating for  a user (based on their email address), and store all the Rating associated with them in some sequence/list such that they can be retrieved in one go based on a a key, where the key would be the users email, and the value would be this list of Rating objects.I think with the formal discussion from the official Kafka docs and my actual Rating requirement, the above should hopefully be pretty clear.

     

     

    Walking through Kafka Streams interactive queries

    So now that we have gone through how data is produced, and transformed (well actually I did not do too much transformation other than a simple map, but trust me you can), and how we aggregate results from a KStream to a KTable (and its state store), we will move on to see how we can use Kafka interactive queries to query the state stores.

     

    One important concept is that if you used multiple partitions for your original topic, the state may be spread across n-many processing node. For this project I have only chosen to use 1 partition, but have written the code to support n-many.

     

    So lets assume that each node could read a different segment of data, or that each node must read from n-many partitions (there is not actually a mapping to nodes and partitions these are 2 mut read chapters elastic-scaling-of-your-application and parallelism-model) we would need each node to expose a REST API to allow its OWN state store to be read. By reading ALL the state stores we are able to get a total view of ALL the persisted data across ALL the partitions. I urge all of you to read this section of the official docs : http://docs.confluent.io/current/streams/developer-guide.html#querying-remote-state-stores-for-the-entire-application

     

    This diagram has also be shamelessly stolen from the official docs:

     

    ../_images/streams-interactive-queries-api-02.png

    I think this diagram does an excellent job of showing you 3 separate processor nodes, and each of them may have a bit of state. ONLY be assembling ALL the data from these nodes are we able to see the ENTIRE dataset.

     

    Kafka allows this via metadata about the streams, where we can use the exposed metadata to help us gather the state store data. To do this we first need a MetadataService, which for me is as follows:

     

    package Processing.Ratings
    
    import org.apache.kafka.streams.KafkaStreams
    import org.apache.kafka.streams.state.StreamsMetadata
    import java.util.stream.Collectors
    import Entities.HostStoreInfo
    import org.apache.kafka.common.serialization.Serializer
    import org.apache.kafka.connect.errors.NotFoundException
    import scala.collection.JavaConverters._
    
    
    /**
      * Looks up StreamsMetadata from KafkaStreams
      */
    class MetadataService(val streams: KafkaStreams) {
    
    
       /**
        * Get the metadata for all of the instances of this Kafka Streams application
        *
        * @return List of { @link HostStoreInfo}
        */
      def streamsMetadata() : List[HostStoreInfo] = {
    
        // Get metadata for all of the instances of this Kafka Streams application
        val metadata = streams.allMetadata
        return mapInstancesToHostStoreInfo(metadata)
      }
    
    
      /**
        * Get the metadata for all instances of this Kafka Streams application that currently
        * has the provided store.
        *
        * @param store The store to locate
        * @return List of { @link HostStoreInfo}
        */
      def streamsMetadataForStore(store: String) : List[HostStoreInfo] = {
    
        // Get metadata for all of the instances of this Kafka Streams application hosting the store
        val metadata = streams.allMetadataForStore(store)
        return mapInstancesToHostStoreInfo(metadata)
      }
    
    
      /**
        * Find the metadata for the instance of this Kafka Streams Application that has the given
        * store and would have the given key if it exists.
        *
        * @param store Store to find
        * @param key   The key to find
        * @return { @link HostStoreInfo}
        */
      def streamsMetadataForStoreAndKey[T](store: String, key: T, serializer: Serializer[T]) : HostStoreInfo = {
        // Get metadata for the instances of this Kafka Streams application hosting the store and
        // potentially the value for key
        val metadata = streams.metadataForKey(store, key, serializer)
        if (metadata == null)
          throw new NotFoundException(
            s"No metadata could be found for store : ${store}, and key type : ${key.getClass.getName}")
    
        HostStoreInfo(metadata.host, metadata.port, metadata.stateStoreNames.asScala.toList)
      }
    
    
      def mapInstancesToHostStoreInfo(metadatas : java.util.Collection[StreamsMetadata]) : List[HostStoreInfo] = {
    
        metadatas.stream.map[HostStoreInfo](metadata =>
          HostStoreInfo(
            metadata.host(),
            metadata.port,
            metadata.stateStoreNames.asScala.toList))
          .collect(Collectors.toList())
          .asScala.toList
      }
    
    }
    

     

    This metadata service is used to obtain the state store information, which we can then use to extract the state data we want (it’s a key value store really).

     

    The next thing we need to do is expose a REST API to allow us to get the state. lets see that now

     

    package Processing.Ratings
    
    import org.apache.kafka.streams.KafkaStreams
    import org.apache.kafka.streams.state.HostInfo
    import akka.actor.ActorSystem
    import akka.http.scaladsl.Http
    import akka.http.scaladsl.model._
    import akka.http.scaladsl.server.Directives._
    import akka.stream.ActorMaterializer
    import akka.http.scaladsl.marshallers.sprayjson.SprayJsonSupport._
    import spray.json.DefaultJsonProtocol._
    import Entities.AkkaHttpEntitiesJsonFormats._
    import Entities._
    import Stores.StateStores
    import akka.http.scaladsl.marshalling.ToResponseMarshallable
    import org.apache.kafka.common.serialization.Serdes
    import scala.concurrent.{Await, ExecutionContext, Future}
    import akka.http.scaladsl.unmarshalling.Unmarshal
    import spray.json._
    import scala.util.{Failure, Success}
    import org.apache.kafka.streams.state.QueryableStoreTypes
    import scala.concurrent.duration._
    
    
    
    object RestService {
      val DEFAULT_REST_ENDPOINT_HOSTNAME  = "localhost"
    }
    
    
    class RatingRestService(val streams: KafkaStreams, val hostInfo: HostInfo) {
    
      val metadataService = new MetadataService(streams)
      var bindingFuture: Future[Http.ServerBinding] = null
    
      implicit val system = ActorSystem("rating-system")
      implicit val materializer = ActorMaterializer()
      implicit val executionContext = system.dispatcher
    
    
      def start() : Unit = {
        val emailRegexPattern =  """\w+""".r
        val storeNameRegexPattern =  """\w+""".r
    
        val route =
    
    
          path("test") {
            get {
              parameters('email.as[String]) { (email) =>
                complete(HttpEntity(ContentTypes.`text/html(UTF-8)`,
                            s"<h1>${email}</h1>"))
              }
            }
          } ~
          path("ratingByEmail") {
            get {
              parameters('email.as[String]) { (email) =>
                try {
    
                  val host = metadataService.streamsMetadataForStoreAndKey[String](
                    StateStores.RANKINGS_BY_EMAIL_STORE,
                    email,
                    Serdes.String().serializer()
                  )
    
                  var future:Future[List[Ranking]] = null
    
                  //store is hosted on another process, REST Call
                  if(!thisHost(host))
                    future = fetchRemoteRatingByEmail(host, email)
                  else
                    future = fetchLocalRatingByEmail(email)
    
                  val rankings = Await.result(future, 20 seconds)
                  complete(rankings)
                }
                catch {
                  case (ex: Exception) => {
                    val finalList:List[Ranking] = scala.collection.immutable.List[Ranking]()
                    complete(finalList)
                  }
                }
              }
            }
          } ~
          path("instances") {
            get {
              complete(ToResponseMarshallable.apply(metadataService.streamsMetadata))
            }
          }~
          path("instances" / storeNameRegexPattern) { storeName =>
            get {
              complete(ToResponseMarshallable.apply(metadataService.streamsMetadataForStore(storeName)))
            }
          }
    
        bindingFuture = Http().bindAndHandle(route, hostInfo.host, hostInfo.port)
        println(s"Server online at http://${hostInfo.host}:${hostInfo.port}/\n")
    
        Runtime.getRuntime.addShutdownHook(new Thread(() => {
          bindingFuture
            .flatMap(_.unbind()) // trigger unbinding from the port
            .onComplete(_ => system.terminate()) // and shutdown when done
        }))
      }
    
    
      def fetchRemoteRatingByEmail(host:HostStoreInfo, email: String) : Future[List[Ranking]] = {
    
        val requestPath = s"http://${hostInfo.host}:${hostInfo.port}/ratingByEmail?email=${email}"
        println(s"Client attempting to fetch from online at ${requestPath}")
    
        val responseFuture: Future[List[Ranking]] = {
          Http().singleRequest(HttpRequest(uri = requestPath))
            .flatMap(response => Unmarshal(response.entity).to[List[Ranking]])
        }
    
        responseFuture
      }
    
      def fetchLocalRatingByEmail(email: String) : Future[List[Ranking]] = {
    
        val ec = ExecutionContext.global
    
        val host = metadataService.streamsMetadataForStoreAndKey[String](
          StateStores.RANKINGS_BY_EMAIL_STORE,
          email,
          Serdes.String().serializer()
        )
    
        val f = StateStores.waitUntilStoreIsQueryable(
          StateStores.RANKINGS_BY_EMAIL_STORE,
          QueryableStoreTypes.keyValueStore[String,List[Ranking]](),
          streams
        ).map(_.get(email))(ec)
    
        val mapped = f.map(ranking => {
          if (ranking == null)
            List[Ranking]()
          else
            ranking
        })
    
        mapped
      }
    
      def stop() : Unit = {
        bindingFuture
          .flatMap(_.unbind()) // trigger unbinding from the port
          .onComplete(_ => system.terminate()) // and shutdown when done
      }
    
      def thisHost(hostStoreInfo: HostStoreInfo) : Boolean = {
        hostStoreInfo.host.equals(hostInfo.host()) &&
          hostStoreInfo.port == hostInfo.port
      }
    }
    

     

    With that final class we are able to run the application and query it using the url http://localhost:8080/ratingByEmail?email=sacha@here.com (the key to the Kafka store here is “sacha@here.com” and the value could either be an empty list or a List[Ranking] objects as JSON, the results of which are shown below after we have run the producer and used Chrome (or any other REST tool of your picking) to get the results

     

    image

     

     

    Conclusion

    I have found the journey to get here an interesting one. The main issue being that the Kafka docs and example are all written in Java and some are not even using Java Lambdas (Java 1.8) so the translation from that to Scala code (where there is lambda everywhere) is sometimes trickier than you might think.

     

    The other thing that has caught me out a few times is that the Scala type system is pretty good at inferring the correct types, so you kind of let it get on with its job. But occasionally it doesn’t/can’t infer the type correctly, this may happen at compile time if you are lucky, or at run time. In the case of a runtime issue, I found it fairly hard to see exactly which part of the Kafka streams API would need to be told a bit more type information.

     

    As a general rule of thumb, if there is an overloaded method that takes a serde, and  one that doesn’t ALWAYS use the one that takes a serde and specify the generic type parameters explicitly. The methods that take Serdes are usually ones that involve some sort of shuffling around within partitions so need Serdes to serialize and deserialize correctly.

     

    Other than that I am VERY happy with working with Kafka streams, and once you get into it, its not that different from working with Apache Spark and RDDs

     

     

    Next time

    Next time we will be turning our attention back to the web site, where we will expose an endpoint that can be called from the ratings dialog that is launched at the end of a job. This endpoint will take the place of the RatingsProducerApp demonstrated in this app. For that we will be using https://github.com/akka/reactive-kafka. We would also expose a new end point to fetch the rating (via email address) fro the Kafka stream processor node

     

    The idea being that when a job is completed a Rating from a driver to passenger is given, this is sent to the Kafka stream processor node, and the combined rating are accumulated for users (by email address) and are exposed to be queried. As you can see this post covers the later part of this requirement already. The only thing we would need to do (as stated above) is replace the RatingsProducerApp demonstrated in this app with new reactive kafka producer in the main Play application

    MADCAP IDEA PART 8 : INTERMEDIATE STEP, REST API FOR INTERACTIVE KAFKA STREAM KTABLE QUERIES

    Last Time

     

    So this one has taken a very long time to get up, I have been on a few weeks holiday and a few other things cropped up. This is also an intermediate post that the next one will build on, I just thought it might be useful to show this one in isolation, before we move on to the real one which is to allow interactive queries over live Kafka streams. In order to carry out the live queries we need to expose a REST endpoint over the Kafka stream nodes, which will be the focus of the next article. This one will focus on a simple REST example in Scala.

     

    PreAmble

    Just as a reminder this is part of my ongoing set of posts which I talk about here :

    https://sachabarbs.wordpress.com/2017/05/01/madcap-idea/, where we will be building up to a point where we have a full app using lots of different stuff, such as these

     

    • WebPack
    • React.js
    • React Router
    • TypeScript
    • Babel.js
    • Akka
    • Scala
    • Play (Scala Http Stack)
    • MySql
    • SBT
    • Kafka
    • Kafka Streams

     

    Ok so now that we have the introductions out of the way, lets crack on with what we want to cover in this post. Which is how to create a simple Client/Server REST API in Scala.

     

    There are several choices available, that I wanted to consider

     

    I did actually try out all 3 of these, with varying degrees of success. I am using Scala 2.12 and IntelliJ IDEA 17. So even though I settled on Akka Http (as I have used it before and it just works), I thought it may be useful to show a simple hello world of the server examples in the other 2

     

    Finch Server example

    So this would be a simple REST server written in Finch

     

    import io.finch._
    import com.twitter.finagle.Http
    
    val api: Endpoint[String] = get("hello") { Ok("Hello, World!") }
    
    Http.server.serve(":8080", api.toServiceAs[Text.Plain])
    

     

     

    The main points that I liked about using Finch were

    • Its functional
    • I liked the ScalaZ composable style endpoints
    • It had good JSON support (Circe, Argonaut, Jackson, Json4s, PlayJson)
    • It was very concise

    The reason I chose not to use it

    • The endpoint composition seems to get very confused in IntelliJ IDEA and required you to resort to use SBT command line, and IntelliJ IDEA would still complain about the syntax
    • No client library

     

     

    HTTP4S Server/Client example

    So this would be a simple REST server/client written in Http4s

     

    SBT

    you will need a SBT file something like this, I opted to go for a version that was considered stable, not the current develop version

    name := "SImpleRestApi"
    
    version := "1.0"
    
    scalaVersion := "2.12.1"
    
    val http4sVersion = "0.15.16"
    
    
    libraryDependencies ++= Seq(
      "org.http4s" %% "http4s-dsl"          % http4sVersion,
      "org.http4s" %% "http4s-blaze-server" % http4sVersion,
      "org.http4s" %% "http4s-blaze-client" % http4sVersion,
      "org.http4s" %% "http4s-circe"        % http4sVersion,
      "io.circe"   %% "circe-generic"       % "0.6.1",
      "io.circe"   %% "circe-literal"       % "0.6.1"
    
    )
            
    

     

    Common Entities

    case class User(name: String)
    case class Hello(greeting: String)
    

     

    The Server

    This is a simple http4s server that will accept a “User” case class JSON payload on the “hello” route, and will return a “Hello” case class as JSON

    import Entities.{Hello, User}
    import io.circe.generic.auto._
    import io.circe.syntax._
    import org.http4s._
    import org.http4s.circe._
    import org.http4s.dsl._
    import org.http4s.server.blaze._
    
    object RestServer extends App {
    
      run()
    
    
      def run() : Unit = {
    
        val jsonService = HttpService {
          case req @ POST -> Root / "hello" =>
            for {
            // Decode a User request
              user <- req.as(jsonOf[User])
              // Encode a hello response
              resp <- Ok(Hello(user.name).asJson)
            } yield (resp)
    
          case req @ GET -> Root / "test" =>
            for {
    
              resp <- Ok(Hello("Monkey").asJson)
            } yield (resp)
    
        }
    
        val builder = BlazeBuilder.bindHttp(8080).mountService(jsonService, "/")
        val blazeServer = builder.run
    
        scala.io.StdIn.readLine()
        ()
      }
    
    }
    

     

    There are a couple of things to note here

    • It uses Blaze server
    • The routes are constructed in a pretty intuitive manner
    • The JSON encoding/decoding is done seamlessly and did not require anything other than including circle.generic (this is nice)

     

     

    The Client

     

    import Entities.{Hello, User}
    import org.http4s.client.blaze.PooledHttp1Client
    import scalaz.concurrent.Task
    import io.circe.generic.auto._
    import io.circe.syntax._
    import org.http4s.circe._
    import org.http4s.dsl._
    import org.http4s.client._
    
    
    object RestClient extends App {
    
      val httpClient = PooledHttp1Client()
    
      run()
    
    
      def run() : Unit = {
    
        val hello = helloClient("foo").run
        println(s"Hello ${hello}")
        scala.io.StdIn.readLine()
        ()
      }
    
      def helloClient(name: String): Task[Hello] = {
        // Encode a User request
        val req = POST(uri("http://localhost:8080/hello"), User(name).asJson)
        // Decode a Hello response
        httpClient.expect(req)(jsonOf[Hello])
      }
    }
    

    The client API is quite nice and required very little code, and just like the server portion the encoding/decoding to/from JSON is seamless. I also like the use of scalaz.concurrent.Task to manage the asynchronisity

     

     

    The main points that I liked about using http4s were

    • Its functional
    • It looked like a nice API to use
    • It had good JSON support (Circe, Argonaut, Json4s)
    • It was very concise
    • It had a server API + client API

    The reason I chose not to use it

    • It took me a fair while to get the client to work, due to the documentation being a mish mash of versions where they have not kelp documentation for specific versions. So I had to dig around quite a bit to get the version I was working for to work. Once it did work I was happy with it though
    • Documentation is clearly a mish mash of different version, this could be better

     

    So that really only left me one choice, the tried and tested but rather unadventurous Akka-Http, so lets look at that next

     

    Akka Http Server/Client example

    So this would be a simple REST server/client written in Akka Http

     

    SBT

    This is the SBT file that I am using

    name := "AkkaHttpRESTExample"
    
    version := "0.1"
    
    scalaVersion := "2.12.3"
    
    libraryDependencies ++= Seq(
      "com.typesafe.akka" %% "akka-http-spray-json" % "10.0.9",
      "com.typesafe.akka" %% "akka-http" % "10.0.9"
    )
            
    

     

    Common Entities

    These are the common entities.

    package Entities
    
    import akka.http.scaladsl.marshallers.sprayjson.SprayJsonSupport._
    import spray.json.DefaultJsonProtocol._
    
    final case class Item(name: String, id: Long)
    final case class Order(items: List[Item])
    
    object DomainEntitiesJsonFormats {
      implicit val itemFormat = jsonFormat2(Item)
      implicit val orderFormat = jsonFormat1(Order)
    }
    

     

    There are a couple of things to note above.

    • We need to use the JSON marshallers/unmarshallers jsonFormat2 / jsonFormat1 that are available within the spray.json package. These represent JSON marshallers/unmarshallers for case class(es) with 2 and 1 parameters respectively
    • That the actual entities are simple case class(es)

     

     

    The Server

    This is a simple server

    import akka.actor.ActorSystem
    import akka.http.scaladsl.Http
    import akka.http.scaladsl.model._
    import akka.http.scaladsl.server.Directives._
    import akka.stream.ActorMaterializer
    import scala.io.StdIn
    
    
    import akka.http.scaladsl.marshallers.sprayjson.SprayJsonSupport._
    import spray.json.DefaultJsonProtocol._
    import Entities.DomainEntitiesJsonFormats._
    import Entities._
    
    object RestServer extends App {
    
      run()
    
    
      def run() {
        val itemNameRegexPattern =  """\w+""".r
        implicit val system = ActorSystem("my-system")
        implicit val materializer = ActorMaterializer()
        implicit val executionContext = system.dispatcher
    
        val route =
          path("hello") {
            get {
              complete(HttpEntity(ContentTypes.`text/html(UTF-8)`,
                "<h1>Say hello to akka-http</h1>"))
            }
          }~
          path("order" / itemNameRegexPattern) { itemName =>
            get {
              complete(Order(List[Item](Item(itemName, 1))))
            }
          }
    
        val bindingFuture = Http().bindAndHandle(route, "localhost", 8080)
        println(s"Server online at http://localhost:8080/\nPress RETURN to stop...")
    
        Runtime.getRuntime.addShutdownHook(new Thread(() => {
          bindingFuture
            .flatMap(_.unbind()) // trigger unbinding from the port
            .onComplete(_ => system.terminate()) // and shutdown when done
        }))
    
        StdIn.readLine() // let it run until user presses return
      }
    }
    

    The main points to note above are:

    • How the routing DSL works, and we can build multiple routes which are chained together
    • How we send JSON payload back using the complete routing directive
    • How we bind the server to the port and address

     

    The Client

    This is a client that goes hand in hand with the server code above

    import Entities.DomainEntitiesJsonFormats._
    import Entities._
    import akka.actor.ActorSystem
    import akka.http.scaladsl.Http
    import akka.http.scaladsl.marshallers.sprayjson.SprayJsonSupport._
    import akka.http.scaladsl.model._
    import akka.stream.ActorMaterializer
    import scala.concurrent.Future
    import scala.io.StdIn
    import akka.http.scaladsl.unmarshalling.Unmarshal
    import spray.json._
    
    import scala.util.{Failure, Success}
    
    object RestClient extends App {
    
      run()
    
    
      def run() {
    
        implicit val system = ActorSystem("my-system")
        implicit val materializer = ActorMaterializer()
        implicit val executionContext = system.dispatcher
    
        println(s"Client attempting to fetch from online " +
          "at http://localhost:8080/order/beans\nPress RETURN to stop...")
    
        val responseFuture: Future[Order] = {
          Http().singleRequest(HttpRequest(uri = "http://localhost:8080/order/beans"))
            .flatMap(response => Unmarshal(response.entity).to[Order])
        }
    
        responseFuture onComplete {
          case Failure(ex) => System.out.println(s"Failed to perform GET, reason: $ex")
          case Success(response) => System.out.println(s"Server responded with $response")
        }
    
        Runtime.getRuntime.addShutdownHook(new Thread(() => {
          system.terminate() // and shutdown when done
        }))
    
        StdIn.readLine() // let it run until user presses return
      }
    }
    

    There is not too much to note here, the main points are

    • How the response from the Http methods are Future[T] and as such must be handled with standard Future code
    • How we need to Unmarshal the JSON string response back into a case class. This is thanks to the jsonFormat2/jsonFormat1 (marshallers/unmarshallers) we saw earlier

     

    That’s It For Now

    So I have shown a few options here both http4s and Akka Http are fairly easy to use, it’s a shame about Finch and IntelliJ, but at the end of the day I also wanted something with a Client API too

     

     

    Next Time

    Next time we will look the actual Ratings workflow in terms of Kafka Streams, and how we can aggregate incoming rating messages inside a Kafka KStream (which will come from the React from end in the end). We will the look at how we can store these rolling stream values into a Kafka KTable and query the results using Kafka interactive queries

    MADCAP IDEA PART 7 : REGISTRATION/LOGIN BACKEND

     

    Last Time

     

    So let me start with an apology that it has taken me so long to get this new post up since last time. I have snuck in a holiday and been on a training course which really only left me 1 week of train rides to do this in, lame excuse but there you have it.

    Anyway last time we looked at doing all the react markup for the screens, this time we will be doing the backend registration and login functions which will make use of reactive mongo for the play side of things.

     

    PreAmble

    Just as a reminder this is part of my ongoing set of posts which I talk about here :

    https://sachabarbs.wordpress.com/2017/05/01/madcap-idea/, where we will be building up to a point where we have a full app using lots of different stuff, such as these

     

    • WebPack
    • React.js
    • React Router
    • TypeScript
    • Babel.js
    • Akka
    • Scala
    • Play (Scala Http Stack)
    • MySql
    • SBT
    • Kafka
    • Kafka Streams

     

    So I think the best way to cover this post is just dive in to the 2 main topics covered

     

    Registration

     

    Passenger Registration 

     

    As a reminder this is what the Passenger Registration screen looks like

     

    image

     

    And this is the relevant code that deals with the “Register” button being clicked

     

    _handleValidSubmit = (values) => {
        var driver = values;
        var self = this;
    
        $.ajax({
            type: 'POST',
            url: 'registration/save/driver',
            data: JSON.stringify(driver),
            contentType: "application/json; charset=utf-8",
            dataType: 'json'
        })
        .done(function (jdata, textStatus, jqXHR) {
            var redactedDriver = driver;
            redactedDriver.password = "";
            console.log("redacted ${redactedDriver}");
            console.log(redactedDriver);
            console.log("Auth Service");
            console.log(self.props.authService);
            self.props.authService.storeUser(redactedDriver);
            self.setState(
                {
                    okDialogHeaderText: 'Registration Successful',
                    okDialogBodyText: 'You are now registered',
                    okDialogOpen: true,
                    okDialogKey: Math.random()
                });
        })
        .fail(function (jqXHR, textStatus, errorThrown) {
            self.setState(
                {
                    okDialogHeaderText: 'Error',
                    okDialogBodyText: jqXHR.responseText,
                    okDialogOpen: true,
                    okDialogKey: Math.random()
                });
        });
    }
    
    _okDialogCallBack = () => {
        this.setState(
            {
                okDialogOpen: false
            });
        if (this.state.wasSuccessful) {
            hashHistory.push('/');
        }
    }
    

     

     

    There are a couple of things of note there

    • That we use a standard POST that will post the registration data as JSON to the Play API backend code
    • That we also show a standard Boostrap OkDialog which we looked at last time, which when the Ok button is clicked will use the React Router to navigate to the route page

     

    Let’s now turn our attention to the Play API backend code that goes with the “Passenger Registration”

     

    JSON Read/Write support

    The first thing we need to do is to set up support for reading JSON into Scala objects from a JSON string, and also allowing Scala objects to be turned back into a JSON string. For a “Passenger Registration” the Play API Json support comes via 3 Traits namely

    • Reads : Which allows reading a JSON string into a Scala object
    • Writes : Which allows a Scala object to be turned into a JSON string
    • Format : is just a mix of the Reads and Writes Traits and can be used for implicit conversion in place of its components.

    The recommendation for both of these is that they are exposed as implicit vals you can read more about it here  : https://www.playframework.com/documentation/2.6.x/ScalaJsonCombinators

    For the “Passenger Registration” it looks like this

    package Entities
    
    import play.api.libs.json._
    import play.api.libs.functional.syntax._
    
    case class PassengerRegistration(
      fullName: String,
      email: String,
      password: String)
    
    object PassengerRegistration {
      implicit val formatter = Json.format[PassengerRegistration]
    }
    
    object PassengerRegistrationJsonFormatters {
    
      implicit val passengerRegistrationWrites = new Writes[PassengerRegistration] {
        def writes(passengerRegistration: PassengerRegistration) = Json.obj(
          "fullname" -> passengerRegistration.fullName,
          "email" -> passengerRegistration.email,
          "password" -> passengerRegistration.password
        )
      }
    
      implicit val passengerRegistrationReads: Reads[PassengerRegistration] = (
        (JsPath \ "fullname").read[String] and
          (JsPath \ "email").read[String] and
          (JsPath \ "password").read[String]
        )(PassengerRegistration.apply _)
    }
    

     

    Once we have that in place we need to turn our attention to the actual endpoint to support the POST of a PassengerRegistration object. We first need to set up the route in the conf/routes file as follows:

     

    # Registration page

    POST  /registration/save/passenger             controllers.RegistrationController.savePassengerRegistration()

     

    Reactive Mongo

    Now that the route is in place its just a standard Play controller that we need. However I have chosen to use Reactive Mongo as my storage mechanism for registration/login data. This means we have a few things that we need to install

    1. Mongo itself which you can just grab from here : https://www.mongodb.com/download-center
    2. We then need to provision the actual Reactive Mongo Scala library, which we can do using the standard build.sbt file, where we specify the following dependency

     

    libraryDependencies ++= Seq(
      "org.reactivemongo" %% "play2-reactivemongo" % "0.11.12"
    )
    

     

    Ensure Mongod.exe Is Running

    In order to run the app we must ensure that the Mongo server is actually running which for my installation of Mongo means starting “Mongod.exe” from “C:\Program Files\MongoDB\Server\3.5\bin\mongod.exe” before I run the app

     

    The Registration Controller

    So now that we have talked about the JSON Reads/Writes and we know that we need Mongo downloaded and running, lets see what the actual controller looks like shall we. Here is the FULL code for the “Passenger Registration”

     

    package controllers
    
    import javax.inject.Inject
    import play.api.mvc.{Action, Controller, Result}
    import Entities._
    import Entities.DriverRegistrationJsonFormatters._
    import Entities.PassengerRegistrationJsonFormatters._
    import scala.concurrent.{ExecutionContext, Future}
    import play.modules.reactivemongo._
    import play.api.Logger
    import utils.Errors
    import play.api.libs.json._
    import reactivemongo.api.ReadPreference
    import reactivemongo.play.json._
    import collection._
    
    class RegistrationController @Inject()(val reactiveMongoApi: ReactiveMongoApi)
      (implicit ec: ExecutionContext)
      extends Controller with MongoController with ReactiveMongoComponents {
    
      def passRegistrationFuture: Future[JSONCollection] = 
    	database.map(_.collection[JSONCollection]("passenger-registrations"))
    
      def savePassengerRegistration = Action.async(parse.json) { request =>
        Json.fromJson[PassengerRegistration](request.body) match {
          case JsSuccess(newPassRegistration, _) =>
    
            //https://github.com/ReactiveMongo/ReactiveMongo-Extensions/blob/0.10.x/guide/dsl.md
            val query = Json.obj("email" -> Json.obj("$eq" -> newPassRegistration.email))
    
            dealWithRegistration[PassengerRegistration](
              newPassRegistration,
              passRegistrationFuture,
              query,
              PassengerRegistration.formatter)
          case JsError(errors) =>
            Future.successful(BadRequest(
    			"Could not build a PassengerRegistration from the json provided. " +
    			Errors.show(errors)))
        }
      }
    
      private def dealWithRegistration[T](
              incomingRegistration: T,
              jsonCollectionFuture: Future[JSONCollection],
              query: JsObject,
              formatter: OFormat[T])
              (implicit ec: ExecutionContext): Future[Result] = {
    
        def hasExistingRegistrationFuture = jsonCollectionFuture.flatMap {
            //http://reactivemongo.org/releases/0.11/documentation/advanced-topics/collection-api.html
            _.find(query)
            .cursor[JsObject](ReadPreference.primary)
            .collect[List]()
          }.map(_.length match {
              case 0 => false
              case _ => true
          }
        )
    
        hasExistingRegistrationFuture.flatMap {
          case false => {
            for {
              registrations <- jsonCollectionFuture
              writeResult <- registrations.insert(incomingRegistration)(formatter,ec)
            } yield {
              Logger.debug(s"Successfully inserted with LastError: $writeResult")
              Ok(Json.obj())
            }
          }
          case true => Future(BadRequest("Registration already exists"))
        }
      }
    }
    

     

    Lets break this down into chunks

     

    • The controller constructor
      • This takes a ReactiveMongoApi (this is mandatory to satisfy the base trait MongoController requirements)
      • Inherits from MongoController which provides a lot of use functionality
      • It also inherits from ReactiveMongoComponents in order to allow the cake pattern/self typing requirements of the base MongoController  which expects a ReactiveMongoComponents
    • The use of JSONCollection
      • There is a Future[JSONCollection] that represents the passenger collection in Mongo. This is a collection that stores JSON. When using reactive Mongo you have a choice about whether to use the standard BSON collections of JSON. I opted for JSON
    • The Guts Of The Logic
      • So now we have discussed the controller constructor and the Mongo collections. We just need to talk about the actual work that happens on registration. In a nutshell it works like this
      • The incoming JSON string is turned into a PassengerRegistration object via Play
      • We then create a new JSON query object to query the Mongo JSONCollection to see if a registration already exists
      • If a registration already exists we exit with a BadRequest output
      • If a registration does NOT already exist we insert the new registration details into the Mongo JSONCollection, and the we return an Ok output

     

    And that is how the “Passenger Registration” works.

     

     

    Driver Registration

    The driver registration works in much the same way as described above, its just slightly different JSON, but it does share the same core logic/controller as the “Passenger Registration”

     

     

    Login

    Although the Login share some of the same ideas as Registration it has to do slightly different work, we will talk about what is different along the way, but this is what the Login screen looks like

    image

     

    And this is what the relevant code on the client looks like to send across the JSON payload.

    _handleValidSubmit = (values) => {
        var logindetails = values;
        var self = this;
    
        $.ajax({
            type: 'POST',
            url: 'login/validate',
            data: JSON.stringify(logindetails),
            contentType: "application/json; charset=utf-8",
            dataType: 'json'
        })
        .done(function (jdata, textStatus, jqXHR) {
    
            console.log("result of login");
            console.log(jqXHR.responseText);
            let currentUser = jqXHR.responseText;
            self._authService.storeUser(currentUser);
    
            self.setState(
                {
                    okDialogHeaderText: 'Login Successful',
                    okDialogBodyText: 'You are now logged in',
                    okDialogOpen: true,
                    okDialogKey: Math.random()
                });
        })
        .fail(function (jqXHR, textStatus, errorThrown) {
            self.setState(
                {
                    okDialogHeaderText: 'Error',
                    okDialogBodyText: jqXHR.responseText,
                    okDialogOpen: true,
                    okDialogKey: Math.random()
                });
        });
    }
    

     

    What is different this time is that when the Login is successful we make use of an injected AuthService which we push out a true or false to indicate login status on a Rx subject, which all the other screens/components listen to, where each component can changes its own state/rendering depending on the login status. Here is the full code for the AuthService

     

    import { injectable, inject } from "inversify";
    import { TYPES } from "../types";
    import Rx from 'rx';  
    
    @injectable()
    export class AuthService {
    
        private _isAuthenticated: boolean;
        private _authenticatedSubject = new Rx.Subject<boolean>();
    
        constructor() {
    
        }
    
        clearUser = () => {
            this._isAuthenticated = false;
            sessionStorage.removeItem('currentUserProfile');
            this._authenticatedSubject.onNext(false);
        }
    
        storeUser = (currentUser) => {
            this._isAuthenticated = true;
            sessionStorage.setItem('currentUserProfile', currentUser);
            this._authenticatedSubject.onNext(true);
        }
    
        userName = () => {
            var user = JSON.parse(sessionStorage.getItem('currentUserProfile'));
            return user.fullName;
        }
    
        isAuthenticated = () => {
            return this._isAuthenticated;
        }
    
        getAuthenticationStream = () => {
            return this._authenticatedSubject.asObservable();
        }
    }
    

     

     

    And here is example of how a React component may listen to this Rx subject to affect its own rendering

     

    import * as React from "react";
    import * as ReactDOM from "react-dom";
    
    ....
    ....
    
    let authService = ContainerOperations.getInstance().container.get<AuthService>(TYPES.AuthService);
    
    export interface MainNavProps {
        authService: AuthService;
    }
    
    export interface MainNavState {
        isLoggedIn: boolean;
    }
    
    class MainNav extends React.Component<MainNavProps, MainNavState> {
    
        private _subscription: any;
    
    
        constructor(props: any) {
            super(props);
            console.log(props);
            this.state = {
                isLoggedIn: false
            };
        }
    
        componentWillMount() {
            this._subscription = this.props.authService.getAuthenticationStream().subscribe(isAuthenticated => {
                this.state = {
                    isLoggedIn: isAuthenticated
                };
                if (this.state.isLoggedIn) {
                    hashHistory.push('/createjob');
                }
                else {
                    hashHistory.push('/');
                }
            });
        }
    
        componentWillUnmount() {
            this._subscription.dispose();
        }
    
        render() {
           ....
           ....
        }
    }
    
    
    

    Now moving on to the server side Play API code, as before we need a Play backend route for the Login

     

    # Login page

    POST  /login/validate                          controllers.LoginController.validateLogin()

     

    And as before we also have a Reactive Mongo enabled Play controller. I won’t go over the common stuff again, but here is the guts of the LoginController, and shown below is a bullet point list of what it does

     

    package controllers
    
    import javax.inject.Inject
    
    import Entities.DriverRegistrationJsonFormatters._
    import Entities.PassengerRegistrationJsonFormatters._
    import Entities._
    import play.api.Logger
    import play.api.libs.json._
    import play.api.mvc.{Action, Controller, Result}
    import play.modules.reactivemongo._
    import reactivemongo.api.ReadPreference
    import reactivemongo.play.json._
    import reactivemongo.play.json.collection._
    import utils.Errors
    
    
    import scala.concurrent.{ExecutionContext, Future}
    
    class LoginController @Inject()(val reactiveMongoApi: ReactiveMongoApi)
                                   (implicit ec: ExecutionContext)
      extends Controller with MongoController with ReactiveMongoComponents {
    
      def passRegistrationFuture: Future[JSONCollection] = database.map(_.collection[JSONCollection]("passenger-registrations"))
      def driverRegistrationFuture: Future[JSONCollection] = database.map(_.collection[JSONCollection]("driver-registrations"))
    
    
      def validateLogin = Action.async(parse.json) { request =>
        Json.fromJson[Login](request.body) match {
          case JsSuccess(newLoginDetails, _) =>
            newLoginDetails.isDriver match {
              case false => {
                val maybePassengerReg = extractExistingRegistration(
                  passRegistrationFuture.flatMap {
                    _.find(Json.obj("email" -> Json.obj("$eq" -> newLoginDetails.email))).
                      cursor[JsObject](ReadPreference.primary).
                      collect[List]()
                  })
                returnRedactedRegistration[PassengerRegistration](
                  maybePassengerReg,
                  (reg: PassengerRegistration) => Ok(Json.toJson(reg.copy(password = "")))
                )
    
              }
              case true => {
                val maybeDriverReg = extractExistingRegistration(
                  driverRegistrationFuture.flatMap {
                    _.find(Json.obj("email" -> Json.obj("$eq" -> newLoginDetails.email))).
                      cursor[JsObject](ReadPreference.primary).
                      collect[List]()
                  })
                returnRedactedRegistration[DriverRegistration](
                  maybeDriverReg,
                  (reg: DriverRegistration) => Ok(Json.toJson(reg.copy(password = "")))
                )
              }
            }
          case JsError(errors) =>
            Future.successful(BadRequest("Could not build a Login from the json provided. " +
              Errors.show(errors)))
        }
      }
    
      private def returnRedactedRegistration[T]
      (
        maybeDriverRegFuture: Future[Option[JsObject]],
        redactor : T => Result
      )(implicit reads: Reads[T]): Future[Result] = {
        maybeDriverRegFuture.map {
          case Some(json) => {
            val reg = Json.fromJson[T](json)
            reg match {
              case JsSuccess(reg, _) => {
                redactor(reg)
              }
              case _ => BadRequest("Registration already exists")
            }
          }
          case None => BadRequest("Could not find registration")
        }
      }
    
      private def extractExistingRegistration[T]
      (incomingRegistrations: Future[List[T]])
      (implicit writes: Writes[T], ec: ExecutionContext): Future[Option[T]] = {
        incomingRegistrations.map(matchedRegistrations =>
          matchedRegistrations.length match {
            case 0 => None
            case _ => Some(matchedRegistrations(0))
          }
        )
      }
    }
    

     

    Essentially the code above does the following:

     

    • Takes the deserialized JSON Login information, and decides whether it’s a Driver or a Passenger that is trying to login based on the IsDriver boolean.
    • Once we know if it’s a Driver or a Passenger that we are dealing with we continue to check if there is a registration for someone that has the login email
    • If we find a registered Driver or Passenger we obtain the registration that was previously stored and redact the password, and then convert it to JSON and send it back over the wire to the React code where it is stored in Local Storage and exposes out to the JS code using the AuthService.ts code above, and will notify any Rx subscribers of the login status, such that any listening React components can adjust their state/render information

     

     

    Conclusion

    I hope you have enjoyed this post, I know I have had fun with this one, I like the Reactive Mongo stuff, and I like how its all Future based which makes for a nice non-blocking workflow. We are now past the 1st phase of this project, the next phase will be to start with the Kafka messaging, where I may end up doing some more experimental/self contained code, rather than trying to shoe horn things into the main app in one go. I will still get it into the main app, but I am just thinking the Kafka posts may be better as stand alone things. I’ll see how it goes, at any rate the Kafka Streams stuff is phase 2, and we will be looking at that next

    MADCAP IDEA PART 6 : STATIC SCREEN DESIGN

    Last Time

     

    Last time we looked at setting up the react router, which used some dummy screens. This time we will actually be implementing the static screen designs. The screens will be pretty much what the end application will look like, with the exception that they are not wired up to the Play framework backend, and for now might use dummy data to show what they would look like for real, when the entire app is completed and working

     

    PreAmble

    Just as a reminder this is part of my ongoing set of posts which I talk about here :

    https://sachabarbs.wordpress.com/2017/05/01/madcap-idea/, where we will be building up to a point where we have a full app using lots of different stuff, such as these

     

    • WebPack
    • React.js
    • React Router
    • TypeScript
    • Babel.js
    • Akka
    • Scala
    • Play (Scala Http Stack)
    • MySql
    • SBT
    • Kafka
    • Kafka Streams

     

    So I think the best way to cover this post is to do it screen by screen, so lets start at the beginning and end at the end.

     

    The Screens

    In this section we will simply go through each of the screens and examine the code and the final rendering and discuss and important code that might be contained within the screen being discussed

     

    Login

    This is the entire code for the login screen

    import * as React from "react";
    import * as ReactDOM from "react-dom";
    
    import 'bootstrap/dist/css/bootstrap.css';
    import
    {
        Well,
        Grid,
        Row,
        Col,
        ButtonInput
    } from "react-bootstrap";
    
    import { Form, ValidatedInput } from 'react-bootstrap-validation';
    
    import revalidator from 'revalidator';
    
    
    let schema = {
        properties: {
            email: {
                type: 'string',
                maxLength: 255,
                format: 'email',
                required: true,
                allowEmpty: false
            },
            password: {
                type: 'string',
                minLength: 8,
                maxLength: 60,
                required: true,
                allowEmpty: false
            }
        }
    };
    
    
    
    export class Login extends React.Component<undefined, undefined> {
        render() {
            return (
                <Well className="outer-well">
                    <Form
                        // Supply callbacks to both valid and invalid
                        // submit attempts
                        validateAll={this._validateForm}
                        onInvalidSubmit={this._handleInvalidSubmit}
                        onValidSubmit={this._handleValidSubmit}>
                        <Grid>
                            <Row className="show-grid">
                                <Col xs={10} md={6}>
                                    <h4>ENTER YOUR LOGIN DETAILS</h4>
                                </Col>
                            </Row>
                            <Row className="show-grid">
                                <Col xs={10} md={6}>
                                    <ValidatedInput type='text'
                                        label='Email'
                                        name='email'
                                        errorHelp='Email address is invalid'/>
    
                                </Col>
                            </Row>
                            <Row className="show-grid">
                                <Col xs={10} md={6}>
                                    <ValidatedInput type='password'
                                        name='password'
                                        label='Password'
                                        errorHelp='Password is invalid'/>
    
                                </Col>
                            </Row>
                            <Row className="show-grid">
                                <Col xs={10} md={6}>
                                    <ButtonInput
                                        id="loginBtn"
                                        type='submit'
                                        bsSize='small'
                                        bsStyle='primary'
                                        value='Register'>Login</ButtonInput>
                                </Col>
                            </Row>
                        </Grid>
                    </Form>
                </Well>
            )
        }
    
    
        _validateForm = (values) => {
            let res = revalidator.validate(values, schema);
    
            // If the values passed validation, we return true
            if (res.valid) {
                return true;
            }
    
            // Otherwise we should return an object containing errors
            // e.g. { email: true, password: true }
            return res.errors.reduce((errors, error) => {
                // Set each property to either true or
                // a string error description
                errors[error.property] = true;
    
                return errors;
            }, {});
        }
    
        _handleInvalidSubmit = (errors, values) => {
            // Errors is an array containing input names
            // that failed to validate
            alert("Form has errors and may not be submitted");
        }
    
        _handleValidSubmit = (values) => {
            // Values is an object containing all values
            // from the inputs
            console.log("Form may be submitted");
            console.log(values);
        }
    }
    

    Where this will look like this in a browser

    image

     

    There are a couple of things to talk about on this screen.

     

    Firstly there is the use of React-bootstrap components, which are all imported at the top of the file like this

     

    import 'bootstrap/dist/css/bootstrap.css';
    import
    {
        Well,
        Grid,
        Row,
        Col,
        ButtonInput
    } from "react-bootstrap";
    

     

    This allows us to import the actual React-bootstrap components and the CSS for bootstrap. React-bootstrap doesn’t really mind which CSS it uses, as long as it is a version of regular bootstrap CSS. I am using bootstrap v3.0 CSS here. React-bootstrap is more about the actual markup, and doing things in a more React like manner, to reduce some of the regular Bootstrap markup, which is possible thanks to using React.

     

    The other important part about this screen is that it does client side validation. This is done using this library : react-bootstrap-validation. There are a few React validation libraries out there, but I like the simplicity and elegance of this one a lot. You really only have to do a couple of thing (as can be seen in full in the code above).

     

    You need to import the library

     

    import { Form, ValidatedInput } from 'react-bootstrap-validation';
    
    import revalidator from 'revalidator';
    

     

    Then you need a schema describing your form that you are trying to validate

    let schema = {
        properties: {
            email: {
                type: 'string',
                maxLength: 255,
                format: 'email',
                required: true,
                allowEmpty: false
            },
            password: {
                type: 'string',
                minLength: 8,
                maxLength: 60,
                required: true,
                allowEmpty: false
            }
        }
    };
    

     

    And then you need to create the actual form (here is a stripped down one that matches the schema above)

     

    <Form>
    	<ValidatedInput type='text'
    		label='Email'
    		name='email'
    		errorHelp='Email address is invalid'/>
    	<ValidatedInput type='password'
    		name='password'
    		label='Password'
    		errorHelp='Password is invalid'/>
    </Form>
    

     

    And we must also supply some validation callback hooks for the form, which are done like this

     

    <Form
        // Supply callbacks to both valid and invalid
        // submit attempts
        validateAll={this._validateForm}
        onInvalidSubmit={this._handleInvalidSubmit}
        onValidSubmit={this._handleValidSubmit}>
    
    </Form>
    

     

    Where the code for these callbacks looks like this

     

    export class Login extends React.Component<undefined, undefined> {
        render() {
            return (
              .....
            )
        }
    
    
        _validateForm = (values) => {
            let res = revalidator.validate(values, schema);
    
            // If the values passed validation, we return true
            if (res.valid) {
                return true;
            }
    
            // Otherwise we should return an object containing errors
            // e.g. { email: true, password: true }
            return res.errors.reduce((errors, error) => {
                // Set each property to either true or
                // a string error description
                errors[error.property] = true;
    
                return errors;
            }, {});
        }
    
        _handleInvalidSubmit = (errors, values) => {
            // Errors is an array containing input names
            // that failed to validate
            alert("Form has errors and may not be submitted");
        }
    
        _handleValidSubmit = (values) => {
            // Values is an object containing all values
            // from the inputs
            console.log("Form may be submitted");
            console.log(values);
        }
    }
    

    So that pretty much explains how the validation works, what does it look like when run? Well it looks like this when its not valid

     

    image

     

    And that is pretty much the static design of the Login screen, lets move on

     

     

    Register

    The Register screen uses the same validation stuff as above, so I will not cover it again. The only thing of note in the Register screen is that it will either need to render a Passenger registration or a Driver registration depending on what option was chosen by the user.

     

    To do this conditional rendering, I opted for a set of 2 simple buttons that modify the internal state of the main Register React component, and will then either display a PassengerRegistration component of a DriverRegistration component.

     

    Here is the full code for the main Register React component

     

    import * as React from "react";
    import * as ReactDOM from "react-dom";
    
    import 'bootstrap/dist/css/bootstrap.css';
    import
    {
        Well,
        Grid,
        Row,
        Col,
        ButtonInput,
        ButtonGroup,
        Button
    } from "react-bootstrap";
    
    import { PassengerRegistration } from "./PassengerRegistration";
    import { DriverRegistration } from "./DriverRegistration";
    
    
    export interface RegisterState {
        option: string;
    }
    
    
    export class Register extends React.Component<any, RegisterState> {
    
        constructor(props: any) {
            super(props);
            this.state = { option: "passenger" };
        }
    
        render() {
            return (
                <Well className="outer-well">
                    <Grid>
                        <Row className="show-grid">
                            <Col xs={10} md={6}>
                                <h4>PLEASE ENTER YOUR REGISTRATION DETAILS</h4>
                            </Col>
                        </Row>
                        <Row className="show-grid">
                            <Col xs={10} md={6}>
                                <h6>Choose your registration type </h6>
                            </Col>
                        </Row>
                        <Row className="show-grid">
                            <Col xs={10} md={6}>
                                <ButtonGroup>
                                    <Button bsSize='small' onClick={this._onOptionChange.bind(this, 'passenger') } active={this.state.option === 'passenger'}>Passenger</Button>
                                    <Button bsSize='small' onClick={this._onOptionChange.bind(this, 'driver') } active={this.state.option === 'driver'}>Driver</Button>
                                </ButtonGroup>
                            </Col>
                        </Row>
                        <Row className="show-grid">
                            <Col xs={10} md={6}>
                                {this.state.option === 'passenger' ?
                                    <div><PassengerRegistration/></div> :
                                    <div><DriverRegistration/></div>
                                }
                            </Col>
                        </Row>
                    </Grid>
                </Well>
            )
        }
    
        _onOptionChange(option) {
            this.setState({
                option: option
            });
        }
    
    }
    

     

    As you can see this code contains the state that is set to determine which type of sub registration component to actually render (see the Render() method above)

     

    The sub components are as shown below (as stated above we already discussed validation, so I will not cover that again).

     

    PassengerRegistration

    Here is the full code for this sub component

    import * as React from "react";
    import * as ReactDOM from "react-dom";
    
    import 'bootstrap/dist/css/bootstrap.css';
    import
    {
        Well,
        Grid,
        Row,
        Col,
        ButtonInput
    } from "react-bootstrap";
    
    import { Form, ValidatedInput } from 'react-bootstrap-validation';
    
    import revalidator from 'revalidator';
    
    
    let schema = {
        properties: {
            fullname: {
                type: 'string',
                minLength: 8,
                maxLength: 12,
                required: true,
                allowEmpty: false
            },
            email: {
                type: 'string',
                maxLength: 255,
                format: 'email',
                required: true,
                allowEmpty: false
            },
            password: {
                type: 'string',
                minLength: 8,
                maxLength: 60,
                required: true,
                allowEmpty: false
            }
        }
    };
    
    
    
    export class PassengerRegistration extends React.Component<undefined, undefined> {
        render() {
            return (
                <Form className="submittable-form-inner"
                    // Supply callbacks to both valid and invalid
                    // submit attempts
                    validateAll={this._validateForm}
                    onInvalidSubmit={this._handleInvalidSubmit}
                    onValidSubmit={this._handleValidSubmit}>
                    <Grid>
                        <Row className="show-grid">
                            <Col xs={10} md={6}>
                                <h4>Passenger details</h4>
                            </Col>
                        </Row>
                        <Row className="show-grid">
                            <Col xs={10} md={6}>
                                <ValidatedInput type='text'
                                    label='FullName'
                                    name='fullname'
                                    errorHelp='FullName is invalid'/>
    
                            </Col>
                        </Row>
                        <Row className="show-grid">
                            <Col xs={10} md={6}>
                                <ValidatedInput type='text'
                                    label='Email'
                                    name='email'
                                    errorHelp='Email address is invalid'/>
                            </Col>
                        </Row>
                        <Row className="show-grid">
                            <Col xs={10} md={6}>
                                <ValidatedInput type='password'
                                    label='Password'
                                    name='password'
                                    errorHelp='Password is invalid'/>
                            </Col>
                        </Row>
                        <Row className="show-grid">
                            <Col xs={10} md={6}>
                                <ButtonInput
                                    id="registerBtn"
                                    type='submit'
                                    bsSize='small'
                                    bsStyle='primary'
                                    value='Register'>Register</ButtonInput>
                            </Col>
                        </Row>
                    </Grid>
                </Form>
            )
        }
    
        _validateForm = (values) => {
            let res = revalidator.validate(values, schema);
    
            // If the values passed validation, we return true
            if (res.valid) {
                return true;
            }
    
            // Otherwise we should return an object containing errors
            // e.g. { email: true, password: true }
            return res.errors.reduce((errors, error) => {
                // Set each property to either true or
                // a string error description
                errors[error.property] = true;
    
                return errors;
            }, {});
        }
    
        _handleInvalidSubmit = (errors, values) => {
            // Errors is an array containing input names
            // that failed to validate
            alert("Form has errors and may not be submitted");
        }
    
        _handleValidSubmit = (values) => {
            // Values is an object containing all values
            // from the inputs
            console.log("Form may be submitted");
            console.log(values);
        }
    }
    
    
    

     

    DriverRegistration

    Here is the full code for this sub component

    import * as React from "react";
    import * as ReactDOM from "react-dom";
    
    import 'bootstrap/dist/css/bootstrap.css';
    import
    {
        Well,
        Grid,
        Row,
        Col,
        ButtonInput
    } from "react-bootstrap";
    
    import { Form, ValidatedInput } from 'react-bootstrap-validation';
    
    import revalidator from 'revalidator';
    
    
    let schema = {
        properties: {
            fullname: {
                type: 'string',
                minLength: 8,
                maxLength: 12,
                required: true,
                allowEmpty: false
            },
            email: {
                type: 'string',
                maxLength: 255,
                format: 'email',
                required: true,
                allowEmpty: false
            },
            password: {
                type: 'string',
                minLength: 8,
                maxLength: 60,
                required: true,
                allowEmpty: false
            },
            vehicleDescription: {
                type: 'string',
                minLength: 6,
                maxLength: 60,
                required: true,
                allowEmpty: false
            },
            vehicleRegistrationNumber: {
                type: 'string',
                minLength: 6,
                maxLength: 30,
                required: true,
                allowEmpty: false
            }
        }
    };
    
    
    
    export class DriverRegistration extends React.Component<undefined, undefined> {
        render() {
            return (
                <Form className="submittable-form-inner"
                    // Supply callbacks to both valid and invalid
                    // submit attempts
                    validateAll={this._validateForm}
                    onInvalidSubmit={this._handleInvalidSubmit}
                    onValidSubmit={this._handleValidSubmit}>
                    <Grid>
                        <Row className="show-grid">
                            <Col xs={10} md={6}>
                                <h4>Driver details</h4>
                            </Col>
                        </Row>
                        <Row className="show-grid">
                            <Col xs={10} md={6}>
                                <ValidatedInput type='text'
                                    label='FullName'
                                    name='fullname'
                                    errorHelp='FullName is invalid'/>
    
                            </Col>
                        </Row>
                        <Row className="show-grid">
                            <Col xs={10} md={6}>
                                <ValidatedInput type='text'
                                    label='Email'
                                    name='email'
                                    errorHelp='Email address is invalid'/>
    
                            </Col>
                        </Row>
                        <Row className="show-grid">
                            <Col xs={10} md={6}>
                                <ValidatedInput type='password'
                                    name='password'
                                    label='Password'
                                    errorHelp='Password is invalid'/>
                            </Col>
                        </Row>
                        <Row className="show-grid">
                            <Col xs={10} md={6}>
                                <h4>Vehicle details</h4>
                            </Col>
                        </Row>
                        <Row className="show-grid">
                            <Col xs={10} md={6}>
                                <ValidatedInput type='text'
                                    label='Vehicle Description'
                                    name='vehicleDescription'
                                    errorHelp='Vehicle description is invalid'/>
                            </Col>
                        </Row>
                        <Row className="show-grid">
                            <Col xs={10} md={6}>
                                <ValidatedInput type='text'
                                    label='Vehicle Registration Number'
                                    name='vehicleRegistrationNumber'
                                    errorHelp='Vehicle registration number is invalid'/>
                            </Col>
                        </Row>
    
                        <Row className="show-grid">
                            <Col xs={10} md={6}>
                                <ButtonInput
                                    id="registerBtn"
                                    type='submit'
                                    bsSize='small'
                                    bsStyle='primary'
                                    value='Register'>Register</ButtonInput>
                            </Col>
                        </Row>
                    </Grid>
                </Form>
            )
        }
    
    
        _validateForm = (values) => {
            let res = revalidator.validate(values, schema);
    
            // If the values passed validation, we return true
            if (res.valid) {
                return true;
            }
    
            // Otherwise we should return an object containing errors
            // e.g. { email: true, password: true }
            return res.errors.reduce((errors, error) => {
                // Set each property to either true or
                // a string error description
                errors[error.property] = true;
    
                return errors;
            }, {});
        }
    
        _handleInvalidSubmit = (errors, values) => {
            // Errors is an array containing input names
            // that failed to validate
            alert("Form has errors and may not be submitted");
        }
    
        _handleValidSubmit = (values) => {
            // Values is an object containing all values
            // from the inputs
            console.log("Form may be submitted");
            console.log(values);
        }
    
    }
    
    
    

    And this is the end result, with the Passenger registration option selected.

     

    image

     

     

    And this is the driver registration

    image

     

     

    CreateJob

    Ok so now things are getting slightly more exotic we need to use a Google map along with React. There are many good tutorials that walk you through this, which are great to read so you can see how the Google maps API gets wrapped (typically this is what is done) and you can also see how to use the React Ref attribute to grab the ACTUAL element from the real DOM to host the map. There are also many available react google maps components. I had a hunt around and tried out a few and settled on this one : https://github.com/tomchentw/react-google-maps where there are some great demos here : https://tomchentw.github.io/react-google-maps/

     

    The job of the “CreateJob” screen for me was to allow the user to set their current position on a map, and issue the job out to drivers that may be logged in at that time.

     

    Remember I will be doing all this work on my laptop. So that is my laptop pretending to be many drivers/users etc etc, so I cant really use the inbuilt GPS position, everyone would always be at the same place, ie where ever my laptop is at that moment.

     

    Ok so now we know what the job of the “CreateJob” screen is lets have a look at some of the code.

     

    For the React-Google-Maps we need this import, otherwise its just stuff you have seen before

     

    import { withGoogleMap, GoogleMap, Marker, InfoBox } from "react-google-maps";
    

     

    The next part of interest is how we create an actual map which might have a marker on the map with a custom icon. Thanks to the great work of the React-Google-Maps code, this is as simple as this

     

    const CreateJobGoogleMap = withGoogleMap(props => (
        <GoogleMap
            ref={props.onMapLoad}
            defaultZoom={16}
            defaultCenter={{ lat: 50.8202949, lng: -0.1406958 }}
            onClick={props.onMapClick}>
            <Marker
                position={props.currentPosition}
                icon='/assets/images/passenger.png'>
            </Marker>
        </GoogleMap>
    ));
    

     

    It can be seen there that the Marker is using the position that is passed down as props to it from the parent React component. Lets now have a look at the parent React component

     

    export interface CreateJobState {
        currentPosition: any;
    }
    
    export class CreateJob extends React.Component<undefined, CreateJobState> {
    
        constructor(props: any) {
            super(props);
            this.state = {
                currentPosition: { lat: 50.8202949, lng: -0.1406958 }
              };
        }
    
    
        render() {
            return (
                <Well className="outer-well">
                    <Grid>
                        <Row className="show-grid">
                            <Col xs={10} md={6}>
                                <h4>SET YOUR CURRENT LOCATION</h4>
                                <h6>Click the map to set your current location</h6>
                            </Col>
                        </Row>
                        <Row className="show-grid">
                            <Col xs={10} md={6}>
                                <CreateJobGoogleMap
                                    containerElement={
                                        <div style={{
                                            position: 'relative',
                                            top: 0,
                                            left: 0,
                                            right: 0,
                                            bottom: 0,
                                            width: 600,
                                            height: 600,
                                            justifyContent: 'flex-end',
                                            alignItems: 'center',
                                            marginTop: 20,
                                            marginLeft: 0,
                                            marginRight: 0,
                                            marginBottom: 20
                                        }} />
                                    }
                                    mapElement={
                                        <div style={{
                                            position: 'relative',
                                            top: 0,
                                            left: 0,
                                            right: 0,
                                            bottom: 0,
                                            width: 600,
                                            height: 600,
                                            marginTop: 20,
                                            marginLeft: 0,
                                            marginRight: 0,
                                            marginBottom: 20
                                        }} />
                                    }
                                    onMapLoad={this._handleMapLoad}
                                    onMapClick={this._handleMapClick}
                                    currentPosition={this.state.currentPosition}
                                    />
                            </Col>
                        </Row>
                        <Row className="show-grid">
                            <ButtonInput
                                id="createJobBtn"
                                type='submit'
                                bsSize='small'
                                bsStyle='primary'
                                value='Register'>Create Job</ButtonInput>
                        </Row>
                    </Grid>
                </Well>
            );
        }
    
        _handleMapLoad = (map) => {
            if (map) {
                console.log(map.getZoom());
            }
        }
    
        _handleMapClick = (event) => {
            this.setState({
                currentPosition: event.latLng
            });
        }
    }
    

     

    If you cut through all the bootstrap code to render things nicely it really boils down to what happens when the map gets clicked?

     

    Well what happens is that the map getting clicked will fire the _handleMapClick function, which will set the internal currentPosition state value, which was handed down to the map sub component as props, so the Marker will be shown at the clicked location.

     

    This is what the final rendering looks like

     

    image

     

    ViewJob

    This is by far the most complex of all the screens, and here is why

    • It will have to show the incoming movements (geo-coords wise) of both driver(s) and passenger (icon distinguishes them from each other, as well as metadata)
    • A passenger should be able to accept a driver for a job
    • It will need to allow a job to be cancelled, with confirmation
    • It will need to allow a job to be completed
    • On completion of a job the driver should be able to rate a passenger
    • On completion of a job the passenger should be able to rate a driver

     

    So as you can see it WILL do quite a lot, obviously a lot of this will come later once the streaming aspects of the code are completed. It is however quite a lot to do statically.

     

    In fact it is so much that I have had to create 3 utility components to help me out

     

    YesNoDialog

    This represents a generic re-usable yes/no dialog that can be triggered, here is the code for this one. The important part is that the various prop values can be controlled via the parent component state values

     

    import * as React from "react";
    import * as ReactDOM from "react-dom";
    import * as _ from "lodash";
    
    import 'bootstrap/dist/css/bootstrap.css';
    import
    {
        Button, 
        Modal
    } from "react-bootstrap";
    
    
    //TODO : Fix this
    export interface YesNoDialogProps {
        headerText: string;
        theId: string;
        launchButtonText: string;
        yesCallBack: any;
        noCallBack: any;
    }
    
    export interface YesNoDialogState {
        showModal: boolean;
    }
    
    
    export class YesNoDialog extends React.Component<YesNoDialogProps, YesNoDialogState> {
    
        constructor(props) {
            super(props);
            console.log(this.props);
            //set initial state
            this.state = {
                showModal: false
            };
        }
    
        _yesClicked = () => {
            this.setState({ showModal: false });
            this.props.yesCallBack();
        }
    
        _noClicked = () => {
            this.setState({ showModal: false });
            this.props.noCallBack();
        }
    
        _close = () => {
            this.setState({ showModal: false });
        }
    
        _open = () => {
            this.setState({ showModal: true });
        }
    
        render() {
            return (
                <div className="leftFloat">
    
                    <Button
                        id={this.props.theId}
                        type='button'
                        bsSize='small'
                        bsStyle='primary'
                        onClick={this._open}>{this.props.launchButtonText}</Button>
    
                    <Modal show={this.state.showModal} onHide={this._close}>
                        <Modal.Header closeButton>
                            <Modal.Title>{ this.props.headerText }</Modal.Title>
                        </Modal.Header>
                        <Modal.Body>
                            <h4>Are you sure?</h4>
                        </Modal.Body>
                        <Modal.Footer>
                            <Button
                                type='button'
                                bsSize='small'
                                bsStyle='primary'
                                onClick={this._yesClicked}>Yes</Button>
                            <Button
                                type='button'
                                bsSize='small'
                                bsStyle='danger'
                                onClick={this._noClicked}>Cancel</Button>
                        </Modal.Footer>
                    </Modal>
                </div>
            );
        }
    }
    

    This looks like this when rendered

    image

     

     

    OkDialog

    This represents a generic re-usable ok dialog that can be triggered, here is the code for this one. The important part is that the various prop values can be controlled via the parent component state values

    import * as React from "react";
    import * as ReactDOM from "react-dom";
    import * as _ from "lodash";
    
    import 'bootstrap/dist/css/bootstrap.css';
    import
    {
        Button, 
        Modal
    } from "react-bootstrap";
    
    
    //TODO : Fix this
    export interface OkDialogProps {
        headerText: string;
        bodyText: string;
        open: boolean;
        okCallBack: any;
    }
    
    export interface OkDialogState {
        showModal: boolean;
    }
    
    
    export class OkDialog extends React.Component<OkDialogProps, OkDialogState> {
    
        constructor(props) {
            super(props);
            console.log(this.props);
            //set initial state
            this.state = {
                showModal: false
            };
        }
    
        componentDidMount() {
            if (this.props.open === true) {
                this.setState({ showModal: true });
            }
        }
    
        _okClicked = () => {
            this.setState({ showModal: false });
            this.props.okCallBack();
        }
    
        _close = () => {
            this.setState({ showModal: false });
            this.props.okCallBack();
        }
    
        _open = () => {
            this.setState({ showModal: true });
        }
    
        render() {
            return (
                <div className="leftFloat">
    
                    <Modal show={this.state.showModal} onHide={this._close}>
                        <Modal.Header closeButton>
                            <Modal.Title>{ this.props.headerText }</Modal.Title>
                        </Modal.Header>
                        <Modal.Body>
                            <h4>{this.props.bodyText}</h4>
                        </Modal.Body>
                        <Modal.Footer>
                            <Button
                                type='button'
                                bsSize='small'
                                bsStyle='primary'
                                onClick={this._okClicked}>Ok</Button>
                        </Modal.Footer>
                    </Modal>
                </div>
            );
        }
    }
    

    This looks like this when rendered

     

    image

     

     

    RatingDialog

    This represents a generic re-usable rating control where rating can be from 1-5, here is the code for this one. The important part is that the various prop values can be controlled via the parent component state values

    import * as React from "react";
    import * as ReactDOM from "react-dom";
    import * as _ from "lodash";
    
    import 'bootstrap/dist/css/bootstrap.css';
    import
    {
        Button, 
        Modal
    } from "react-bootstrap";
    
    
    import ReactStars from 'react-stars';
    
    
    export interface RatingDialogProps {
        headerText: string;
        theId: string;
        okCallBack: any;
    }
    
    export interface RatingDialogState {
        showModal: boolean;
        rating: number;
    }
    
    
    export class RatingDialog extends React.Component<RatingDialogProps, RatingDialogState> {
    
        constructor(props) {
            super(props);
            console.log(this.props); 
            //set initial state
            this.state = {
                showModal: false,
                rating:0
            };
        }
    
        _close = () => {
            this.setState(
                {
                    showModal: false,
                    rating:0
                }
            );
        }
    
        _open = () => {
            this.setState(
                {
                    showModal: true,
                    rating: 0
                }
            );
        }
    
        _ratingChanged = (newRating) => {
            console.log(newRating)
            this.setState(
                {
                    rating: newRating
                }
            );
        }
    
        _okClicked = () => {
            this._close();
            this.props.okCallBack();
        }
    
        render() {
            return (
                <div className="leftFloat">
    
                    <Button
                        id={this.props.theId}
                        type='button'
                        bsSize='small'
                        bsStyle='primary'
                        onClick={this._open}>Complete</Button>
    
                    <Modal show={this.state.showModal} onHide={this._close}>
                        <Modal.Header closeButton>
                            <Modal.Title>{ this.props.headerText }</Modal.Title>
                        </Modal.Header>
                        <Modal.Body>
                            <h4>Give your rating between 1-5</h4>
                            <ReactStars count={5}
                                        onChange={this._ratingChanged}
                                        size={24}
                                        color2={'#ffd700'} />
                        </Modal.Body>
                        <Modal.Footer>
                            <Button
                                type='submit'
                                bsSize='small'
                                bsStyle='primary'
                                onClick={this._okClicked}>Ok</Button>
                        </Modal.Footer>
                    </Modal>
                </div>
            );
        }
    }
    

    This looks like this when rendered

     

    image

     

    For the rating component I make use of this React library : https://www.npmjs.com/package/react-stars

     

     

    So this just leaves the main component that represents the page, the full code is as follows:

     

    import * as React from "react";
    import * as ReactDOM from "react-dom";
    import * as _ from "lodash";
    
    import { RatingDialog } from "./components/RatingDialog";
    import { YesNoDialog } from "./components/YesNoDialog";
    import { OkDialog } from "./components/OkDialog";
    
    
    import 'bootstrap/dist/css/bootstrap.css';
    import
    {
        Well,
        Grid,
        Row,
        Col,
        ButtonInput,
        ButtonGroup,
        Button, 
        Modal,
        Popover,
        Tooltip,
        OverlayTrigger
    } from "react-bootstrap";
    
    import { withGoogleMap, GoogleMap, Marker, OverlayView } from "react-google-maps";
    
    const STYLES = {
        overlayView: {
            background: `white`,
            border: `1px solid #ccc`,
            padding: 15,
        }
    }
    
    
    const GetPixelPositionOffset = (width, height) => {
        return { x: -(width / 2), y: -(height / 2) };
    }
    
    
    
    const ViewJobGoogleMap = withGoogleMap(props => (
        <GoogleMap
            ref={props.onMapLoad}
            defaultZoom={14}
            defaultCenter={{ lat: 50.8202949, lng: -0.1406958 }}>
    
    
            {props.markers.map((marker, index) => (
                <OverlayView
                    key={marker.key}
                    mapPaneName={OverlayView.OVERLAY_MOUSE_TARGET}
                    position={marker.position}
                    getPixelPositionOffset={GetPixelPositionOffset}>
                    <div style={STYLES.overlayView}>
                        <img src={marker.icon} />
                        <strong>{marker.key}</strong>
                        <br/>
                        <Button
                            type='button'
                            bsSize='xsmall'
                            bsStyle='primary'
                            onClick={() => props.onMarkerClick(marker) }
                            value='Accept'>Accept</Button>
                    </div>
                </OverlayView>
            )) }
        </GoogleMap>
    ));
    
    export interface ViewJobState {
        markers: any;
        okDialogOpen: boolean;
        okDialogKey: any;
        okDialogHeaderText: string;
        okDialogBodyText: string;
    }
    
    export class ViewJob extends React.Component<undefined, ViewJobState> {
    
        constructor(props: any) {
            super(props);
            this.state = {
                markers: [{
                        position: {
                            lat: 50.8202949,
                            lng: -0.1406958
                        },
                        key: 'driver_1',
                        icon: '/assets/images/driver.png'
                    },
                    {
                        position: {
                            lat: 50.8128187,
                            lng: -0.1361418
                        },
                        key: 'driver_2',
                        icon: '/assets/images/driver.png'
                    }
                ],
                okDialogHeaderText: '',
                okDialogBodyText: '',
                okDialogOpen: false,
                okDialogKey:0
            };
        }
    
    
        render() {
            return (
                <Well className="outer-well">
                    <Grid>
                        <Row className="show-grid">
                            <Col xs={10} md={6}>
                                <h4>CURRENT JOB</h4>
                            </Col>
                        </Row>
                        <Row className="show-grid">
                            <Col xs={10} md={6}>
                                <ViewJobGoogleMap
                                    containerElement={
                                        <div style={{
                                            position: 'relative',
                                            top: 0,
                                            left: 0,
                                            right: 0,
                                            bottom: 0,
                                            width: 600,
                                            height: 600,
                                            justifyContent: 'flex-end',
                                            alignItems: 'center',
                                            marginTop: 20,
                                            marginLeft: 0,
                                            marginRight: 0,
                                            marginBottom: 20
                                        }} />
                                    }
                                    mapElement={
                                        <div style={{
                                            position: 'relative',
                                            top: 0,
                                            left: 0,
                                            right: 0,
                                            bottom: 0,
                                            width: 600,
                                            height: 600,
                                            marginTop: 20,
                                            marginLeft: 0,
                                            marginRight: 0,
                                            marginBottom: 20
                                        }} />
                                    }
                                    markers={this.state.markers}
                                    onMarkerClick={this._handleClick}/>
                            </Col>
                        </Row>
                        <Row className="show-grid">
                            <span>
                                <RatingDialog
                                    theId="viewJobCompleteBtn"
                                    headerText="Rate your driver/passenger"
                                    okCallBack= {this._ratingsDialogOkCallBack}/>
    
                                <YesNoDialog
                                    theId="viewJobCancelBtn"
                                    launchButtonText="Cancel"
                                    yesCallBack={this._jobCancelledCallBack}
                                    noCallBack={this._jobNotCancelledCallBack}
                                    headerText="Cancel the job"/>
    
                                <OkDialog
                                    open= {this.state.okDialogOpen}
                                    okCallBack= {this._okDialogCallBack}
                                    headerText={this.state.okDialogHeaderText}
                                    bodyText={this.state.okDialogBodyText}
                                    key={this.state.okDialogKey}/>
                            </span>
                        </Row>
                    </Grid>
                </Well>
            );
        }
    
        _handleClick = (targetMarker) => {
            console.log('button on overlay clicked:' + targetMarker.key);
        }
    
        _ratingsDialogOkCallBack = () => {
            console.log('RATINGS OK CLICKED');
            this.setState(
                {
                    okDialogHeaderText: 'Ratings',
                    okDialogBodyText: 'Rating successfully recorded',
                    okDialogOpen: true,
                    okDialogKey: Math.random()
                });
        }
    
        _jobCancelledCallBack = () => {
            console.log('YES CLICKED');
            this.setState(
                {
                    okDialogHeaderText: 'Job Cancellaton',
                    okDialogBodyText: 'Job successfully cancelled',
                    okDialogOpen: true,
                    okDialogKey: Math.random() 
                });
        }
    
        _jobNotCancelledCallBack = () => {
            console.log('NO CLICKED');
            this.setState(
                {
                    okDialogHeaderText: 'Job Cancellaton',
                    okDialogBodyText: 'Job remains open',
                    okDialogOpen: true,
                    okDialogKey: Math.random() 
                });
        }
    
        _okDialogCallBack = () => {
            console.log('OK on OkDialog CLICKED');
            this.setState(
                {
                    okDialogOpen: false
                });
        }
    }
    

     

    Which makes use of some of the React-Google-Maps goodness we saw before, but instead of Marker objects we are now using Overlay objects to represent the driver/passenger info, obviously we make use of an array of items this time, as many driver(s) might bid for a job. Also shown here is how you can create the dialogs on demand and have their content tailored to the specific scenario in hand.

     

    Right now when this component renders it looks like this

     

    image

     

    ViewRating

    The Rating screen is perhaps the easiest of all the screens it simply allows you view your overall rating and who made the rating for you so far. This would like be an active query over a Kafka streams KTable.

     

    Here is the full code for this screen, there is not much to say on this one

    import * as React from "react";
    import * as ReactDOM from "react-dom";
    
    import 'bootstrap/dist/css/bootstrap.css';
    import
    {
        Well,
        Grid,
        Row,
        Col,
        Label
    } from "react-bootstrap";
    
    
    export class ViewRating extends React.Component<undefined, undefined> {
        render() {
            return (
                <Well className="outer-well">
                        <Grid>
                            <Row className="show-grid">
                                <Col xs={6} md={6}>
                                    <div>
                                        <h4>YOUR RANKING <Label>4.2</Label></h4>
                                    </div>
                                </Col>
                            </Row>
                            <Row className="show-grid">
                                <Col xs={10} md={6}>
                                    <h6>The finer details of your ranking are shown below</h6>
                                </Col>
                            </Row>
                            <Row className="show-grid">
                                <Col xs={10} md={6}>
                                    <div className="table-responsive">
                                        <table className="table table-striped table-bordered table-condensed factTable">
                                            <thead>
                                                <tr>
                                                    <th>Ranked By</th>
                                                    <th>Rank Given</th>
                                                </tr>
                                            </thead>
                                            <tbody>
                                                <tr>
                                                    <td>John Doe</td>
                                                    <td>4.2</td>
                                                </tr>
                                                <tr>
                                                    <td>Mary Moe</td>
                                                    <td>4.7</td>
                                                </tr>
                                                <tr>
                                                    <td>July Dooley</td>
                                                    <td>4.5</td>
                                                </tr>
                                            </tbody>
                                        </table>
                                    </div>
                                </Col>
                            </Row>
                        </Grid>
                </Well>
            )
        }
    }
    
    
    

     

    When rendered it looks like this

     

    image

     

     

    Conclusion

    I am no web designed, but I am fairly pleased with the outcome of the static screen designs, and have found using React-bootstrap/react/the map components and the React-validation to be quite simple and they all seemed to play together quite nicely.

    MADCAP IDEA PART 5 : ADDING REACT-ROUTER

     

    Last Time

     

    Last time we looked at using Balsamiq mockups to prototype our screens. This post is a small one, but one that I have found slightly trickier to get to work. This time we will look at routing inside of a React application.

     

    PreAmble

    Just as a reminder this is part of my ongoing set of posts which I talk about here :

    https://sachabarbs.wordpress.com/2017/05/01/madcap-idea/, where we will be building up to a point where we have a full app using lots of different stuff, such as these

     

    • WebPack
    • React.js
    • React Router
    • TypeScript
    • Babel.js
    • Akka
    • Scala
    • Play (Scala Http Stack)
    • MySql
    • SBT
    • Kafka
    • Kafka Streams

     

    What is routing?

    Routing is the act of taking a uri (with our without parameters), and finding a resource (be that a page, react component file) that matches the given uri and returning that content to be rendered.

     

    What choices do we have?

    When it comes to React there are not than many choices at all. We need to either hand roll our own routing mechanism which might be accomplished either using

    window.location.hash
    window.location.href
    

    This is however quite a pain, and you would need to hand roll your own support for richer routes that requires route parameters such as /orders/{orderId}. So perhaps there is a better way out there already. There is of course, the https://reacttraining.com/react-router/

     

     

     

    React-Router

    This the defacto go to solution for routing in React. The react-router was built with react in mind, which means at its heart it is a component based router. It is quite stable and been around quite a while, The current version is v4, which is quite a different beast from the previous versions, a lot changed. So for this series of posts, and the code that goes with it, I will be using v3 of the react-router.

     

    Installing react-router

    To install react-router, you just need to use NPM (though this article has used a specific version of react-router)

    npm install react-router --save
    

     

    Creating the components

    As I say, at its heart the react-router is a component based router. So we need some components to render, route to. For the demo all my routes are simple routes that don’t require any further parameters. But rest assure the react-router can deal with routing parameters just fine.

     

    So here are the demo route components that we would like to route to.

     

    class ReDirecter extends React.Component<undefined, undefined> {
    
        handleClick = () => {
            hashHistory.push('/contact');
        };
    
        render() {
            return (
                 <button onClick={this.handleClick} type="button">go to contact</button>
    
            )
        }
    }
    
    
    
    const Home = () => (
      <div>
        <h2>Home</h2>
      </div>
    )
    
    const Contact = () => (
      <div>
        <h2>Contact</h2>
      </div>
    )
    
    
    const About = () => (
      <div>
        <h2>About</h2>
      </div>
    )
    

     

    It can be seen that these are very simple components, but they are good enough to route to. There is also an example above of how to use the router to navigate to a new route programmatically in code, but we will talk more about this later.

     

    Types of history

    So now that we have some components that can be routed to, we need to just understand a bit more about history, and what we get out of the box with react-router. There is great page describing this here : https://github.com/ReactTraining/react-router/blob/v2.0.0-rc5/docs/guides/basics/Histories.md

     

    The real crux of it is that there are several types of history providers that can be used with the router

     

    hashHistory

    Which is a simple hash history which DOES NOT need server side route support.

     

    browserHistory

    This type of history DOES require FULL server side route support. Which means you really need to think about how your components need to be split up and served from the server side code. This history provider does support isomorphic rendering though.

     

    For me the hashHistory above was what I wanted, so I have used that.

     

     

    Creating the router

    Ok so now that we have talked about the types of history and talked about having components to route to, how we create the router?

     

    Well quite simply we do it like this (remember this is v3 of the router, v4 may be different). Note that I am using ES6 style imports

     

    import * as React from "react";
    import * as ReactDOM from "react-dom";
    
    import 'bootstrap/dist/css/bootstrap.css';
    import {Nav, Navbar, NavItem, NavDropdown, MenuItem} from "react-bootstrap";
    
    import { Router, Route, hashHistory  } from 'react-router'
    
    
    
    
    
    ReactDOM.render((
        <Router history={hashHistory}>
                <Route component={App}>
                    <Route path="/" component={Home}/>
                    <Route path="/contact" component={Contact}/>
                    <Route path="/about" component={About}/>
                    <Route path="/redirecter" component={ReDirecter}/>
                </Route>
            </Router>
    ), document.getElementById('root'));
    

     

    It can be seen that the router itself is a React component and gets rendered to a target DOM element. It can also be seen that we include a hashHistory which we imported from react-router.  Each route is added as a new Route where we match the expected route path, with the component to render.

     

    I will also be showing you how to use the react-router with a bootstrap-react Navbar too, so there are imports for that too.

     

     
    How does that work with a bootstrap-react navbar?

    So I just mentioned I would like to use a bootstrap-react Navbar, so what does that look like. Well some of the eagle eyed amongst you may have noticed the line component={App} in the router setup we just saw. Lets have a look at that.

     

    import * as React from "react";
    import * as ReactDOM from "react-dom";
    
    import 'bootstrap/dist/css/bootstrap.css';
    import {Nav, Navbar, NavItem, NavDropdown, MenuItem} from "react-bootstrap";
    
    import { Router, Route, hashHistory  } from 'react-router'
    
    
    class MainNav extends React.Component<undefined, undefined> {
      render() {
        return (
         <Navbar brand='React-Bootstrap'>
             <Nav>
                 <NavItem eventKey={1} href='#/'>Home</NavItem>
                 <NavItem eventKey={2} href='#/contact'>Contact</NavItem>
                 <NavItem eventKey={2} href='#/about'>About</NavItem>
                 <NavItem eventKey={2} href='#/redirecter'>Redirect</NavItem>
             </Nav>
         </Navbar>
        )
      }
    }
    
    
    
    class App extends React.Component<undefined, undefined> {
      render() {
        return (
    
            <div>
                <MainNav/>
                {this.props.children}
            </div>
        )
      }
    }
    

     

    It can be seen that the App component is another React component that does 2 things:

    • Renders another component, namely MainNav (which is a bootstrap-react Navbar component, which is also shown above, where the Navbar has links  for the required routes)
    • Also renders the children of the router, which for this example will be a single component (so for the demo code will be one of ReDirecter / Home / Contact / About components)
    •  
     
    What about on the fly route changes?

    But what about when we are inside of a React component and we may wish to navigate somewhere else, how do we do that? Well it is done via the use of the routers history provider (hashHistory / browserHistory etc etc), here in an example of that from a simple React component that shows a button, which when clicked will perform a redirect to the Contact component, the trick is to use the history provider push(..) method to set a new route location

     

    class ReDirecter extends React.Component<undefined, undefined> {
    
        handleClick = () => {
            hashHistory.push('/contact');
        };
    
        render() {
            return (
                 <button onClick={this.handleClick} type="button">go to contact</button>
    
            )
        }
    }
    

     

     

    Demo

    Ok so now we have covered all the basics, lets run the code up and see what we get.

     

    When we first start we see this, where we can see the Navbar, and the address bar contains a # style address (this is due to using the hashHistory history provider for the Router)

     

    image

     

    We can click on the various links representing the routes, and the relevant component will get rendered by the Router. The Redirect route is shown below, where clicking the button will perform a redirect to the Contact component

     

    image

     

     

    Conclusion

    As you can see even something as page navigation in React requires a bit of thought, and depending on what type of routing you chose to use, may even require a fair amount of server side work, which could change the entire workflow quite a bit.

     

    I went for the hashHistory for this demo but for a real world production ready application, you should use the browserHistory, and ensure you have supporting server side code to deal with the expected routing requests.

    MADCAP IDEA PART 4 : PROTOTYPING THE SCREENS

     

    Last Time

     

    Last time we looked at bringing in the Play Framework (scala based MVC web framework) and making the front end work with Play. This time we will  be look at the initial prototypes of the screens.

    This is my best guess of what they may look like right now, based on my initial requirements, but as with all things once you get into the guts of it, changes will occur.

     

     

    PreAmble

    Just as a reminder this is part of my ongoing set of posts which I talk about here :

    https://sachabarbs.wordpress.com/2017/05/01/madcap-idea/, where we will be building up to a point where we have a full app using lots of different stuff, such as these

    • WebPack
    • React.js
    • React Router
    • TypeScript
    • Babel.js
    • Akka
    • Scala
    • Play (Scala Http Stack)
    • MySql
    • SBT
    • Kafka
    • Kafka Streams

     

    Mockup tool of choice

    I am a big fan of the balsamiq mockup tools https://balsamiq.com/. This comes as a stand alone installed version or as a plugin for JIRA.

    balsamiq provides the following (I am just listing the features I used, there are many more)

     

    • Drag and drop from a wide range of forms, containers, controls
    • Set content for controls (usually using some fancy design time behavior)
    • Set navigation links
    • Set properties like IsSelected, IsEnabled etc etc

     

    This is what the windows installed balsamiq desktop version looks like, see how you have many categories of items to choose from

     

    image

     

    And here is what I mean by the clever design time support. This is a data grid that I have double clicked on, where the text in design mode described the rendered results of the control

     

    image

     

    It really is a very nice tool. Anyway on with the initial screen designs

     

    Navbar

    image

     

    This will be a simple react-router / react-bootstrap based navigation bar. There is nothing much more to say about that.

     

     

    Login

    image

     

    This will be a login form which will be validated, and submitted to a Play framework controller, for further validation. The Play controller would look up the user details from a MySQL database, and if an entry is found the user is considered logged on. Keeping in simple here no oAuth no JWT, just simple lookup

     

    Passenger Register

    image

     

    If the user is a passenger the sort of information that they will need to enter to register will be different from a driver who may need to register, as such there is a specific passenger registration form, which will be validated and sent to a Play controller endpoint for storage in MySQL.

     

    Driver Register

    image

     

    If the user is a driver, we need more information about the vehicle, as such there is a specific driver registration form, which will be validated and sent to a Play controller endpoint for storage in MySQL.

     

     

    Create Job

    image

     

    Only a passenger will be able to create new jobs. Since I am doing all this work on a single laptop which is ALWAYS in a single location, I am having to SIMULATE the geo-coordinates of a job by accepting the current users input for their current position. The passenger/driver users will provide this geo information by clicking on a google map. The geo co-ordinate update will either travel through a Kafka stream, –> Akka –> Comet, or may just use Akka –> Comet. I have not fully decided on this part yet.

     

    There may only EVER be 1 active job, so if a logged in passenger tries to create a 2nd job this should cause an error

     

    image

     

    View Job

    Both passengers/drivers may view an active job. Drivers may “bid for a job” by clicking on the map providing the job is not already paired with a driver.  A driver symbol will be a car, as before the driver will update their geo co-ordinates by clicking on the map. A before the geo co-ordinate update will either travel through a Kafka stream, –> Akka –> Comet, or may just use Akka –> Comet.

     

    image

     

    A passenger may inspect a drivers details, and chose to accept the driver, at which point the passengers job become assigned the chosen driver.

     

    image

     

    Drivers that are not allocated to the job will be removed from the map, and only geo updates from the paired passenger/driver will be reflected on the map. 

     

    Passenger Completion

     

    image

    Once a job has been completed (by clicking the”Complete” button) the passenger will be able to rank the driver. This will store the ranking for the driver. This could be stored directly in MySQL, but I want to play with Kafka Streams a bit more, so we use a Kafka Publisher –> Kafka Streams –> KTable arrangement to store the state. And then use Kafka active queries to get the data out again.

     

     

    Driver Completion

    image

     

    The driver is also able to complete the job from their end (using the “complete” button), and is able to rank the passenger. This will work as described above.

     

    View Ranking

    image

     

    Depending on which way I go with the ranking storage this will either be a direct MySQL query or a Kafka Streams active query over a KTable.

     

    Conclusion

    This is perhaps the simplest of all the posts in this series, but it is an important one. Next time we will try and statically implement these screens, and the associated routing that goes with them.

    Madcap Idea Part 3 : Bringing Play Back End Into The Fold + Some Basic Streaming

    Last Time

    Last time we looked at bringing in the Inversify.js JS IOC container. This time we will bring a Play Framework (scala based MVC web framework) into the fold, and shall be using the front end we have been working on so far to be the front end for the Play Framework back end.

     

    PreAmble

    Just as a reminder this is part of my ongoing set of posts which I talk about here :

    https://sachabarbs.wordpress.com/2017/05/01/madcap-idea/, where we will be building up to a point where we have a full app using lots of different stuff, such as these

    • WebPack
    • React.js
    • React Router
    • TypeScript
    • Babel.js
    • Akka
    • Scala
    • Play (Scala Http Stack)
    • MySql
    • SBT
    • Kafka
    • Kafka Streams

     

    What Is The Play Framework?

    The Play Framework is a Scala based MVC (model view controller) type web application framework. As such it has in built mechanisms for things typical of a MVC web framework (certainly if you have done any ASP MVC . NET you would find it very familiar).

    So we have the typical MVC concerns covered by the Play Framework

    • Controllers
    • Actions
    • Routing
    • Model binding
    • JSON support
    • View engine

    Thing is I will not be doing any actual HTML in the Play Framework back end code, I want to do all of that using the previously covered webpack/typescript/react starter code I have shown so far. Rather I will be using the Play Framework as a API backend, where we will simply be using various controllers as endpoint to accept/serve JSON, and Event streamed data. All the actual front end work/routing will be done via webpack and React.

     

    There are still some very appealing parts in Play that I did want to make use of, such as:

    • It is Scala, which means when I come to integrate Kafka / Kafka Streams it will be ready to do so
    • It uses Akka which I wanted to use. I also want to use Akka streams, which Play also supports
    • Play controllers lend themselves quite nicely to being able to create a fairly simple REST API
    • It can be used fairly easily to serve static files (think of these as the final artifacts that come out of the webpack generation pipeline). So things like minimized CSS / JS etc etc

     

    So hopefully you can see that using Play Framework still made a lot of sense, even if we will only end up using 1/2 of what it has to offer. To be honest the idea of using controllers for a REST API is something that is done in ASP MVC .NET all time either by using of actual controllers or by using the WebApi.

    Ok so now that we know what we will be using Play Framework for, how about we dive into the code for this post.

     

    Play Framework Basics

    Lets start by looking at the bare bones structure of a Play Framework application, which looks like this (I am using IntelliJ IDEA as my IDE)

    image

    Lets talk a bit about each of these folders

     

    app

    This folder would hold controllers/views (I have added the Entities folder there that is not part of a Play Framework application requirements). Inside the controllers folder you would find controllers, and inside the views folder you would find views. For the final app there will be no views folder, I simply kept that in this screenshot to talk about what a standard Play Framework application looks like

     

    conf

    This folder contains the configuration for the Play Framework application. This would include the special routes file, and any other application specific configuration would might have.

    Lets just spend a minute having a look at the Play Framework routes file, which you can read more about here : https://www.playframework.com/documentation/2.5.x/ScalaRouting

    The routes file has its own DSL, that is responsible for matching a give route with a controller + controller action. The controller action that matches the route is ultimately responsible for servicing the http request. I think the DSL shown in routes file below is pretty self explanatory with perhaps the exception of the assets based routes.

     

    All assets based http requests (ie ones that start with /assets for example http://localhost:9000/assets/images/favicon.png would actually be routed through to a special controller called Assets. You dont see any code for this one, its part of the Play Framework application codebase. This special Assets inbuilt play controller is responsible for serving up static data files which it expects to find in the public folder. So for example our initial request of http://localhost:9000/assets/images/favicon.png would get translated into this file (relative path from project root) /public/images/favicon.png. As I say this is handled for you by the special Assets built in controller.

     

    The only other funky part to the Assets based route is that it uses a *file in its route. Which essentially boils down to the play framework being able match a multi-part path. Which we actually just saw with the example above http://localhost:9000/assets/images/favicon.png , see how that contains not only the file name, but also a directory of images. The Assets controller + routing is able to deal with that path just fine.

     

    # Routes
    # This file defines all application routes (Higher priority routes first)
    # ~~~~
    
    # Home page
    
    GET        /                                   controllers.HomeController.index()
    
    GET        /scala/comet/liveClock              controllers.ScalaCometController.streamClock()
    GET        /scala/comet/kick                   controllers.ScalaCometController.kickRandomTime()
    
    # Map static resources from the /public folder to the /assets URL path
    GET        /assets/*file                       controllers.Assets.at(path="/public", file)

     

    Ok so moving on to the rest of the standard folders that come with a Play Framework application

     

    public

    This is where you will need to put any static content that you wish to be served. Obviously views (if you use that part of play) will be within in the app/views folder. Like I say I am not using the views aspect of Play so you will not be seeing any views in my views folder. I instead want to let webpack et all generate my routing, web page etc etc. I do however want to serve bundles so later on I will be showing you how my webpack generated bundles fit in with the Play Framework ecco system.

     

    target

    Since this is a scala based project we get the standard scala based folders, and target is one of them, that has the generated/compiled code in it.

     

    SBT

    It is worth pointing out that my Play Framework application is an SBT based project, as such there is an SBT aspect to it, which largely boils down to these files

     

    Project [root-build] / plugs.sbt file

    This file adds Play as a plugin for the SBT project

     

    // The Lightbend repository
    resolvers += Resolver.typesafeRepo("releases")
    
    // Use the Play sbt plugin for Play projects
    addSbtPlugin("com.typesafe.play" % "sbt-plugin" % "2.5.14")

     

    build.sbt

    This is the main SBT build file for the project. This is where all our external dependencies are brought in etc etc (standard SBT stuff)

     

    import play.sbt._
    import sbt.Keys._
    import sbt._
    
    name := "play-streaming-scala"
    
    version := "1.0-SNAPSHOT"
    
    scalaVersion := "2.11.11"
    
    lazy val root = (project in file(".")).enablePlugins(play.sbt.PlayScala)
    
    javacOptions ++= Seq("-source", "1.8", "-target", "1.8", "-Xlint")
    
    initialize := {
      val _ = initialize.value
      if (sys.props("java.specification.version") != "1.8")
        sys.error("Java 8 is required for this project.")
    }

     

    So I think that covers the basics of a standard Play Framework application, the remainder of this post will look at the following aspects

    • How do we provide a stream based request, that allows server sent events to be sent back to JavaScript from server side code
    • How do we integrate the webpack based front end we have been playing with in previous posts

     

    How do provide a stream based request (server sent events)

    One of the main ingredients of the application we have chosen to build is the ability to stream data back to the JavaScript in response to incoming Kafka data. Ok we have not got to the Kafka part yet, but we know we will need some way of pushing data from the Play Framework application code to the front end JavaScript code. So how do we do that?

    Well there are several parts to this, so lets just go through them in turn

     

    Play streaming route

    There are 2 routes for the example stream for this post

    • The actual stream endpoint route itself
    • An endpoint that allows us to produce a new value for the Akka Stream that is exposes in the stream endpoint route

    Here is the full routing code for these 2 routes

     

    GET        /scala/comet/liveClock              controllers.ScalaCometController.streamClock()
    GET        /scala/comet/kick                   controllers.ScalaCometController.kickRandomTime()

     

    Play stream provider

    The job of creating the stream is handled by the ScalaCometController where it uses Akka Streams to provide the stream itself. The clever part is what parts of the Akka Streams API we used.

    I have chosen to use MergeHub and BroadCastHub, which act as Fan-In and Fan-Out stages respectively. This allows us to have many producers and many consumers that are able to push and listen to the stream.

     

    I have also supplied a route kickRandomTime() which will produce a new random JSON value that will either be a Location or a Resident domain object entity converted to JSON.

    Another thing to note is how we handle stream errors. We use ideas borrowed from regular Akka where we use a strategy to manage the stream

    Here is the full code to the ScalaCometController

     

    package controllers
    
    import java.time.ZonedDateTime
    import java.time.format.DateTimeFormatter
    import javax.inject.{Inject, Singleton}
    
    import akka.actor.ActorSystem
    import akka.stream.{ActorMaterializer, ActorMaterializerSettings, Materializer, Supervision}
    import akka.stream.scaladsl.{BroadcastHub, Keep, MergeHub, Source}
    import play.api.http.ContentTypes
    import play.api.libs.Comet
    import play.api.libs.json._
    import play.api.mvc.{Controller, _}
    import Entities._
    import Entities.JsonFormatters._
    
    import scala.concurrent.ExecutionContext
    
    
    object flipper
    {
      var current: Boolean = false
    }
    
    
    @Singleton
    class ScalaCometController @Inject()
      (
        implicit actorSystem: ActorSystem,
        ec: ExecutionContext
      )
      extends Controller  {
    
      //Error handling for streams
      //http://doc.akka.io/docs/akka/2.5.2/scala/stream/stream-error.html
      val decider: Supervision.Decider = {
        case _                      => Supervision.Restart
      }
    
      implicit val mat = ActorMaterializer(
        ActorMaterializerSettings(actorSystem).withSupervisionStrategy(decider))
    
    
      val (sink, source) =
        MergeHub.source[JsValue](perProducerBufferSize = 16)
          .toMat(BroadcastHub.sink(bufferSize = 256))(Keep.both)
          .run()
    
      def streamClock() = Action {
        Ok.chunked(source via Comet.json("parent.clockChanged")).as(ContentTypes.HTML)
      }
    
      def kickRandomTime() = Action {
    
        var finalJsonValue:JsValue=null
    
        flipper.current = !flipper.current
        if(flipper.current) {
          finalJsonValue = Json.toJson(Location(1.0,1.0))
        } else {
          val rand = DateTimeFormatter.ofPattern("ss mm HH").format(ZonedDateTime.now().minusSeconds(scala.util.Random.nextInt))
          finalJsonValue = Json.toJson(Resident("Hove", 12, Some(rand)))
        }
        Source.single(finalJsonValue).runWith(sink)
        Ok(s"We sent '$finalJsonValue'")
      }
    
    }

    Client side Comet frame

    The Play Framework has support for web sockets, and we could have used a web socket, but there are times when you dont want to initiate some comms, you just want notifications of something that happened on the server. Luckily the Play Framework has support for this too using Comet, which you read more about here : https://www.playframework.com/documentation/2.5.x/ScalaComet

    What this really boils down to is have a IFRAME that is permanently part of the rendered HTML (forever-iframe technique) that is linked up to a comet play url. For me this is like this

     

    /scala/comet/liveClock

     

    Where it can be seen that this uses the route

     

    GET        /scala/comet/liveClock              controllers.ScalaCometController.streamClock()

     

    Which we saw above

     

    Client side RX comet listener

    So we have now seen how the back end produces a stream of differing JSON object (ok toggles between 2 fixed JSON objects), and how we use the forever-iframe idea with a comet based Play route. So what about the message coming into the JavaScript what does that look like?

     

    Well here is the relevant code, where we wrap the code we want to run in a self executing function. There are a couple of points here of note

    Ok.chunked(source via Comet.json("parent.clockChanged")).as(ContentTypes.HTML))
    • We create a top level window function called clockChanged which ties up with the back end comet route code (
    • We create a custom event that we dispatch via the window
    • We use that custom event to create an Rx Observable (I want this separation as long term the Rx code will become a service that is injected into the props of a React component from the IOC container)

     

    import Rx from 'rx';  
    
    
    (function () {
    
        var evt;
    
        window['clockChanged'] = function (incomingJsonPayload) {
            evt = new CustomEvent('onClockChanged', { detail: incomingJsonPayload });
            window.dispatchEvent(evt);
        }
    
        var source = Rx.Observable.fromEvent(window, 'onClockChanged');
    
        var subscription = source.subscribe(
            function (x) {
                console.log('RX saw onClockChanged');
                console.log('RX x = ', x.detail);
            },
            function (err) {
                console.log('Error: %s', err);
            },
            function () {
                console.log('Completed');
            });
    
    } ());

    And that is all there is to that part. As I say this is demo code and will be moved about a bit over time, but it does show the moving parts ok.

    How do integrate our existing webpack based front end with Play

     

    Up until now I have avoided talking about the view, and Play and the code from the webpack based front end we have seen in the previous posts. Things is we can do views in play but I was kind of keen to do all my front end work in TypeScript/react/webpack. So I wanted play to merely serve up my front end code from those tools. What that means is really serving up the final JS/CSS bundles and index.html produced by webpack and the awesome HtmlWebpackPlugin (that uses a template and injects the right script bundles). So can this webpack based workflow work ok with Play. I thought about it a bit, and came up with this.

     

    Making sure webpack generates bundles to correct place

    The first thing to do was to include the code in the Play application at some location. For me this as follows:

    image

    You want to avoid putting to much stuff under the public folder in Play as Play treats these files as static files, that can be cached etc etc, so 100MB of node modules should defo not go under public if you can help it.

     

    Then from there we need to ensure the bundles get built to the correct location, which for my setup and my Play application meant the public folder

     

    Making sure bundles are prefixed

    The other thing I needed to do was make sure the webpack output was setup to have the correct public path.

    To do the 2 things above, I changed my webpack config file to this

    image

    With these changes in place this is what I now get for my index.html page in the public\dist folder

     

    <!DOCTYPE html>
    <html>
        <head>
            <meta charset="UTF-8" />
            <title>Hello React!</title>
    
           
    		<link href="/assets/dist/vendor.bundle.994d68871d44c4fc440c.css" rel="stylesheet">
    		<link href="/assets/dist/indexCss.bundle.994d68871d44c4fc440c.css" rel="stylesheet"></head>
        <body>
             
            
    <!-- Main --> /assets/dist/vendor.bundle.994d68871d44c4fc440c.js /assets/dist/index.bundle.994d68871d44c4fc440c.js </body> </html>

     

    See how the bundles point to the assets folder, which we now know Play will serve using the AssetsController.

     

    Serving the initial webpack generated HTML page

    The last piece of the puzzle is how do we get a standard play route to serve up this index.html file? Well here is how

     

    package controllers
    
    
    import javax.inject.Inject
    
    import play.api.mvc.{Action, Controller}
    
    class HomeController @Inject() (environment: play.api.Environment)
      extends Controller {
    
      def index() = Action {
        val fullpath = s"${environment.rootPath}\\public\\dist\\index.html"
        val htmlContents = scala.io.Source.fromFile(fullpath).mkString
        Ok(htmlContents).as("text/html")
      }
    
    }

     

    Quite cunning no, we just read the file as text, and serve the string contents as HTML. Where the route for this is still using the standard Play route

     

    GET        /                                   controllers.HomeController.index()

     

    And bam, we now have a streaming Akka/Play/Webpack app working. Neato

     

    A Demo

    Now that I have talked about most of the parts, lets see a little demo of it working, so we need to load our site up (which is the GET / route above)

    image

    Our ugly but functional, react page is shown. Not very exciting, lets see the console (clear it), and load up the great REST tool Postman, and poke the stream simulation endpoint, and watch our Rx JavaScript code write out to the console

     

    Here is the console after a few Send clicks in postman for the http://localhost:9000/scala/comet/kick route

    image

     

    Conclusion

    So that is all I wanted to say this time. I think it has worked out fairly well. Next time we will be looking at adding react-routing into the fold, and create some routes, and dummy place holder pages which we develop along the way (after we do the post on prototyping the screens of course)

    MADCAP IDEA PART 2 : ADDING DI/IOC TO THE CLIENT SIDE FRONT END WEB SITE

     

    Last Time

    Last time we built a bare bones react/webpack/babel/typescript/bootstrap web site. You can read that post here should you wish to: https://sachabarbs.wordpress.com/2017/05/15/madcap-idea-part-1-start-of-the-client-side-portion-of-the-web-site/

     

    PreAmble

    This post will be about adding DI/IOC to the bear bones no thrills client portion of the web site that we built last time. Just as a reminder this is part of my ongoing set of posts which I talk about here :

    https://sachabarbs.wordpress.com/2017/05/01/madcap-idea/, where we will be building up to a point where we have a full app using lots of different stuff, such as these

    • WebPack
    • React.js
    • React Router
    • TypeScript
    • Babel.js
    • Akka
    • Scala
    • Play (Scala Http Stack)
    • MySql
    • SBT
    • Kafka
    • Kafka Streams

     

    What Is DI/IOC?

    In software engineering, dependency injection is a technique whereby one object supplies the dependencies of another object. A dependency is an object that can be used (a service). An injection is the passing of a dependency to a dependent object (a client) that would use it. The service is made part of the client’s state. Passing the service to the client, rather than allowing a client to build or find the service, is the fundamental requirement of the pattern.

    This fundamental requirement means that using values (services) produced within the class from new or static methods is prohibited. The class should accept values passed in from outside.

    The intent behind dependency injection is to decouple objects to the extent that no client code has to be changed simply because an object it depends on needs to be changed to a different one.

    Dependency injection is one form of the broader technique of inversion of control. Rather than low level code calling up to high level code, high level code can receive lower level code that it can call down to. This inverts the typical control pattern seen in procedural programming.

    As with other forms of inversion of control, dependency injection supports the dependency inversion principle. The client delegates the responsibility of providing its dependencies to external code (the injector). The client is not allowed to call the injector code. It is the injecting code that constructs the services and calls the client to inject them. This means the client code does not need to know about the injecting code. The client does not need to know how to construct the services. The client does not need to know which actual services it is using. The client only needs to know about the intrinsic interfaces of the services because these define how the client may use the services. This separates the responsibilities of use and construction.

    There are three common means for a client to accept a dependency injection: setter-, interface- and constructor-based injection. Setter and constructor injection differ mainly by when they can be used. Interface injection differs in that the dependency is given a chance to control its own injection. All require that separate construction code (the injector) take responsibility for introducing a client and its dependencies to each other

     

    From https://en.wikipedia.org/wiki/Dependency_injection

     

     

    What Choices Do We To Do This When Working With React?

    There are several choices we have when working with React, such as

    • Using react context
    • Using the module system
    • Using a 3rd party DI/IOC system (I will be covering this, in this post)

     

    There is VERY good post on these different techniques here : https://github.com/krasimir/react-in-patterns/tree/master/patterns/dependency-injection it is a great read and I suggest you take a look to gain a better understanding of some of the more obscure areas of react (context I am looking at you)

     

     

    Which 3rd Party DI/IOC Library Did I Chose And Why?

    I have decided to use the https://github.com/inversify/InversifyJS (Inversify JS) DI/IOC library. Why did I chose this one, well there were quite a few reasons:

     

    • It is written in TypeScript, and I wanted to use TypeScript where possible
    • Is looks like a good full featured container, that reminds me of many other containers that I have worked with in .NET/Scala
    • It is quite mature and is in version 2.0
    • It has quite a bit of good press around it
    • The documentation is good
    • It did what I wanted

     

    So those were my reasons, so what do we have to do to get Inversify JS to work?

     

    Installation

     

    Node Module Installation

    The first steps is to make sure we have the correct node packages, to do this we just need to issue the following  NPM (node package manager) command line :

    npm install inversify reflect-metadata --save
    

     

    Once this is done you should have the following entries in your package.json file

     

    image

     

     

    Changes To The TypeScript “tsconfig.json” File

    The other thing that Inversify JS needs is a couple of specific TypeScript settings. These need to go in the tsconfig.json file, the new lines since last time are these

     

    image

     

     

     

     

    That is all the setup we need, so let’s now have a look at using the Inversify JS DI/IOC classes, decorators etc etc

     

     

    Creating Some Injectable Thing

    Lets start with creating something that can be injected with other values, and can be resolved from the container.

     

    In Inversify JS  a type has to be marked as @Injectable, which is not something you have to do in other DI/IOC offerings. This is more than likely a requirement because JavaScript is a dynamic language and these decorators are used to create extra metadata to ease in the resolution mechanisms used by ALL IOC containers.

     

    To actual get something injected into a class we need to use the @Inject decorater.

     

    Here is an example of a class that can be resolved from the Inversify JS  IOC container, and will also have it constructor dependencies resolves by the Inversify JS  IOC container.

     

    import { injectable, inject } from "inversify";
    import { TYPES } from "../types";
    
    @injectable()
    export class Foo {
    
        private _num: number;
    
        constructor(@inject(TYPES.SomeNumber) num: number) {
            this._num = num;
        }
    
        getNum() {
            return this._num * 2;
        }
    
    }
    

     

    You can see that we also use some TYPES. What are these, lets take a look at that.

    export const TYPES = {
        Foo: Symbol("Foo"),
        SomeNumber: Symbol("SomeNumber")
    };
    

     

    It can be seen this TYPES constant just offers us a way of using Symbol as runtime identifiers for our dependencies.

     

    Ok so now that we have a class that expects to have its dependencies satisfied from the Inversify JS IOC Container, and is itself resolvable, lets see how we can configure the container.

     

    Creating The Container

    I have chosen to create a singleton object for my container, which looks like this

    import "reflect-metadata";
    import { Container } from "inversify";
    import { TYPES } from "../types";
    import { Foo } from "../domain/Foo";
    
    export class ContainerOperations {
        private static instance: ContainerOperations;
        private _container:Container = new Container();
    
        private constructor() {
            
        }
    
        static getInstance() {
            if (!ContainerOperations.instance) {
                ContainerOperations.instance = new ContainerOperations();
                ContainerOperations.instance.createInversifyContainer();
            }
            return ContainerOperations.instance;
        }
    
        private createInversifyContainer() {
            this.container.bind<number>(TYPES.SomeNumber).toConstantValue(22);
            this.container.bind<Foo>(TYPES.Foo).to(Foo);
        }
    
        public get container(): Container {
            return this._container;
        }
    }
    

     

    There are a couple of things to note in the above code:

    • We import “reflect-metadata”, “Container” and “TYPES”
    • We have our singleton that simple wraps the Inversify JS IOC container.
    • We configure the the container registrations (bindings in Inversify JS speak)

     

    And that is all there is to that part. So all that is left, is to actually resolve something from the container. We will look at that next

     

    Resolving Something From The Container

     

    As I am using react, I will likely be using the Inversify JS IOC container to assist me with creating the props for the react components. That is not strictly relevant to this discussion, so lets just see how we can resolve an instance of the Foo IOC registered class using Inversify JS

     

    We do this as follows:

    import { Foo } from "./domain/Foo";
    import { TYPES } from "./types";
    import { ContainerOperations } from "./ioc/ContainerOperations"; 
    
    let foo = ContainerOperations.getInstance().container.get<Foo>(TYPES.Foo);
    

     

    It can be seen that we simple make use of the singleton (that wraps the container) to resolve our Foo class. Happy days

     

    Conclusion

    I have to say I did struggle a bit with getting Inversify JS up and running in my project. But I also have to say that I asked a question on the Inversify forum and the author Remo Jansen was absolutely brilliant in helping me to get my stuff to run. To the point where I pointed him at my GitHub repo, and he looked at, got it to work, and sent me a pull request.

     

    Remo I tip my hat to you sir, top library, top fella. And as promised I owe you that beer

     

    So once it was installed, I found it very easy to work with, it soon felt like many other IOC frameworks I have used (NInject, Funq,Munq, Castle, AutoFac, Unity, MEF take your pick). I was very happy with the results.

     

    As I previously stated I will be continuing to write posts which will be tracked on Trello : https://trello.com/b/F4ykCOOM/kafka-play-akka-react-webpack-tasks