Released Mocked Streams for Apache Kafka

 

Kafka Streams is a de­ploy­ment-ag­nos­tic stream pro­cess­ing li­brary writ­ten in Java. Even though Kafka has a great test cov­er­age, there is no easy way to write unit-tests for pro­cess­ing topolo­gies, until now. I re­leased Mocked Streams for Scala, which sim­ply al­lows you to unit-test multi-in­put and out­put state­ful stream topolo­gies (since Apache Kafka >= 0.10.1) with­out Zookeeper and Kafka Bro­kers in your fa­vorite Scala test­ing frame­work e.g. Sca­laT­est and Specs2.

import com.madewithtea.mockedstreams.MockedStreams

val input = Seq(("x", "v1"), ("y", "v2"))
val expe = Seq(("x", "V1"), ("y", "V2"))
val strings = Serdes.String()

MockedStreams()
  .topology { builder => builder.stream(...) [...] }
  .input("topic-in", strings, strings, input)
  .output("topic-out", strings, strings, expe.size) shouldEqual expe

Mocked Streams

Mocked Streams 1.0 is a li­brary which al­lows you to do the lat­ter with­out much boil­er­plate code and in your favourite Scala test­ing frame­work. It wraps the org.​apache.​kafka.​test.​Proces­sor­Topol­o­gyTest­Driver class, but adds more syn­tac­tic sugar to keep your test code sim­ple. The fea­tures of the 1.0 re­lease in­clude:

• Mul­ti­ple input and out­put streams
• Spec­ify your topolo­gies as a func­tion: (KStream­Builder => Unit)
• Use .con­fig to pass your Kafka Streams con­fig­u­ra­tion (e.g. Time­stamp ex­trac­tor)
• Use .out­put­Table to match against the com­pacted table out­put as Map[K,V]
• Sup­port for Scala 2.11.8
• Sup­port for Apache Kafka 0.​10.​1.​0

Mul­ti­ple In­puts and Out­puts

val ms = MockedStreams()
  .topology { builder => builder.stream(...) [...] }
  .input("in-a", strings, ints, inputA)
  .input("in-b", strings, ints, inputB)
  .stores(Seq("store-name"))

ms.output("out-a", strings, ints, expA.size) shouldEqual(expectedA)
ms.output("out-b", strings, ints, expB.size) shouldEqual(expectedB)

Get­ting Started

See the README for using Mocked Streams. For ex­am­ples check out the test code of Mocked Streams it­self. Mocked Streams is also lo­cated in the Maven Cen­tral Repos­i­tory, hence you just need to add:

libraryDependencies += "com.madewithtea" %% "mockedstreams" % "1.0.0" % "test"

If you have any ques­tions or is­sues re­gard­ing Mocked Streams, get in touch via Github. Al­ter­na­tively, you can get in con­tact via mail.