Onzo

In the end, I built two “platforms” to support the batch and streaming analytics. I discovered that there was very little overlap in terms of functional or non-functional requirements between the two platforms. The streaming platform run on the Akka stack. I chose core Akka, Streams, Persistence & Clustering. The batch platform is based on Apache Spark. Both systems used Apache Cassandra as their data store. The machine learning elements of the system are built primarily in Python/Tensorflow

• Scala
• Akka
• Python
• Google Tensorflow
• Kubernetes

Firstly I learned a lot about the industry and was able to transfer my experience to the eon optimum project.

When the direction of the business is unclear it’s best to get something up and running quickly. We need to accept that software developed this way will have to be rewritten at a later date. This is usually a hard sell to CFOs and investors. However, there is tremendous value in writing “throw away” code. The business can validate which ideas are actually viable. The tech teams can discover the challenges and pitfalls of a particular domain at relatively low cost.

It’s not the first time I’ve discovered this principle. Over the years, I’ve worked on many software projects, I would have to admit that we rarely get it right first time. My time at Onzo really reinforced this belief.

The data scientists need to “own” their models. When I joined Onzo the data scientists created models that were essentially proof of concepts. They would often be developed in Python notebooks. The POCs were then handed to Scala developers who were responsible for turning them into Scala code. This process didn’t work well. We moved to a position in which the data scientists owned the models through to production. This was far more sustainable.