Deepu Mohan Puthrote

• Saved ~1000USD/month on S3 by identifying and deleting millions of unnecessary objects
• Designed and implemented feature to prevent duplicates in historical records spanning multiple services
• Reduced time for adding new external integration from months to 1 week by automating the process
• Reduced time taken for configuring various data feeds by building a self service tool for the ops team. Built backend APIs in Spring/Kotlin and frontend in React
• Optimised code base by deleting ~10,000 lines of code, but still delivering the same business requirements with zero downtime. This improved the maintenance time of the system.
• Improved developer workflow by identifying and automating a task that took 45mins to 1hour per developer
• Designed and implemented prevention of historic duplicates in outgoing data files
• Implemented a paginated search api over postgres and jOOQ
• Identified and fixed several bugs and security vulnerabilities

Kotlin · Spring Boot · Test Containers · Kubernetes · AWS · Airflow · jOOQ · React

Building a self-service data platform for batch analytics, machine learning development, and near-realtime processing, improving data accessibility and processing efficiency. Led technical initiatives, collaborating with the VP of Data, Director of Data, Head of Data Engineering, Head of Data Governance, Head of ML/AI, Head of Self Service Analytics, and Senior Manager of Data Platform, along with internal partners like Product, InfoSec, and SRE, to ensure alignment and technical excellence

Various POCs, RFCs, standardisations, data reliability engineering and support the data platform engineering teams. Working an next generation machine learning piplines using Sagemaker. Data access protection using Lakeformation.

RFC for AWS account segregation within organisation for data, POC for metadata platform using DataHub, metadata tagging, migration tools and projects etc. was delivered while being Principal Engineer at Funding Circle.

Wrote down Data Platfrom Principles that makes it clear for stakeholders and teams to meet the expectations. Wrote Standards for Monitoring and Alerting.

Introduced AWS Fargate as an option to run tasks instead of Mesos within Data Platform. Brought down cost of AWS Glue and S3 by redesiging pipeline with a compaction step. Introduced the idea of "decentralised" projects to isolate related resources in their on repositories. Lots of interviews.

Funding Circle is a leading small business loans platform, where investors lend to small businesses.

Built software and AWS infrastructure for the data platform: ETL tooling on Kafka, Athena and Glue, and streaming, batch and event-driven jobs.

• Introduced the concept of “inversion of control” in the way data related products were setup at Funding Circle, that fundamentally changed how we built tools. This concept allowed us to deliver solutions faster for a wide range of data products and data science projects.
• Designed and developed data validation tool using Clojure that can validate data on multiple datasources like Postgres and AWS Athena
• Proposed a better way for deploying Airflow DAG within our infrastructure which my team implemented later.
• Built ETL’s based on Kafka streaming, Athena SQL and Glue
• Streamlined Glue and Lambda deployments using automation tools.
• Proposed solution for a generic data loader for documents in multiple formats like JSON, XML and CSV, that supports schema validation and evolution.

Software Engineer/DevOps · London, UK

• Research on AWS Sagemaker
• Prepare JSON data from Bson using PyMongo library
• Design and implement data pipeline using EMR
• Design and implement machine learning pipeline using EMR
• Streaming data with Kinesis
• Streaming KMeans Clustering algorithm using Apache Spark Streaming context
• Persistance in S3 and DynamoDB
• Visualising streaming clusters with LightningServer and Dash
• Infrastructure automation with Terraform and Ansible

AWS · Spark · Hive · Python · Jupyter · EMR · Terraform · Zeppelin · AWS Glue · AWS Athena · S3 · DynamoDB

Software Engineer/DevOps · Norway, London

Data project to analyse large volumes of data on Hadoop echo system on Microsoft Azure platform. This project has a data pipeline part, some data scrapping and data analysis part. Data came from various motors and sensors.

• Get vessel and weather information from web using Microdata and API’s in Python
• Data pipeline built on Azure HDInsights platform
• Apply transformations to Hive tables using PySpark based on algorithms by data scientists
• Created a Workload report of HDInsights using Azure Runbooks, using Yarn and Livy APIs
• Build a data analytics platform on Azure using HDInsights, Spark and Jupyter
• Setup Ansible playbooks for configuring JupyterHub and users
• Explored Apache Ranger using Hortonworks Sandbox on Ubuntu

Jupyter · Apache Spark · HDInsights · Azure · Ansible · Plotly · Apache Livy · Horton-works · Python · Azure Storage accounts

Senior Scala Consultant · London, UK · Client: CSHR

Project Type: Website, Customer responsive platforms

This is a UK government project delivered following the standards set by Government Digital Services. The UK government uses Fast Track and Fast Stream to recruit talents for the Civil Service.

• Re-design the two services, with new GOV.UK services, so that candidates can apply and attend online tests
• Back office system for candidate support
• This is a responsive website service that is easy to use on any device.

This project involved creating API, candidate frontend and admin frontend for UK Government Civil Service HR Faststream candidate application.

Part of this project is open source and is available at hmrc/fset-faststream, hmrc/fset-faststream-frontend

Scala (Play framework) · HTML5 · CSS3 · JS · Progressive enhancement · Respon-sive web design

The project involved building a reporting platform using micro-services architecture in a Kanban environment. RESTful API’s were built using Node.js and front end was built with React.

• Developed user identity management service API and implemented authorisation algorithm using JSON Web Tokens(JWT) and oAuth.
• Designed and implement various REST API endpoints in Node.js and Restify for getting the data. HATEOAS (Hypermedia As The Engine Of Application State) pattern was used for linking between resources.
• Wrote a framework Chef Cookbook (Library cookbook) for deploying Node.js based services.
• Wrote Chef cookbooks (Application Cookbook) for various services, including rspec tests and integration tests with kitchen vagrant. Acted as pseudo DevOps member within the team.
• Implemented API Gateway using nginx and Chef.
• Implemented Service Discovery using Hashicorp Consul. This radically improved the time required for the service to be available in the gateway after deployment.
• Implemented next iteration of API Gateway with nginx, Consul and chef.
• Deployment in AWS Cloud. Setup nodes using Scalr. Fix various deployment issues.
• Did a brown bag session on Hashicorp Consul and service discovery, which was well-received.
• Worked closely with Operations team to deploy services to Production environment.

Node.js · Chef · cookbooks · rspec · serverspec · chefspec · AWS Cloud · Teradata · Node.js · Ruby · React · redux · rubocop · Jenkinss · Scalr · bamboo · stash · gitlab · DevOps · restify · Bamboo · upstart

Big data analytics project for detecting anomalies in various time series data using machine learning techniques. Worked closely with Data scientists to develop models and the platform.

• Worked closely with Data scientists to create models in Scala to run on Apache Spark. Original models were written in Python and R.
• Used Spark to parallelise forecasting based on splits of data. Wrote Scala based tool for triggering AWS Simple Notification Service (SNS).
• Automated Airflow DAG creation using Python based on configuration files in AWS S3.
• Wrote AWS Lambda functions in Python to trigger data pipeline.
• Setup Elastic Search cluster on AWS cloud. Wrote a script to automate the process.
• Optimised data ingestion to Elasticsearch from Hive on AWS Elastic Map Reduce (EMR), brought down the time taken from 3.5 hours to 20 minutes.
• Used Spark to move data from Teradata to S3 in Parquet format.

Apache Spark · Hive · AWS Cloud · AWS Lambdas · Python · AWS SNS · AWS Elastic Map Reduce · Scala · Python · Airflow · Bamboo

• In-house BigData Analytics product, which produces reports from the ad delivery/engagement logs generated by the ad server.
• Designed and developed secure REST API for reporting dashboard using Play Framework and Redis for caching.
• Play framework and NodeJS/ExpressJS based secure RESTful APIs
• Rich UI using AngularJS and RequireJS

• Setup and lead Play (Scala) project from scratch (Proposed the use of this framework).
• Slick (Scala) for database access.
• Implemented a state machine based Authentication header parser and developed algorithm.
• Single sign-on integration with Authentication API
• Redis for caching
• Jackson for JSON generation.
• JAXB annotations for converting authentication message to object.
• Spring Batch for batch processing and pre populating the Cache.
• Vagrant boxes for database and caching during development.
• Pair Programming
• Test Driven Development
• In-memory H2 database with YAML data files for unit testing.
• Knowledge transfer to team members.
• Build with Jenkins, deployment with Chef

• Setup and lead Node/Express project from scratch.
• Visualization using HighCharts and D3JS
• Build with Jenkins, deployment with Chef

• POC for the very ambitious dashboard project
• Created using Play (Java/Scala)
• UI with Angular JS
• Data visualization with D3JS and NVD3

• Backbone.js and Bootstrap based app for Tableau views built on top of Tableau JavaScript API
• Yeoman based project setup.
• UAT in Heroku
• Build with Grunt and Jenkins, deployment with Chef

• Maintained this already matured product.
• Modernised UI using Bootstrap, Struts, jspx and tagx
• Added new reports
• Scheduling with Quartz
• IBatis for SQL queries
• Python scripts for chores.
• Build and deployment with in-house tool

• Dissertation project on Social Network Analysis and Centrality metrics
• This project was considered the initial phase of bigger data analysis project.
• Created a friend recommendation system using Neo4J graph database.
• Analysis of social network data using R, IGraph
• Happiest's network data was analysed and centrality points were given to each node in the network.
• Graph data visualization using R and Gephi.

• Developed various API end points in PHP
• Database programming in PostgreSQL
• Responsible for GitHub commits and merges.
• Scrum methodology
• Participate in daily stand-up sessions and discussions.

• Was deployed in client site and acted as single point of contact.
• Lead the team in design and development and also gave training to team members and knowledge transfer to new members.
• Designed and developed Java based web framework for development.
• Training in Flexcube (OFSS), PL/SQL and Oracle Forms.
• Offshore support for Flexcube.
• Responsible for code maintenance and version control.
• Responsible for UAT deployment.

Created the complete design document for the project.

Lead developer in Java, JQuery, Hibernate, Jasper Reports, production deployment in Websphere server

Lead developer in Java Swing interface between Flexcube and cheque truncation automation vendor. The Swing application generates XML files as specified by the client in a given location.

Developed in Java, production deployment in Websphere server. The credit card tab worked by consuming Webservice (SOAP) provided by Credit card vendor.

Upload scanned signature images from a given location. Single-handedly developed. Used XML for representing data and XSLT to translate to html.

Web application to verify same data entered by two employees. Single-handedly developed. Used XML for representing data and XSLT to translate to html.

Built on DotNetNuke. Team member.

Webmaster