Hi,
I'm a software Engineer with hands-on experience in designing and executing solutions for complex business problems involving large-scale data warehousing, real-time analytics and reporting solutions. Supportive and enthusiastic team player dedicated to streamlining processes and able to analyze code and engineer well-researched, cost-effective and responsive solutions.
I 'm working in a multinational and leading firm in big data technologies in Pakistan. I am working as a team Lead in data ingestion and Transformation to build a Big Data Platform since August, 2014. We are building a platform that helps enterprises overcome the business and technological complexity of Big Data. The platform is built on the existing Big Data ecosystem by integrating diverse set of technologies, such as Apache Spark, Hadoop, Solr, etc., into a unified platform. It addresses the needs of both business users and developers through a user-friendly graphics interface for easy configuration and a set of APIs for developing the Big Data Application. My Responsibilities include following areas
Data Ingestion
I am responsible for data ingestion from different sources like file systems and databases, web-services and messaging queues. Experience designing and implementing fast and efficient data acquisition using Big Data processing techniques and tools
Tools: Spark, Spark Streaming, MYSQL, SQL Server, Hadoop Ecosystem, AWS S3, Kinesis, RedShift, Apache Kafka, Flume, Cassandra, Scala, Java, Eclipse, intellij-idea
Data Analysis
I have worked on different machine learning algorithms to analyze the data. I have used high-performance analytics, high-performance data mining, predictive analytics, text mining, forecasting and optimization on big data to continuously drive innovation and make the best possible decisions.
Tools: Spark MLIB and Stanford NLP, HIVE, Spark SQL, Scala, Java, Eclipse, Intellij-idea, R, Rstudio, Python
Data Transformations
I got experience to design and implement different set of mathematical and statistical operation to transform incoming data. I have worked on data cleansing to clean data from anomalies. The set of operations include join, intersect, merge, split, min,max,average, standard deviation, sum and encryption etc.
Tools: Spark SQL, HDFS, HIVE, Scala, Java, Eclipse, Intellij idea
IoT
got experience to work in IoT related technologies like arduino, MQQT, AMQP, Coap and many other protocols and built different interesting applications like shipment tracking and fire detection etc.
Tools: Arduino, C++, python, Raspberry pi, Intel Edison
