The Sr. Big Data Engineer will be part of a team of data experts to architect, build and operate cloud scale big data platform to deliver data analytics & AI/ML capability. In this role you will be working with team of Architects, Data Engineers and Data Experts who are Data ninjas. You will be working with new generation data streaming technologies like CDC, Kafka, Apache Pulsar , cloud scale data warehouses like AWS Redshift, Azure Synapse, Snowflake, Data lake technologies like ADLF, Lake Formation or Databricks Delta Lake, Data catalogs like Azure data catalog or AWS Glue or Apache Atlas. The data pipelines and data flow technologies like Apache Nifi, Airflow, Azure Data Factory, AWS Data Pipelines.
Essential Job Duties
• Follow agile and SAFe methodologies, Collaborate with SCRUM team consisting of product and engineering members
• Communicate effectively with team members, internal and external customers
• Mentor other engineers on database design standards
• Work with enterprise DBAs, Architecture, and Engineering to drive a data platform architecture roadmap
• Troubleshoot and solve problems quickly and efficiently
• Bachelor’s Degree in Computer Science or equivalent experience and knowledge
• 5+ Years Big data Engineering using Spark, Kafka and Similar technologies
• Hands on Experience in Handling Spark and Kafka Cluster management
• Experience in Cloud Spark Implementation is Must. (Aws,Azure)
• Hands on Experience in Streaming technologies handling TB’s of Datasets
• Experience in CI,CD pipelines deploying Kafka and spark clusters
• Experience in any BI tools like (Tableau, MicroStrategy ,Power BI ) will be preferred
• Engineering and Implementing enterprise level software applications
Skills & Expertise (Intermediate Level)
• Experience in Aws and Azure Spark Implementations
• Databricks Suite Development will be preferable
• Working on Big Data infrastructure for batch processing as well as real-time processing. Responsible for building scalable distributed data solutions using Apache Spark .
• Experience in manipulating/analyzing large datasets and finding patterns and insights within structured and unstructured data.
• Involved in collecting, aggregating and moving data from servers to Data Lake using Apache Spark .
• Implemented data ingestion and handling clusters in real time processing using Kafka .
• Developed scripts and automated data management from end to end and sync up between all the clusters.
• Involved in converting queries into Spark transformations using Spark RDDs, Java and Scala .
• Experienced in working with spark eco system using Spark SQL and Scala queries on different formats like Text file, CSV file.
• Developed and Configured Kafka brokers to pipeline data into spark streaming.
• Developed Spark scripts by using scala shell commands as per the requirement.
• Developed spark code and spark-SQL/streaming for faster testing and processing of data.
• Team player who is detailed oriented
• Strong logical & analytical thinking, verbal and written communication skills
• Works efficiently across distributed teams
• Aptitude for learning new technologies and concepts
• Excellent time management skills and ability to prioritize
• Manage several activities in parallel
• Strong analytical, verbal, and written communication skills
• Able to translate customer and business needs to technical solutions
Physical Demands (Travel, etc.)
• Available to solve critical issues as necessary
DealerSocket is a leading provider of software for the automotive industry. Our mission is to drive the future of automotive, by simplifying the experience for its customers and partners with a suite of seamlessly integrated products. DealerSocket’s suite of products includes advanced Customer Relationship Management (CRM), innovative Digital Marketing and Websites, robust Vehicle Inventory Management, insightful Analytics Reporting, and solutions to streamline dealer operations such as Desking, Credit Reporting, Compliance, and a Dealer Management System (DMS) for dealers of all sizes. To put it simply: we help dealers sell and service vehicles more profitably, while improving their customers’ experience. The DealerSocket Team is more than 1,200 employees strong, serving nearly 9,000 dealerships. We hire the best and brightest employees with offices in Dallas Texas (HQ), Southern California, Utah, Wisconsin, Arizona, Albany NY, Kaliningrad RU, and Bangalore, India. Visit dealersocket.com/careers to learn more about how you can join our winning team!