Big Data Engineer- Scala

Job Title: Big Data Engineer - SCala

Job Description: Big Data Engineer – Scala

We are looking for an experienced Big Data Engineer with strong expertise in Scala and Apache Spark to design, develop, and optimize scalable data-processing solutions. The ideal candidate should have hands-on experience with distributed data platforms, large-scale ETL/data pipelines, and big data technologies.

About Capco

“Capco, a Wipro company, is a global technology and management consulting firm. Awarded with

Consultancy of the year in the British Bank Award and has been ranked

Top 100 Best Companies for Women in India 2022 by Avtar & Seramount. With our presence across 32 cities across globe, we support 100+ clients across banking, financial and Energy sectors. We are recognized for our deep transformation execution and delivery.

You will work on engaging projects with the largest international and local banks, insurance companies, payment service providers and other key players in the industry. The projects that will transform the financial services industry.

Capco has a tolerant, open culture that values diversity, inclusivity, and creativity.

With no forced hierarchy at Capco, everyone has the opportunity to grow as we grow, taking their career into their own hands.

We believe that diversity of people and perspective gives us a competitive advantage.

Responsibilities

• Innovative thinking, delivery excellence and thought leadership to help our clients transform their business. Together with our clients and industry partners, we deliver disruptive work that is changing energy and financial services.
• ETL/ELT pipelines
• for large volumes of structured and unstructured data.
• Work with technologies such as
• Hadoop, HDFS, Hive, Kafka, and Spark SQL

Perform Spark job

• Develop reusable and maintainable data-processing frameworks.
• Troubleshoot production issues related to data pipelines and distributed applications.
• Implement data-quality checks, monitoring, and error-handling mechanisms.
• Collaborate with data architects, analysts, business teams, and other engineering teams.
• Follow software engineering best practices, including
• Git, code reviews, testing, and CI/CD

Requirements

• Big Data applications and data pipelines
• Develop high-performance data-processing solutions using
• Scala and Apache Spark
• Strong hands-on programming experience in Scala
• Apache Spark / Spark SQL
• Good understanding of
• Big Data and distributed computing concepts
• Hadoop ecosystem technologies, including Hadoop/HDFS and Hive.
• Experience building large-scale
• ETL/data engineering pipelines
• skills.

Kafka

• or similar messaging/streaming technologies.
• Understanding of data formats such as Parquet, Avro, and JSON
• Experience with performance optimization and troubleshooting of Spark applications.
• Good understanding of software development lifecycle and version-control systems.
• AWS, Azure, or GCP
• Knowledge of
• Databricks
• Airflow
• Exposure to

Delta Lake

• or other modern lakehouse technologies.
• Experience working in Agile/Scrum environments.
• Scala | Apache Spark | Spark SQL | SQL | Hadoop | HDFS | Hive | Kafka | ETL | Data Engineering | Big Data

Preferred

• or similar orchestration tools.
• Knowledge of Docker, Kubernetes, and CI/CD pipelines.