Staff Platform Engineer - Big Data Platform Management

Responsibilities

• Design, implement and manage the Big Data Infrastructure platform, including Hadoop, Spark, NoSQL databases, and data lakes.
• Manage Big Data clusters, including monitoring, troubleshooting, and optimizing cluster performance, as well as managing job scheduling and resource allocation.
• Design and implement storage solutions for Big Data workloads, including HDFS, S3, and other object storage systems.
• Ensure the security and governance of the Big Data Infrastructure platform, including managing access controls, implementing data encryption, and conducting regular security audits.
• Collaborate with data engineers and data scientists to optimize Big Data application performance and provide technical support and guidance on using the Big Data Infrastructure platform.
• Plan and manage Big Data resource capacity, including forecasting, procurement, and deployment of new hardware and software.

Preferred

• Certifications in Big Data, such as Hadoop Certified Developer or Spark Certified Developer.
• Experience in managing physical and virtual infrastructures.
• knowledgeable in deploying and scaling LLM-based and agentic AI/ML applications.

Requirements (Minimum Qualifications)

• A degree or professional qualification in Computer Science, Engineering, or a related field
• At least 8 years of experience in managing Big Data Platforms. Proven experience with technologies like Spark, Iceberg, Trino, Elastic Stack, NiFi, Kafka, Airflow.
• Expertise in Big Data Engineering, especially in building streaming/batch pipelines, architecting data warehouses, and deploying AI/ML infrastructure.
• Capable of managing and utilizing diverse storage protocols, including S3, NFS, SMB, and HDFS.
• Strong knowledge in securing applications through SSL/TLS encryption, data at rest encryption, and robust authentication/authorization (AuthN/AuthZ) using OIDC/OAuth2.
• Comprehensive understanding of core networking principles, including firewall configuration, DNS management, routing and subnetting.
• Experience with scripting languages, such as Bash, Python, Scala and Java.
• Proficient in Kubernetes orchestration, leveraging its ecosystem for automated scaling, self-healing, and seamless CI/CD integration.
• Experience in supporting search workloads in multi-terabytes and billions of records
• Experience with agile development methodologies and version control systems, such as Git.
• Experience in leading engineering teams.

Why join us?

• The work is purposeful and meaningful
• You will work with the best engineers
• We work with modern technologies and tech stacks
• We have excellent engineering culture and work-life balance
• We aspire to engineering and operational excellence
• We empower to innovate

We grow together as a family

As CSIT is an agency under the Ministry of Defence (Singapore), only Singapore Citizens will be considered.