Senior Data Engineers
VIDAA (Netherlands) International Holdings B.V.
- Singapore
- Senior Level
- 3+ years
- Bachelor's degree
- Posted
Skills
Required
- Software Engineering
- Data Science
- big data development
- real-time
- data warehouse implementation
- data ingestion
- system construction
- streaming task development
- Python
- Scala
- Java
Preferred
- Google Cloud
- Azure
- cloud
- big data platforms
- data governance
- user behavioral data
- Kafka
- Spark
- SQL
- Job Responsibilities 1. Own the construction and maintenance of the company’s business data ingestion system.
- Lead the implementation and iterative optimization of the real-time data warehouse architecture.
- Job Requirements 1.
About VIDAA (Netherlands) International Holdings B.V.
- VIDAA VIDAA is a leading TV OS distributor, pioneering the next-generation platform-to-the-home.
- Starting with a Smart TV OS, we aim to encompass various aspects of consumers' digital and connected lives.
- With millions of households reached daily, we are looking for people that want to drive excellence and innovation.
- We are a diverse and passionate company looking for the best and the brightest people to lead sucees.
- We value people and their needs and we provide a safe and secure environment for everyone.
- If you are self-motivated, entrepreneurial and know how to thrive in a fast growing environment, then we should talk.
Responsibilities
- Responsible for the full lifecycle development of real-time collection, cleaning, and warehousing of business logs, user behavioral data, and database operational data, ensuring the completeness, timeliness, and stability of end-to-end data ingestion pipelines.
- Design multi-layer real-time data models, develop and tune streaming ETL pipelines, and support the stable output of real-time metrics, dashboards, and business monitoring data.
- Collaborate closely with business teams, data analysts, and data product managers.
- Fully understand business logic and statistical calibers, independently complete data requirement decomposition, logical sorting, development implementation, release verification, and continuous iteration.
- 4.
- Maintain and optimize Kafka streaming pipelines for real-time data ingestion and consumption.
- Troubleshoot and resolve core online issues including message backlog, data skew, data loss and duplication, and ensure high availability and stability of real-time data pipelines.
- Develop, tune, and operate Spark real-time tasks.
- Continuously optimize computing resource utilization, data latency, and throughput, and improve the overall performance and stability of streaming data pipelines.
- Participate in data standardization construction, real-time data quality governance, and pipeline redundancy optimization.
- Summarize and standardize data ingestion specifications and development best practices to improve the overall team’s data development efficiency.
- Provide stable real-time data support for downstream business scenarios including BI dashboards, data services, user portrait platforms, and business effectiveness measurement, ensuring reliable online data service delivery.
- Expert-level knowledge of Kafka.
- Familiar with real-time data ingestion, partition strategy, consumer mechanism, and capable of diagnosing and optimizing online risks such as message backlog, data skew, data loss and duplication; experienced in large-scale streaming pipeline stability governance.
- Hands-on experience with Spark, Spark SQL, Flink and Flink SQL.
Requirements
- Bachelor’s degree or above in Computer Science, Software Engineering, Data Science, or related majors.
- 3+ years of professional big data development experience in the internet industry, with proven experience in end-to-end real-time data warehouse implementation and data ingestion system construction.
- 2.
- Proficient in English reading, writing and verbal communication skills, capable of collaborating with overseas teams and business stakeholders.
- Solid capabilities in streaming task development, resource tuning, latency and throughput optimization, and online troubleshooting.
- Familiar with real-time data warehouse layered architecture and dimensional modeling methodologies.
- Deep understanding of the big data ingestion system.
- Experienced in full-process real-time processing of multi-source data, including user behavior logs and business database data, proficient in real-time collection, CDC synchronization, data cleaning, and exception handling with standardized implementation capabilities.
- 6.
- Practical experience with cloud-native data tools including BigQuery (BQ), DBT and Databricks.
- Capable of data warehouse modeling, pipeline orchestration, metric iteration and dashboard development in cloud data environments.
- Proficient in at least one programming language among Python, Scala and Java with solid hands-on big data development capabilities.
- Familiar with Linux commands and Shell scripting, able to independently complete task deployment, log analysis and daily operation maintenance.
- 8.
- Familiar with mainstream relational databases and Binlog/CDC real-time synchronization principles.
- Experienced in business database real-time ingestion implementation.
- Knowledgeable in real-time data SLA governance, data quality monitoring, alert configuration and incident review mechanisms.
Preferred
- Overseas study or work experience is highly preferred.
- Experience with Google Cloud, Azure or other mainstream cloud big data platforms, as well as overseas team collaboration and real-time data governance experience is highly preferred.
