- Singapour, Singapore
- CDI, Plein-temps
- 11 déc. 17 2017-12-11
The Hadoop engineer is a key role in the Transformation project of the bank. He's an expert in Hadoop stacks (Spark, Scala, Sqoop...) and in data related problems and how to solve them.
The Hadoop Engineer will be a part of the Data Lake team. He understands the modern data platform ecosystem & is expert in applying tools in solving the common data pipeline patterns in enterprise data lake projects covering sourcing, ingestion, transformation, consumption & presentation tiers.
The candidate for this position must be capable of playing different roles within the team, from technical lead on technologies around big data & data fabric to highly skilled developer.
The candidate must have a sound understanding of Hadoop and related tools but also very good communication skills, with the ability to adapt to different needs on project from idea to implementation.
Qualification & Experience:
- 10+ years of total IT experience
- 4+ years on Data Sourcing, Quality, Warehousing, Mining & ETL tools
- 3+ years into Big Data platform like Hadoop like Hortonworks HDP 2.5+
- Strong critical thinker with problem solving aptitude.
- Excellent written and oral communication skills
- Hands on design and development experience on Hadoop Data Platform preferably Hortonworks (HDP) covering sourcing, ingestion, processing & consumption workloads
- Hands on programming experience on Big data ETL workloads using Sqoop, Flume, Spark Core/SQL API (Scala), Hive (Tez/LLAP) and Phoenix.
- Hands on experience on Storage covering Schema layout, data modeling, partitioning & Read/Write API for Relational, MPP & NoSQL stack including File Format (Avro, Parquet), Hive, HBase, MongoDB & Oracle.
- Strong experience in setting up data governance, data security, metadata management, lineage tracking on Hadoop Platform using Kerberos, Ranger Policy, Atlas & Ambari
- Job Monitoring, debugging, Scheduling and Performance Tuning using Data Platform Operations Tools like Oozie, YARN Settings, Spark tuning, Ambari Config & Grafana
- Experience in implementing Informatica Data Fabric Tools covering Enterprise Information Catalog, Intelligent Data Lake, Big data management tools on top of Hadoop