data engineer岗位职责

职位描述:
寻找一位经验丰富的数据开发工程师，负责公司内部数据湖的开发与维护，基于业务需求构建并维护数据报表和搭建ETL Pipeline。您将与数据科学家、业务分析师和其他工程师紧密合作，构建高效、可靠的数据处理系统，为公司提供数据驱动的决策支持。
工作职责:
• 参与公司数据湖的开发和维护，确保数据的高效存储、处理和分析。
• 基于业务需求，设计和开发数据报表，为业务部门提供数据洞察。
• 构建和维护高效的ETL Pipeline，处理来自不同数据源的批量数据和实时数据。
• 优化数据处理流程，提高数据处理的效率和稳定性。
• 编写清晰、可维护的技术文档。
技能要求:
• 精通Hive、Yarn、Spark等大数据处理框架，了解Flink者优先。
• 熟练使用Python进行数据处理和分析，熟练掌握Scala或Java其中一种语言。
• 具备ClickHouse和Vertica等数据库的实战经验，能够构建健壮的Streaming或Batch ETL Pipeline。
• 深入了解Kafka原理，具备后端开发经验，熟悉SpringBoot等框架者优先。
• 具备良好的沟通能力和团队合作精神，能够独立解决问题。

岗位要求：
1.计算机科学、工程学或相关领域的学士学位。
2.至少3年在ETL（数据抽取、转换和加载）开发、数据工程或类似角色的经验。
3.熟练掌握ETL工具和技术，能够高效地进行数据抽取、转换和加载工作。
精通SQL语言，并具有使用数据库的经验，能够进行数据查询、操作和管理。
4.熟悉Python编程语言，能够使用Python进行数据处理和自动化脚本编写
5.强大的分析和解决问题的能力。
6. 出色的沟通和协作技能。
7. 云项目经验。
岗位职责：
• 通过CICD自动化设计、开发和维护可扩展且高效的数据管道
• 与利益相关者合作，了解数据需求并开发ETL解决方案
• 优化和改进现有的ETL流程，提高数据质量和性能
• 排查和解决数据相关问题
• 为ETL流程和数据模型编写和维护文档
• 构建API，用于数据供应和实时数据服务
• 参与代码开发和审查，提高代码质量
• 与敏捷教练合作，优先处理任务并确保项目按时交付
• 关注行业趋势和技术，以实现高效的ETL流程

Issued by:
Cathay Pacific (Shenzhen) Information Technology Management Limited
国泰（深圳）信息科技管理有限公司
国泰航空公司的全资子公司
Solution Analyst
解决方案分析师
Location: Shenzhen
Role Introduction:
This role is responsible fensuring requirements are mapped documented to functional technical specifications fsoftware development projects enhancements through working with various internal external stakeholders leveraging external internal resources to deliver quality working software application(s) that is aligned with business vision goal in the delivered software application.
Key Responsibilities:
1. Analyse business requirements document technical design specification
2. Participate in all aspects of solution design, implementation support including Airline Operations Information Data Warehouse data modelling as primary focus; as well as Extract/Transform/Load (ETL) data integration as secondary focus
3. When necessary develop solutions components using combination of AWS services including Redshift, DynamoDB, Lake Formation, Lambda, Glue, Step, SNS CloudWatch
4. Automate integrate system data exchange through API
5. Manage vendors QA their deliverables in detail ensure any defects non-compliance in standards addressed promptly
6. Work as part of project team to deliver solutions potentially support all aspects of the delivery including analysis, implementation, configuration, testing, demonstration service transition
7. To collaborate with Enterprise Quality Governance team to ensure project testing meets the company practices standards
8. Collaborate with other IT teams to drive business technology innovation
9. Work under Scrum Agile development framework
10. Keep abreast of the business strategy, business plan latest industrial trends

11、E&A
1）Emergency Accident Crisis Response responsibilities as required
Qualifications/ Experience:

1) Degree qualification in Information Technology related discipline
2) At least 3 years of working experience in Big Data Engineering with scrum practices
3) Essential to have strong understanding of Data Warehouse, Data Lakehouse competence in SQL preferably AWS Redshift, LakeFormation, Glue
4) Proficiency in Python, JSON, RESTful API and/web services development
5) Preferably experienced in ETL, Data Catalogue BI tool development (asset curation, lineage, metadata data quality management, ETL, cognos, alteryx, Qliksense, tableau etc.)
6) Understanding of Cloud technologies, DevOps practices, Github, modern data pipeline frameworks (PySpark EMR) Generative AI/LLMs would be added advantage
7) Experience in Unix/ Linux as well as AWS cloud platform is preferable
8) Hands-on implementation experience in building real-time data ingestion transformation workflows using modern streaming technologies is beneficial (Apache Kafka, AWS Kinesis)
9) Good understanding of technology concepts, trends capabilities
10) Ability to listen articulate ideas verbally in written formats to a broad range of audiences; ability to ask probing questions deliver presentations
11) Strong interpersonal skills; promoting healthy relationships team dynamics
12) Well-developed analytical, problem-solving, decision-making skills, leveraging both logic creativity; strong troubleshooting skills; ability to identify patterns generate ideas
13) Ability to complete milestones work toward multiple deadlines simultaneously
14) Proficient in both spoken written English Chinese
15）Familiar with Hong Kong Employeedinance & Employee Compensationdinance is beneficial
16）Candidates with less experience may be considered as Solution Specialist.
个人信息和申请程序：
国泰航空致力推动平等就业机会。申请人提供的个人数据将严格依据公司的《个人信息收集声明》仅供招聘所用。申请人如八周内尚未获取进一步通知，可视为此次申请不成功。我们保留您的信息记录的时间，不会超过我们获取信息的目的和任何其他允许的相关目的所必需的时间。如果您的申请没有成功，我们将把您的详细信息保存在档案中，保存的时间将延长至处理您的申请所需的时间，或者在您同意的情况下，保存更长时间以匹配未来的工作机会。
注意：所有中国内地职位的劳动合同遵循当地条款。
Personal & Application Information:
Cathay Pacific is an Equal Opportunities Employer. Personal data provided by job applicants will be used strictly in accordance with our Applicant Personal Information Collection Statement frecruitment purposes only. Candidates not notified within eight weeks may consider their application unsuccessful. We keep records of your data fno longer than is necessary fthe purpose fwhich we obtained them any other permitted linked purposes. If your application is unsuccessful, we will keep your details on file fas long as is necessary to process your application fthe purposes of further job opportunities if you agree to such longer periods.
Notes: All Chinese mainlbased positions are subject to local terms conditions.