Data Engineering on Public Cloud (SF)
About This Course
This 5-day course equips participants with the practical skills to design and implement scalable, cloud-oriented data engineering workflows using open-source tools. Through hands-on labs and real-world patterns, learners will explore cloud data ingestion, orchestration with Airflow, storage using MinIO and PostgreSQL, and distributed processing via Apache Spark. Participants will build modular ETL pipelines, model relational data, process analytics-ready datasets, and adopt cloud-native formats like Parquet. The course emphasizes a cloud-agnostic approach, making it ideal for hybrid or multi-cloud environments, and concludes with a capstone project that integrates all concepts into a real-world data pipeline.
What You'll Learn
LO1: Design staging databases to store and prepare data prior to integration into target system based on data integration design considerations.
LO2: Design extraction process for consolidating data from multiple data sources based on business and data requirements and their implications.
LO3: Develop and manage data integration procedures to align data availability with business process rules of target systems.
LO4: Translate complex functional and technical requirements into detailed data structures and designs.
LO5: Design data validation methodology to verify reliability and validity of data using best practice methodologies.
LO6: Develop loading processes to deploy transformed and integrated data into the live target system based on ETL design principles.
LO7: Develop data warehouse process models with key design elements and considerations of ETL processes.
LO8: Design integration procedures that transform extracted data into structures aligned with business rules embedded in the target system in accordance to key design elements of its data storage mechanisms.
LO9: Verify extracted data with business rules based on key design elements of data storage mechanisms.
LO10: Develop efficient processes to standardise & maintain data definitions, sources and quality in accordance to business & process rules of target systems.
Entry Requirements
• The candidate should hold a diploma and have at least 2 years of experience working as a developer/programmer.
• Candidates who do not meet the above requirements will be assessed on case to case basis