AWS Data Engineering with Data Analytics Training

An Introduction to Data Engineering

The rise of big data as a corporate asset
The challenges of ever-growing datasets
Data engineers – the big data enablers
Understanding the role of the data engineer
Understanding the role of the data scientist
Understanding the role of the data analyst
Understanding other common data-related roles
The benefits of the cloud when building big data analytic solutions

Data Management Architectures for Analytics

The evolution of data management for analytics
Databases and data warehouses
Dealing with big, unstructured data
A lake on the cloud and a house on that lake
Understanding data warehouses and data marts –fountains of truth
Distributed storage and massively parallel processing
Columnar data storage and efficient data compression
Dimensional modeling in data warehouses
Understanding the role of data marts
Feeding data into the warehouse – ETL and ELT pipelines
Building data lakes to tame the variety and volume of big data
Data lake logical architecture
Bringing together the best of both worlds with the lake house architecture
Data lakehouse implementations
Building a data lakehouse on AWS
Hands-on – configuring the AWS
Command Line Interface tool and creating an S3 bucket
Installing and configuring the AWS CLI
Creating a new Amazon S3 bucket

The AWS Data Engineer's Toolkit

AWS services for ingesting data

Overview of Amazon Database Migration Service (DMS)
Overview of Amazon Kinesis for streaming data ingestion
Overview of Amazon MSK for streaming data ingestion
Overview of Amazon AppFlow for ingesting data from SaaS services
Overview of Amazon Transfer Family for ingestion using FTP/SFTP protocols
Overview of Amazon DataSync for ingesting from on-premises storage
Overview of the AWS Snow family of devices for large data transfers

AWS services for transforming data

Overview of AWS Lambda for light transformations
Overview of AWS Glue for serverless Spark processing
Overview of Amazon EMR for Hadoop ecosystem processing

AWS services for orchestrating big data pipelines

Overview of AWS Glue workflows for orchestrating Glue components
Overview of AWS Step Functions for complex workflows
Overview of Amazon managed workflows for Apache Airflow

AWS services for consuming data

Overview of Amazon Athena for SQL queries in the data lake
Overview of Amazon Redshift and Redshift Spectrum for data warehousing and data lakehouse architectures
Overview of Amazon Quick Sight for visualizing data

Hands-on – triggering an AWS Lambda function when a new file arrives in an S3 bucket

Creating a Lambda layer containing the AWS Data Wrangler library
Creating new Amazon S3 buckets
Creating an IAM policy and role for your Lambda function
Creating a Lambda function
Configuring our Lambda function to be triggered by an S3 upload

Data Cataloging, Security, and Governance

Getting data security and governance right
Common data regulatory requirements
Core data protection concepts
Personal data
Encryption
Anonymized data
Pseudonymized data/tokenization
Authentication
Authorization

Cataloging your data to avoid the data swamp

How to avoid the data swamp

The AWS Glue/Lake Formation data catalog

AWS services for data encryption and security monitoring

AWS Key Management Service (KMS)
Amazon Macie
Amazon GuardDuty

AWS services for managing identity and permissions

AWS Identity and Access Management (IAM) service
Using AWS Lake Formation to manage data lake access

Hands-on – configuring Lake Formation permissions

Creating a new user with IAM permissions
Transitioning to managing fine-grained permissions with AWS Lake Formation
Architecting and Implementing Data Lakes and Data Lake Houses
Architecting Data Engineering Pipelines
Approaching the data pipeline architecture
Architecting houses and architecting pipelines
Conducting a whiteboarding session
Identifying data consumers and understanding their requirements
Identifying data sources and ingesting data
Identifying data transformations and optimizations
File format optimizations
Data standardization
Data quality checks
Data partitioning
Data denormalization
Data cataloging
Loading data into data marts
Wrapping up the whiteboarding session

Ingesting Batch and Streaming Data

Understanding data sources
Data variety
Data volume
Data velocity
Data veracity
Data value
Questions to ask
Ingesting data from a relational database
AWS Database Migration Service (DMS)
AWS Glue
Other ways to ingest data from a database
Deciding on the best approach for ingesting from a database

Ingesting streaming data

Amazon Kinesis versus Amazon
Managed Streaming for Kafka (MSK)
Hands-on – ingesting data with AWS DMS
Creating a new MySQL database instance
Loading the demo data using an Amazon EC2 instance
Creating an IAM policy and role for DMS
Configuring DMS settings and performing a full load from MySQL to S3
Querying data with Amazon Athena
Hands-on – ingesting streaming data
Configuring Kinesis Data Firehose for streaming delivery to Amazon S3
Configuring Amazon Kinesis Data Generator (KDG)
Adding newly ingested data to the Glue Data Catalog
Querying the data with Amazon Athena

Transforming Data to Optimize for Analytics

Technical requirements
Transformations – making raw data more valuable
Cooking, baking, and data transformations
Transformations as part of a pipeline
Types of data transformation tools
Apache Spark
Hadoop and MapReduce
SQL
GUI-based tools
Data preparation transformations
Protecting PII data
Optimizing the file format
Optimizing with data partitioning
Data cleansing
Business use case transforms
Data denormalization
Enriching data
Pre-aggregating data
Extracting metadata from unstructured data
Working with change data capture (CDC) data
Traditional approaches – data upserts and SQL views v
Modern approaches – the transactional data lake
Hands-on – joining datasets with AWS Glue Studio
Creating a new data lake zone – the curated zone
Creating a new IAM role for the Glue job
Configuring a denormalization transform using AWS Glue Studio
Finalizing the denormalization transform job to write to S3
Create a transform job to join streaming and film data using AWS Glue Studio

Identifying and Enabling Data Consumers

Understanding the impact of data democratization
A growing variety of data consumers
Meeting the needs of business users with data visualization
AWS tools for business users
Meeting the needs of data analysts with structured reporting
AWS tools for data analysts
Meeting the needs of data scientists and ML models
AWS tools used by data scientists to work with data
Hands-on – creating data transformations with AWS Glue DataBrew
Configuring new datasets for AWS Glue DataBrew
Creating a new Glue DataBrew project 2
Building your Glue DataBrew recipe
Creating a Glue DataBrew job

Loading Data into a Data Mart

Extending analytics with data warehouses/data marts
Cold data
Warm data
Hot data
What not to do – anti-patterns for a data warehouse
Using a data warehouse as a transactional datastore
Using a data warehouse as a data lake
Using data warehouses for real-time, record-level use cases
Storing unstructured data

Redshift architecture review and storage deep dive

Data distribution across slices
Redshift Zone Maps and sorting data
Designing a high-performance data warehouse
Selecting the optimal Redshift node type
Selecting the optimal table distribution style and sort key
Selecting the right data type for columns
Selecting the optimal table type
Moving data between a data lake and Redshift
Optimizing data ingestion in Redshift
Exporting data from Redshift to the data lake
Hands-on – loading data into an Amazon Redshift cluster and running queries Uploading our sample data to Amazon S3
IAM roles for Redshift
Creating a Redshift cluster
Creating external tables for querying data in S3
Creating a schema for a local Redshift table
Running complex SQL queries against our data

Orchestrating the Data Pipeline

Understanding the core concepts for pipeline orchestration
What is a data pipeline, and how do you orchestrate it?
How do you trigger a data pipeline to run?
How do you handle the failures of a step in your pipeline?
Examining the options for orchestrating pipelines in AWS
AWS Data Pipeline for managing ETL between data sources
AWS Glue Workflows to orchestrate Glue resources
Apache Airflow as an open-source orchestration solution
Pros and cons of using MWAA
AWS Step Function for a serverless orchestration solution
Pros and cons of using AWS Step Function
Deciding on which data pipeline orchestration tool to use
Hands-on – orchestrating a data pipeline using AWS Step Function
Creating new Lambda functions
Creating an SNS topic and subscribing to an email address
Creating a new Step Function state machine
Configuring AWS CloudTrail and Amazon Event Bridge

Ad Hoc Queries with Amazon Athena

Amazon Athena – in-place SQL analytics for the data lake
Tips and tricks to optimize Amazon Athena queries
Common file format and layout optimizations
Writing optimized SQL queries
Federating the queries of external data sources with Amazon Athena Query Federation
Querying external data sources using Athena Federated Query
Managing governance and costs with Amazon Athena Workgroups
Athena Workgroups overview
Enforcing settings for groups of users
Enforcing data usage controls
Hands-on – creating an Amazon Athena workgroup and configuring Athena settings
Hands-on – switching Workgroups and running queries

Visualizing Data with Amazon Quick Sight

Representing data visually for maximum impact

Benefits of data visualization
Popular uses of data visualizations

Understanding Amazon Quick Sight’s core concepts

Standard versus enterprise edition
SPICE – the in-memory storage and computation engine for Quick Sight

Ingesting and preparing data from a variety of sources

Preparing datasets in Quick Sight versus performing ETL outside of Quick Sight

Creating and sharing visuals with Quick Sight analyses and dashboards

Visual types in Amazon Quick Sight

AWS Data Engineering with Data Analytics Training

AWS Data Engineering with Data Analytics Course Overview

An Introduction to Data Engineering

Data Management Architectures for Analytics

The AWS Data Engineer's Toolkit

AWS services for ingesting data

AWS services for transforming data

AWS services for orchestrating big data pipelines

AWS services for consuming data

Hands-on – triggering an AWS Lambda function when a new file arrives in an S3 bucket

Data Cataloging, Security, and Governance

Cataloging your data to avoid the data swamp

The AWS Glue/Lake Formation data catalog

AWS services for data encryption and security monitoring

AWS services for managing identity and permissions

Hands-on – configuring Lake Formation permissions

Ingesting Batch and Streaming Data

Ingesting streaming data

Transforming Data to Optimize for Analytics

Identifying and Enabling Data Consumers

Loading Data into a Data Mart

Redshift architecture review and storage deep dive

Orchestrating the Data Pipeline

Ad Hoc Queries with Amazon Athena

Visualizing Data with Amazon Quick Sight

Representing data visually for maximum impact

Understanding Amazon Quick Sight’s core concepts

Ingesting and preparing data from a variety of sources

Creating and sharing visuals with Quick Sight analyses and dashboards