Azure Data Engineer Course | Azure Data Engineer Online Training

Azure Data Engineer + Databricks Developer

Python
DataBricks
Spark
Pyspark
Spark SQL
Delta Lake
Azure Data Factory
Azure Synapse DW (Dedicated SQL POOL)
Azure ADF & Databricks Projects

Azure Databricks Concepts

Azure Databricks Introduction

Databricks Architecture
Databricks Components overview
Benefits for data engineers and data scientists

Azure Databricks concepts

Workspace – Creation and managing workspace.
Notebook – creating notebooks, calling and managing different notebooks.
Library - installing libraries, managing libraries

Data Management

Databricks File System. - DBFS commands copy and manage files using DBFS.
Database - Creating database, tables and managing databases and tables.
Table - Creating Tables, dropping tables, loading data ..
Metastore - managing metadata and delta tables creation, managing delta tables.

Computation Management

Cluster -- Creating Clusters , managing clusters
Pool - creating pools and using pools for Auto scaling.
Databricks Runtime - understanding and using Databricks runtimes based on requirement.
Jobs - creating jobs from notebooks and assigning types of clusters for jobs.
Workload - monitoring jobs and managing loads.
Execution Context – understanding context.

Databricks Advanced topics.

Databricks Workflows
Calling one notebook into another notebook.
Creating global variables (widgets) and using into Azure ADF pipeline.
How to implement parallelism in notebooks execution.
Mounting azure blob storage and data lake storage accounts.
Integrating source code (notebooks) with GitHub
Calling DataBricks notebooks into Azure Data factory.
Databricks Clusters logs monitoring flow.

SPARK Concepts

Introduction to Spark - Getting started

What is Spark and what is its purpose?
Components of the Spark unified stack
Resilient Distributed Dataset (RDD)
Downloading and installing Spark standalone
Scala and Python overview
Launching and using Spark’s Scala and Python shell ©

Resilient Distributed Dataset and Data Frames

Understand how to create parallelized collections and external datasets
Work with Resilient Distributed Dataset (RDD) operations
Utilize shared variables and key-value pairs

Spark application programming

Understand the purpose and usage of the Spark Context
Initialize Spark with the various programming languages
Describe and run some Spark examples
Pass functions to Spark
Create and run a Spark standalone application
Submit applications to the cluster

Introduction to Spark libraries

Understand and use the various Spark libraries

Spark configuration, monitoring and tuning

Understand components of the Spark cluster
Configure Spark to modify the Spark properties, environmental variables, or logging properties
Understand components of the Spark clusterMonitor Spark using the web UIs, metrics, and external instrumentation ,Understand performance tuning considerations

PySpark Content

Introduction To Pyspark

What is Spark Session
How to create spark session
What is Spark Context
How to create Spark Context
What is SQL Context

How to Use Jupiter Notebooks & Databricks notebooks for Python Development.
Install and configure PySpark in Local System for development.
Introduction to Big Data and Apache Spark
Apache Spark Framework & Execution Process.
Introduction To RDDs

Different Ways to Create RDD’s in Pyspark.
RDD Transformations
RDD Actions
RDD Cache & Persist

Introduction to Data Frame.

Different Ways to Create Data Frame’sin Pyspark.
Dataframe Transformations
Dataframe Actions
Dataframe Cache & Persist

Different types of Big Data File systems.

Difference between Row store format and column store format.
Avro File
Parquet file
ORC File

Reading and Writing Different Types of Files using Dataframe.

CSV files
Json files
Xml files
Excel files
Complex Json files
Avro files
Parquet files
Orc files

Need for Spark SQL

What is Spark SQL
SQL Table Creation
SQL Join Types
SQL Nested Queries
SQL DML Operations
SQL Merge Scripts
SQL SCD Type 2 implementation

User-Defined Functions
Create Private Link to different services
Limitation
What is Azure Data Factory
How to create Data Factory + All Components
Creating Pipeline + Copy Activity
General Activities
Iteration and Conditions
Data Flow Resource + Data Flow Activity
Synapse Activities
Azure Monitor and Manage

DELTA LAKE

Delta Lake usage in Databricks.

Delta Lake Architecture
Delta Lake Storage Understanding
Delta lake table creation and API options
Delta Lake DML Operations usage.
Delta Lake partitions
Delta Lake Schema Enforcement
Delta Lake Schema Evolution
Delta Lake Versions
Delta Lake Time Travel
Delta Lake Vaccum
Delta Lake Merge (SCD Type 1 and SCD Type2)

Azure Data Engineer

Overview of the Microsoft Azure Platform

Introduction to Azure
Basics of Cloud computing
Azure Infrastructure
Walkthrough of Azure Portal
Overview of Azure Services

Azure Data Architecture

Traditional RDBMS workloads.
Data Warehousing Approach
Big data architectures.
Transferring data to and from Azure

Azure Storage options

Blob Storage
ADLS Gen1 & Gen2
RDBMS
Hadoop
NoSQL
Disk

Blob Storage

Azure Blob Resources
Types of Blobs in Azure
Azure storage account data objects
Azure storage account types and Options
Replications in distribution
Secure access to an application's data
Azure Import/Export service
Storage Explorer
Practical section on Blob Storage

Azure Data Factory

Azure Data Factory Architecture
Creating ADF Resource and Use in azure cloud
Pipeline Creation and Usage Options
Copy Data Tool in ADF Portal, Use
Linked Service Creation in ADF
Dataset Creation, Connection Reuse
Staging Dataset with Azure Storage
ADF Pipeline Deployments
Pipeline Orchestration using Triggers
ADF Transformations and other tools integration.
Processing different type’s files using ADF.
Integration Runtime
Monitoring ADF Jobs
Manage IR’s and Linked Services.

Azure SQL Database Service

Introduction to Azure SQL Database
Relational Data Services in the Cloud
Azure SQL Database Service Tiers
Database Throughput Units (DTU)
Scalable performance and pools
Creating and Managing SQL Databases
Azure SQL Database Tools
Migrating data to Azure SQL Database

Azure Data Lake Gen1 & Gen2

Explore the Azure Data Lake enterprise-class security features.
Understand storage account keys.
Understand shared access signatures.
Understand transport-level encryption with HTTPS.
Understand Advanced Threat Protection.
Control network access.
Differences between Gen1 & Gen2

Azure Synapse SQL DW (Dedicated SQLPOOL)

Azure Synapse DW (Dedicated SQL POOL)?
Synapse DW Architecture.
Creating Internal table with default distribution
Creating external table in synapse dw
Loading data from databricks to azure synapse dw
Loading data from adls gen2 to azure synapse dw
What is dedicated sql pool
data warehouse unit overview
Distributed table with example
Hash distribution with example
Round robin distribution with example
Replicate distribution with example
What are the types of indexes with examples
Clustered Index with example
Non-Clustered index with example
Clustered Column Store Index with example
Heap Index with example

SPARK SQL

Introduction to Spark SQL.
Spark SQL Create database
Drop databases
Create internal table
Create external table
Create partitioned table
Create partitioned with bucketing table
SPARK DML insert, update, delete and merge operations
SPARK SQL DRL Select queries with different clauses
Spark SQL MERGE With SCD Type 1 and SCD Type 2
Spark SQL WHERE Clause, Group By Clause and Having Clauses
Spark SQL Order by, Sort By clauses
Spark SQL join types, Window , Pivot , Limit and Like
Spark SQL Grouping Sets, Rollup and Cube
Spark SQL Cultured By and Distributed By
Spark SQL Case, With and Take sample

MS Azure Data Engineering training

What Is Data Engineering?

Why Microsoft Azure Data Engineer?

What does a Azure data engineer do?

Azure Data Engineer + Databricks Developer

Azure Databricks Concepts

SPARK Concepts

PySpark Content

DELTA LAKE

Azure Data Engineer

SPARK SQL