[align=center]
Data Engineering Using Aws Data Analytics
Last updated 2/2023
Created by Durga Viswanatha Raju Gadiraju,Pratik Kumar,Sathvika Dandu,Madhuri Gadiraju,Sai Varma,Phani Bhushan Bozzam
MP4 | Video: h264, 1280x720 | Audio: AAC, 44.1 KHz, 2 Ch
Genre: eLearning | Language: English + subtitle | Duration: 434 Lectures ( 26h 15m ) | Size: 7.77 GB[/center]

Build Data Engineering Pipelines on AWS using Data Analytics Services - Glue, EMR, Athena, Kinesis, Lambda, Redshift
What you'll learn
Data Engineering leveraging Services under AWS Data Analytics
AWS Essentials such as s3, IAM, EC2, etc
Understanding AWS s3 for cloud based storage
Understanding details related to virtual machines on AWS known as EC2
Managing AWS IAM users, groups, roles and policies for RBAC (Role Based Access Control)
Managing Tables using AWS Glue Catalog
Engineering Batch Data Pipelines using AWS Glue Jobs
Orchestrating Batch Data Pipelines using AWS Glue Workflows
Running Queries using AWS Athena - Server less query engine service
Using AWS Elastic Map Reduce (EMR) Clusters for building Data Pipelines
Using AWS Elastic Map Reduce (EMR) Clusters for reports and dashboards
Data Ingestion using AWS Lambda Functions
Scheduling using AWS Events Bridge
Engineering Streaming Pipelines using AWS Kinesis
Streaming Web Server logs using AWS Kinesis Firehose
Overview of data processing using AWS Athena
Running AWS Athena queries or commands using CLI
Running AWS Athena queries using Python boto3
Creating AWS Redshift Cluster, Create tables and perform CRUD Operations
Copy data from s3 to AWS Redshift Tables
Understanding Distribution Styles and creating tables using Distkeys
Running queries on external RDBMS Tables using AWS Redshift Federated Queries
Running queries on Glue or Athena Catalog tables using AWS Redshift Spectrum
Requirements
A Computer with at least 8 GB RAM
Programming Experience using Python is highly desired as some of the topics are demonstrated using Python
SQL Experience is highly desired as some of the topics are demonstrated using SQL
Nice to have Data Engineering Experience using Pandas or Pyspark
This course is ideal for experienced data engineers to add AWS Analytics Services as key skills to their profile
Description
Data Engineering is all about building Data Pipelines to get data from multiple sources into Data Lakes or Data Warehouses and then from Data Lakes or Data Warehouses to downstream systems. As part of this course, I will walk you through how to build Data Engineering Pipelines using AWS Data Analytics Stack. It includes services such as Glue, Elastic Map Reduce (EMR), Lambda Functions, Athena, EMR, Kinesis, and many more.Here are the high-level steps which you will follow as part of the course.Setup Development EnvironmentGetting Started with AWSStorage - All about AWS s3 (Simple Storage Service)User Level Security - Managing Users, Roles, and Policies using IAMInfrastructure - AWS EC2 (Elastic Cloud Compute)Data Ingestion using AWS Lambda FunctionsOverview of AWS Glue ComponentsSetup Spark History Server for AWS Glue JobsDeep Dive into AWS Glue CatalogExploring AWS Glue Job APIsAWS Glue Job BookmarksDevelopment Life Cycle of PysparkGetting Started with AWS EMRDeploying Spark Applications using AWS EMRStreaming Pipeline using AWS KinesisConsuming Data from AWS s3 using boto3 ingested using AWS KinesisPopulating GitHub Data to AWS DynamodbOverview of Amazon AWS AthenaAmazon AWS Athena using AWS CLIAmazon AWS Athena using Python boto3Getting Started with Amazon AWS RedshiftCopy Data from AWS s3 into AWS Redshift TablesDevelop Applications using AWS Redshift ClusterAWS Redshift Tables with Distkeys and SortkeysAWS Redshift Federated Queries and SpectrumHere are the details about what you will be learning as part of this course. We will cover most of the commonly used services with hands-on practice which are available under AWS Data Analytics.Getting Started with AWSAs part of this section, you will be going through the details related to getting started with AWS.Introduction - AWS Getting StartedCreate s3 BucketCreate AWS IAM Group and AWS IAM User to have required access on s3 Bucket and other servicesOverview of AWS IAM RolesCreate and Attach Custom AWS IAM Policy to both AWS IAM Groups as well as UsersConfigure and Validate AWS CLI to access AWS Services using AWS CLI CommandsStorage - All about AWS s3 (Simple Storage Service)AWS s3 is one of the most prominent fully managed AWS services. All IT professionals who would like to work on AWS should be familiar with it. We will get into quite a few common features related to AWS s3 in this section.Getting Started with AWS S3Setup Data Set locally to upload to AWS s3Adding AWS S3 Buckets and Managing Objects (files and folders) in AWS s3 bucketsVersion Control for AWS S3 BucketsCross-Region Replication for AWS S3 BucketsOverview of AWS S3 Storage ClassesOverview of AWS S3 GlacierManaging AWS S3 using AWS CLI CommandsManaging Objects in AWS S3 using CLI - LabUser Level Security - Managing Users, Roles, and Policies using IAMOnce you start working on AWS, you need to understand the permissions you have as a non-admin user. As part of this section, you will understand the details related to AWS IAM users, groups, roles as well as policies.Creating AWS IAM UsersLogging into AWS Management Console using AWS IAM UserValidate Programmatic Access to AWS IAM UserAWS IAM Identity-based PoliciesManaging AWS IAM GroupsManaging AWS IAM RolesOverview of Custom AWS IAM PoliciesManaging AWS IAM users, groups, roles as well as policies using AWS CLI CommandsInfrastructure - AWS EC2 (Elastic Cloud Compute) BasicsAWS EC2 Instances are nothing but virtual machines on AWS. As part of this section, we will go through some of the basics related to AWS EC2 Basics.Getting Started with AWS EC2Create AWS EC2 Key PairLaunch AWS EC2 InstanceConnecting to AWS EC2 InstanceAWS EC2 Security Groups BasicsAWS EC2 Public and Private IP AddressesAWS EC2 Life CycleAllocating and Assigning AWS Elastic IP AddressManaging AWS EC2 Using AWS CLIUpgrade or Downgrade AWS EC2 InstancesInfrastructure - AWS EC2 AdvancedIn this section, we will continue with AWS EC2 to understand how we can manage EC2 instances using AWS Commands and also how to install additional OS modules leveraging bootstrap scripts.Getting Started with AWS EC2Understanding AWS EC2 MetadataQuerying on AWS EC2 MetadataFitering on AWS EC2 MetadataUsing Bootstrapping Scripts with AWS EC2 Instances to install additional softwares on AWS EC2 instancesCreate an AWS AMI using AWS EC2 InstancesValidate AWS AMI - LabData Ingestion using Lambda FunctionsAWS Lambda functions are nothing but serverless functions. In this section, we will understand how we can develop and deploy Lambda functions using Python as a programming language. We will also see how to maintain a bookmark or checkpoint using s3.Hello World using AWS LambdaSetup Project for local development of AWS Lambda FunctionsDeploy Project to AWS Lambda consoleDevelop download скачать functionality using requests for AWS Lambda FunctionsUsing 3rd party libraries in AWS Lambda FunctionsValidating AWS s3 access for local development of AWS Lambda FunctionsDevelop upload functionality to s3 using AWS Lambda FunctionsValidating AWS Lambda Functions using AWS Lambda ConsoleRun AWS Lambda Functions using AWS Lambda ConsoleValidating files incrementally downloaded using AWS Lambda FunctionsReading and Writing Bookmark to s3 using AWS Lambda FunctionsMaintaining Bookmark on s3 using AWS Lambda FunctionsReview the incremental upload logic developed using AWS Lambda FunctionsDeploying AWS Lambda FunctionsSchedule AWS Lambda Functions using AWS Event BridgeOverview of AWS Glue ComponentsIn this section, we will get a broad overview of all important Glue Components such as Glue Crawler, Glue Databases, Glue Tables, etc. We will also understand how to validate Glue tables using AWS Athena. AWS Glue (especially Glue Catalog) is one of the key components in the realm of AWS Data Analytics Services.Introduction - Overview of AWS Glue ComponentsCreate AWS Glue Crawler and AWS Glue Catalog Database as well as TableAnalyze Data using AWS AthenaCreating AWS S3 Bucket and Role to create AWS Glue Catalog Tables using Crawler on the s3 locationCreate and Run the AWS Glue Job to process data in AWS Glue Catalog TablesValidate using AWS Glue Catalog Table and by running queries using AWS AthenaCreate and Run AWS Glue TriggerCreate AWS Glue WorkflowRun AWS Glue Workflow and ValidateSetup Spark History Server for AWS Glue JobsAWS Glue uses Apache Spark under the hood to process the data. It is important we setup Spark History Server for AWS Glue Jobs to troubleshoot any issues.Introduction - Spark History Server for AWS GlueSetup Spark History Server on AWSClone AWS Glue Samples repositoryBuild AWS Glue Spark UI ContainerUpdate AWS IAM Policy PermissionsStart AWS Glue Spark UI ContainerDeep Dive into AWS Glue CatalogAWS Glue has several components, but the most important ones are nothing but AWS Glue Crawlers, Databases as well as Catalog Tables. In this section, we will go through some of the most important and commonly used features of the AWS Glue Catalog.Prerequisites for AWS Glue Catalog TablesSteps for Creating AWS Glue Catalog TablesDownload Data Set to use to create AWS Glue Catalog TablesUpload data to s3 to crawl using AWS Glue Crawler to create required AWS Glue Catalog TablesCreate AWS Glue Catalog Database - itvghlandingdbCreate AWS Glue Catalog Table - ghactivityRunning Queries using AWS Athena - ghactivityCrawling Multiple Folders using AWS Glue CrawlersManaging AWS Glue Catalog using AWS CLIManaging AWS Glue Catalog using Python Boto3Exploring AWS Glue Job APIsOnce we deploy AWS Glue jobs, we can manage them using AWS Glue Job APIs. In this section we will get overview of AWS Glue Job APIs to run and manage the jobs.Update AWS IAM Role for AWS Glue JobGenerate baseline AWS Glue JobRunning baseline AWS Glue JobAWS Glue Script for Partitioning DataValidating using AWS AthenaUnderstanding AWS Glue Job BookmarksAWS Glue Job Bookmarks can be leveraged to maintain the bookmarks or checkpoints for incremental loads. In this section, we will go through the details related to AWS Glue Job Bookmarks.Introduction to AWS Glue Job BookmarksCleaning up the data to run AWS Glue JobsOverview of AWS Glue CLI and CommandsRun AWS Glue Job using AWS Glue BookmarkValidate AWS Glue Bookmark using AWS CLIAdd new data to the landing zone to run AWS Glue Jobs using BookmarksRerun AWS Glue Job using BookmarkValidate AWS Glue Job Bookmark and Files for Incremental runRecrawl the AWS Glue Catalog Table using AWS CLI CommandsRun AWS Athena Queries for Data ValidationDevelopment Lifecycle for PysparkIn this section, we will focus on the development of Spark applications using Pyspark. We will use this application later while exploring EMR in detail.Setup Virtual Environment and Install PysparkGetting Started with PycharmPassing Run Time ArgumentsAccessing OS Environment VariablesGetting Started with SparkCreate Function for Spark SessionSetup Sample DataRead data from filesProcess data using Spark APIsWrite data to filesValidating Writing Data to FilesProductionizing the CodeGetting Started with AWS EMR (Elastic Map Reduce)As part of this section, we will understand how to get started with AWS EMR Cluster. We will primarily focus on AWS EMR Web Console. Elastic Map Reduce is one of the key service in AWS Data Analytics Services which provide capability to run applications which process large scale data leveraging distributed computing frameworks such as Spark.Planning for AWS EMR ClusterCreate AWS EC2 Key Pair for AWS EMR ClusterSetup AWS EMR Cluster with Apache SparkUnderstanding Summary of AWS EMR ClusterReview AWS EMR Cluster Application User InterfacesReview AWS EMR Cluster MonitoringReview AWS EMR Cluster Hardware and Cluster Scaling PolicyReview AWS EMR Cluster ConfigurationsReview AWS EMR Cluster EventsReview AWS EMR Cluster StepsReview AWS EMR Cluster Bootstrap ActionsConnecting to AWS EMR Master Node using SSHDisabling Termination Protection for AWS EMR Cluster and Terminating the AWS EMR ClusterClone and Create a New AWS EMR ClusterListing AWS S3 Buckets and Objects using AWS CLI on AWS EMR ClusterListing AWS S3 Buckets and Objects using HDFS CLI on AWS EMR ClusterManaging Files in AWS S3 using HDFS CLI on AWS EMR ClusterReview AWS Glue Catalog Databases and TablesAccessing AWS Glue Catalog Databases and Tables using AWS EMR ClusterAccessing spark-sql CLI of AWS EMR ClusterAccessing pyspark CLI of AWS EMR ClusterAccessing spark-shell CLI of AWS EMR ClusterCreate AWS EMR Cluster for NotebooksDeploying Spark Applications using AWS EMRAs part of this section, we will understand how we typically deploy Spark Applications using AWS EMR. We will be using the Spark Application we deployed earlier.Deploying Applications using AWS EMR - IntroductionSetup AWS EMR Cluster to deploy applicationsValidate SSH Connectivity to Master node of AWS EMR ClusterSetup Jupyter Notebook Environment on AWS EMR ClusterCreate required AWS s3 Bucket for AWS EMR ClusterUpload GHActivity Data to s3 so that we can process using Spark Application deployed on AWS EMR ClusterValidate Application using AWS EMR Compatible Versions of Python and SparkDeploy Spark Application to AWS EMR Master NodeCreate user space for ec2-user on AWS EMR ClusterRun Spark Application using spark-submit on AWS EMR Master NodeValidate Data using Jupyter Notebooks on AWS EMR ClusterClone and Start Auto Terminated AWS EMR ClusterDelete Data Populated by GHAcitivity Application using AWS EMR ClusterDifferences between Spark Client and Cluster Deployment Modes on AWS EMR ClusterRunning Spark Application using Cluster Mode on AWS EMR ClusterOverview of Adding Pyspark Application as Step to AWS EMR ClusterDeploy Spark Application to AWS S3 to run using AWS EMR StepsRunning Spark Applications as AWS EMR Steps in client modeRunning Spark Applications as AWS EMR Steps in cluster modeValidate AWS EMR Step Execution of Spark ApplicationStreaming Data Ingestion Pipeline using AWS KinesisAs part of this section, we will go through details related to the streaming data ingestion pipeline using AWS Kinesis which is a streaming service of AWS Data Analytics Services. We will use AWS Kinesis Firehose Agent and AWS Kinesis Delivery Stream to read the data from log files and ingest it into AWS s3.Building Streaming Pipeline using AWS Kinesis Firehose Agent and Delivery StreamRotating Logs so that the files are created frequently which will be eventually ingested using AWS Kinesis Firehose Agent and AWS Kinesis Firehose Delivery StreamSet up AWS Kinesis Firehose Agent to get data from logs into AWS Kinesis Delivery Stream.Create AWS Kinesis Firehose Delivery StreamPlanning the Pipeline to ingest data into s3 using AWS Kinesis Delivery StreamCreate AWS IAM Group and User for Streaming Pipelines using AWS Kinesis ComponentsGranting Permissions to AWS IAM User using Policy for Streaming Pipelines using AWS Kinesis ComponentsConfigure AWS Kinesis Firehose Agent to read the data from log files and ingest it into AWS Kinesis Firehose Delivery Stream.Start and Validate AWS Kinesis Firehose AgentConclusion - Building Simple Steaming Pipeline using AWS Kinesis FirehoseConsuming Data from AWS s3 using Python boto3 ingested using AWS KinesisAs data is ingested into AWS S3, we will understand how data can ingested in AWS s3 can be processed using boto3.Customizing AWS s3 folder using AWS Kinesis Delivery StreamCreate AWS IAM Policy to read from AWS s3 BucketValidate AWS s3 access using AWS CLISetup Python Virtual Environment to explore boto3Validating access to AWS s3 using Python boto3Read Content from AWS s3 objectRead multiple AWS s3 ObjectsGet the number of AWS s3 Objects using MarkerGet the size of AWS s3 Objects using MarkerPopulating GitHub Data to AWS DynamodbAs part of this section, we will understand how we can populate data to AWS Dynamodb tables using Python as a programming language.Install required libraries to get GitHub Data to AWS Dynamodb tables.Understanding GitHub APIsSetting up GitHub API TokenUnderstanding GitHub Rate LimitCreate New Repository for sinceExtracting Required Information using PythonProcessing Data using PythonGrant Permissions to create AWS dynamodb tables using boto3Create AWS Dynamodb TablesAWS Dynamodb CRUD OperationsPopulate AWS Dynamodb TableAWS Dynamodb Batch OperationsOverview of Amazon AWS AthenaAs part of this section, we will understand how to get started with AWS Athena using AWS Web console. We will also focus on basic DDL and DML or CRUD Operations using AWS Athena Query Editor.Getting Started with Amazon AWS AthenaQuick Recap of AWS Glue Catalog Databases and TablesAccess AWS Glue Catalog Databases and Tables using AWS Athena Query EditorCreate a Database and Table using AWS AthenaPopulate Data into Table using AWS AthenaUsing CTAS to create tables using AWS AthenaOverview of Amazon AWS Athena ArchitectureAmazon AWS Athena Resources and relationship with HiveCreate a Partitioned Table using AWS AthenaDevelop Query for Partitioned ColumnInsert into Partitioned Tables using AWS AthenaValidate Data Partitioning using AWS AthenaDrop AWS Athena Tables and Delete Data FilesDrop Partitioned Table using AWS AthenaData Partitioning in AWS Athena using CTASAmazon AWS Athena using AWS CLIAs part of this section, we will understand how to interact with AWS Athena using AWS CLI Commands.Amazon AWS Athena using AWS CLI - IntroductionGet help and list AWS Athena databases using AWS CLIManaging AWS Athena Workgroups using AWS CLIRun AWS Athena Queries using AWS CLIGet AWS Athena Table Metadata using AWS CLIRun AWS Athena Queries with a custom location using AWS CLIDrop AWS Athena table using AWS CLIRun CTAS under AWS Athena using AWS CLIAmazon AWS Athena using Python boto3As part of this section, we will understand how to interact with AWS Athena using Python boto3.Amazon AWS Athena using Python boto3 - IntroductionGetting Started with Managing AWS Athena using Python boto3List Amazon AWS Athena Databases using Python boto3List Amazon AWS Athena Tables using Python boto3Run Amazon AWS Athena Queries with boto3Review AWS Athena Query Results using boto3Persist Amazon AWS Athena Query Results in Custom Location using boto3Processing AWS Athena Query Results using PandasRun CTAS against Amazon AWS Athena using Python boto3Getting Started with Amazon AWS RedshiftAs part of this section, we will understand how to get started with AWS Redshift using AWS Web console. We will also focus on basic DDL and DML or CRUD Operations using AWS Redshift Query Editor.Getting Started with Amazon AWS Redshift - IntroductionCreate AWS Redshift Cluster using Free TrialConnecting to Database using AWS Redshift Query EditorGet a list of tables querying information schemaRun Queries against AWS Redshift Tables using Query EditorCreate AWS Redshift Table using Primary KeyInsert Data into AWS Redshift TablesUpdate Data in AWS Redshift TablesDelete data from AWS Redshift tablesRedshift Saved Queries using Query EditorDeleting AWS Redshift ClusterRestore AWS Redshift Cluster from SnapshotCopy Data from s3 into AWS Redshift TablesAs part of this section, we will go through the details about copying data from s3 into AWS Redshift tables using the AWS Redshift Copy command.Copy Data from s3 to AWS Redshift - IntroductionSetup Data in s3 for AWS Redshift CopyCopy Database and Table for AWS Redshift Copy CommandCreate IAM User with full access on s3 for AWS Redshift CopyRun Copy Command to copy data from s3 to AWS Redshift TableTroubleshoot Errors related to AWS Redshift Copy CommandRun Copy Command to copy from s3 to AWS Redshift tableValidate using queries against AWS Redshift TableOverview of AWS Redshift Copy CommandCreate IAM Role for AWS Redshift to access s3Copy Data from s3 to AWS Redshift table using IAM RoleSetup JSON Dataset in s3 for AWS Redshift Copy CommandCopy JSON Data from s3 to AWS Redshift table using IAM RoleDevelop Applications using AWS Redshift ClusterAs part of this section, we will understand how to develop applications against databases and tables created as part of AWS Redshift Cluster.Develop application using AWS Redshift Cluster - IntroductionAllocate Elastic Ip for AWS Redshift ClusterEnable Public Accessibility for AWS Redshift ClusterUpdate Inbound Rules in Security Group to access AWS Redshift ClusterCreate Database and User in AWS Redshift ClusterConnect to the database in AWS Redshift using psqlChange Owner on AWS Redshift TablesDownload AWS Redshift JDBC Jar fileConnect to AWS Redshift Databases using IDEs such as SQL WorkbenchSetup Python Virtual Environment for AWS RedshiftRun Simple Query against AWS Redshift Database Table using PythonTruncate AWS Redshift Table using PythonCreate IAM User to copy from s3 to AWS Redshift TablesValidate Access of IAM User using Boto3Run AWS Redshift Copy Command using PythonAWS Redshift Tables with Distkeys and SortkeysAs part of this section, we will go through AWS Redshift-specific features such as distribution keys and sort keys to create AWS Redshift tables.AWS Redshift Tables with Distkeys and Sortkeys - IntroductionQuick Review of AWS Redshift ArchitectureCreate multi-node AWS Redshift ClusterConnect to AWS Redshift Cluster using Query EditorCreate AWS Redshift DatabaseCreate AWS Redshift Database UserCreate AWS Redshift Database SchemaDefault Distribution Style of AWS Redshift TableGrant Select Permissions on Catalog to AWS Redshift Database UserUpdate Search Path to query AWS Redshift system tablesValidate AWS Redshift table with DISTSTYLE AUTOCreate AWS Redshift Cluster from Snapshot to the original stateOverview of Node Slices in AWS Redshift ClusterOverview of Distribution Styles related to AWS Redshift tablesDistribution Strategies for retail tables in AWS Redshift DatabasesCreate AWS Redshift tables with distribution style allTroubleshoot and Fix Load or Copy ErrorsCreate AWS Redshift Table with Distribution Style AutoCreate AWS Redshift Tables using Distribution Style KeyDelete AWS Redshift Cluster with a manual snapshotAWS Redshift Federated Queries and SpectrumAs part of this section, we will go through some of the advanced features of Redshift such as AWS Redshift Federated Queries and AWS Redshift Spectrum.AWS Redshift Federated Queries and Spectrum - IntroductionOverview of integrating AWS RDS and AWS Redshift for Federated QueriesCreate IAM Role for AWS Redshift ClusterSetup Postgres Database Server for AWS Redshift Federated QueriesCreate tables in Postgres Database for AWS Redshift Federated QueriesCreating Secret using Secrets Manager for Postgres DatabaseAccessing Secret Details using Python Boto3Reading Json Data to Dataframe using PandasWrite JSON Data to AWS Redshift Database Tables using PandasCreate AWS IAM Policy for Secret and associate with Redshift RoleCreate AWS Redshift Cluster using AWS IAM Role with permissions on secretCreate AWS Redshift External Schema to Postgres DatabaseUpdate AWS Redshift Cluster Network Settings for Federated QueriesPerforming ETL using AWS Redshift Federated QueriesClean up resources added for AWS Redshift Federated QueriesGrant Access on AWS Glue Data Catalog to AWS Redshift Cluster for SpectrumSetup AWS Redshift Clusters to run queries using SpectrumQuick Recap of AWS Glue Catalog Database and Tables for AWS Redshift SpectrumCreate External Schema using AWS Redshift SpectrumRun Queries using AWS Redshift SpectrumCleanup the AWS Redshift Cluster
Who this course is for
Beginner or Intermediate Data Engineers who want to learn AWS Analytics Services for Data Engineering
Intermediate Application Engineers who want to explore Data Engineering using AWS Analytics Services
Data and Analytics Engineers who want to learn Data Engineering using AWS Analytics Services
Testers who want to learn key skills to test Data Engineering applications built using AWS Analytics Services
Homepage