Project- Finding the most popular movies

1. Downloading the Datasets

2. Steps to run a spark project- Finding the most popular movie

It’s a simple project where you can get the feel of how spark uses dataframes and do the manipulation on data as per requirement.

In this project we will try to find out the most popular movie using spark, it’s a basic project which can give you an understanding of how things work in spark.

I will be using Dataframes for the processing and python’s API pyspark. first, we need to download the datasets for our small project.

I am using the dataset given by movielens, you can download it from here https://grouplens.org/datasets/movielens/

dataset…

Basic Intro and Installation on MAC

1. Intro to Spark

2. Spark Installation on Mac

2.1 Spark installation using Brew

2.2 Using Apache Spark binaries

1. Intro to Spark

According to Apache spark there one line definition is “A fast and general engine for large scale data processing”.

That’s actually a good summary of what it’s all about. Spark can manage to process a massive amount of data that can represent anything weblogs, live data or it could be anything.

A high-Level overview of Data processing

Read===>Process===>Write

Spark can distribute the data amongst the cluster of computers and do the processing, remember it’s only an execution engine it has no storage.

Spark can collect data from various sources either real-time or batch processing…

Beginners guide to Apache Airflow

What is Work Flow

A set of steps to accomplish a given Data Engineering task. These can include any given task, such as downloading a file, copying data, filtering information, writing to a database, and so forth.

A workflow is of varying levels of complexity. Some workflows may only have 2 or 3 steps, while others consist of hundreds of components.

What is Airflow?

Airflow is a platform to program workflows (general), including the creation, scheduling, and monitoring of workflows. Airflow implements workflows as DAGs, or Directed Acyclic Graphs.

Airflow can be accessed and controlled via code, via the command-line, or via…

Guide to clear Azure Fundamentals AZ-900 certification exam.

This is part 2 of Azure Fundamentals, it will help you to clear the AZ-900 Exam.

Link of part 1

2.2 Azure Core Services- Networking

2.2.1 Virtual Network

When we create a virtual machine in Azure it has to be a part of the virtual network. An azure virtual network is a home for your virtual machines, in the virtual network we specify the range of IP address and VMs gets the IP address as per our defined IP address range.

We can also do the subnetting of IP address range, subnetting is a logical separation of resources that we…

Guide to clear Azure Fundamentals AZ-900 certification exam.

1. Basic Definitions

• 1.1 What is Cloud Computing
• 1.2 Resources & Resource Group
• 1.3 What is Subscription
• 1.4 High Availability
• 1.5 Cloud Service Models IAAS, PAAS, SAAS

2. Azure Core Services

2.1 Virtual Machines

• 2.1.1 Creating a Virtual Machine
• 2.1.2 Connecting to the VM
• 2.1.3 VM Details- Types, Series
• 2.1.4 Deleting a VM
• 2.1.5 Availability Sets
• 2.1.6 Availability Zones

2.2 Networking

• 2.2.1 Virtual Network
• 2.2.2 Network Security Groups
• 2.2.3 Application Security Groups

2.3 Storage

• 2.3.1 Creating a Storage account
• 2.3.2 Types of Storage Accounts
• 2.3.3 Service offered by Storage Accounts
• 2.3.4 Storage Accounts- Replications
• 2.3.5 Access Tiers
• 2.3.6 Working with the blob service

3. More Azure Features

• 3.1 Load Balancer
• …

In this post you will learn about how to restore a PostgreSQL database in Point in time.

Open source databases are taking over other conventional Database softwares at very high speed. For me personally, I really like the way PostgreSQL is designed and it’s simple architecture.

I have tried to write it down every possible steps which will help you to restore your Postgres database.

Below Steps will be same, but few commands may vary with any other backup tool. PITR PostgreSQL

• Allow you to restore database to a specific moment in time PITR.
• It make use of live database…

Multiple ways of installation of PostgreSQL database

In this article, I will try to help you with the easiest ways of installation PostgreSQL database.

PostgreSQL (or postgres) is open-source free to use relational database management system (RDBMS). It has reliability, feature robustness, and performance.

PostgreSQL installation

1. Interactive installation by Software EDB
2. Yum Repository Based Installation
3. USING .RPM FILES

1. Interactive installation by Software EDB

EDB is a Company which provides paid support for postgreSQL database, and the paid version of postgreSQL DB is called as EDB advance server.

Here we are interested in downloading the open-source/free version so you can download the DB software from EDB official site…

#Hands on implementation in Python to find out Churn Rate

• What is Churn rate?
• Which Dataset are we using?
• Python implementation

What is Churn rate?

“The annual percentage rate at which customers stop subscribing to a service or employees leave a job.” — Google search result