Understading Apache Spark Architecture

Looking to process and analyze large amounts of data efficiently? Look no further than Apache Spark architecture.

[{"selector":"#anim-bad4a4c7-ee34-4dd4-afb4-eb2d86913de8 [data-leaf-element=\"true\"]","keyframes":{"transform":["translate(0%, 0%) scale(1.5)","translate(0%, 0%) scale(1)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"forwards"}] [{"selector":"#anim-53153739-eca9-49cd-8451-77e34b9e3730","keyframes":{"opacity":[0,1]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"both"}] [{"selector":"#anim-d7e8d4c3-2a82-4f62-9ba4-0b96c48cc26a","keyframes":{"transform":["scale(0.3333333333333333)","scale(1)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"forwards"}] [{"selector":"#anim-063977cf-6cc3-476d-8b45-95567c31aaff","keyframes":{"opacity":[0,1]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"both"}] [{"selector":"#anim-24322cdd-69d4-4b61-b51a-a6e8f96f40d5","keyframes":{"transform":["scale(0.3333333333333333)","scale(1)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"forwards"}] Ready to explore Spark archtecture? Click here

What is Apache Spark?

[{"selector":"#anim-732f32ee-d1c1-4a2e-88eb-428b44fdc846","keyframes":{"opacity":[0,1]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"both"}] [{"selector":"#anim-0874b1dd-1ce0-4b01-a738-acd3032fc974","keyframes":{"transform":["scale(0.3333333333333333)","scale(1)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"forwards"}] [{"selector":"#anim-a01030e5-c455-4304-9bc9-12410e32ad36","keyframes":{"opacity":[0,1]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"both"}] [{"selector":"#anim-1ba2e549-945b-4c2f-a9a5-bec914438fcc","keyframes":{"transform":["scale(0.3333333333333333)","scale(1)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"forwards"}] [{"selector":"#anim-edca83cc-f42a-4691-93a8-1238027d1747 [data-leaf-element=\"true\"]","keyframes":{"transform":["translate(0%, 0%) scale(1.5)","translate(0%, 0%) scale(1)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"forwards"}] [{"selector":"#anim-e5e27a3e-91de-4104-bbc2-42f9c5f6c2a5","keyframes":{"opacity":[0,1]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"both"}] [{"selector":"#anim-c8430a0f-3269-454a-8747-c8a5f8034625","keyframes":{"transform":["scale(0.3333333333333333)","scale(1)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"forwards"}] Want to explore its exciting features? Click here

1. Resilient Distributed Datasets (RDD) A data storage & processing framework that facilitates data recovery & computation in Apache Spark. It has two methods for data modification: transformations and actions.

[{"selector":"#anim-37e41c3b-db51-4cbc-ae88-070005ea2119 [data-leaf-element=\"true\"]","keyframes":{"transform":["translate(0%, 0%) scale(1.5)","translate(0%, 0%) scale(1)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"forwards"}] [{"selector":"#anim-327f3056-40d7-488e-a71c-98c81327c616","keyframes":{"opacity":[0,1]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"both"}] [{"selector":"#anim-123b16cb-be36-4d0c-ad8e-714e8ec2fe6b","keyframes":{"transform":["scale(0.3333333333333333)","scale(1)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"forwards"}] [{"selector":"#anim-8cc3b5cc-3658-42b2-9333-1955b78338fa","keyframes":{"opacity":[0,1]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"both"}] [{"selector":"#anim-7993fb26-5d1d-4e90-8b1b-07a3e9e92538","keyframes":{"transform":["scale(0.3333333333333333)","scale(1)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"forwards"}] [{"selector":"#anim-2cdbe5e8-abdb-4b53-b267-eeadd9480cc2","keyframes":{"opacity":[0,1]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"both"}] [{"selector":"#anim-fa64167e-0d0f-4487-81c4-2987b9441624","keyframes":{"transform":["scale(0.3333333333333333)","scale(1)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"forwards"}] Learn more about RDD in Apache Spark Click here

2. Directed Acyclic Graph (DAG) A series of connections between nodes that represent a program converted by the driver for each job. It helps optimize data processing & management in Spark apps.

[{"selector":"#anim-a53b548a-8093-4fa9-ac39-1d131bef6f70 [data-leaf-element=\"true\"]","keyframes":{"transform":["translate(0%, 0%) scale(1.5)","translate(0%, 0%) scale(1)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"forwards"}] [{"selector":"#anim-041208de-6b5d-44f7-b458-0198039620f3","keyframes":{"opacity":[0,1]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"both"}] [{"selector":"#anim-5394e4e0-79b3-4171-b33b-e8b6a0a091f1","keyframes":{"transform":["scale(0.3333333333333333)","scale(1)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"forwards"}] [{"selector":"#anim-046cb59c-982e-453d-ae14-b1603bb0593e","keyframes":{"opacity":[0,1]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"both"}] [{"selector":"#anim-5674c3b6-0309-4a7e-969e-8160090b4798","keyframes":{"transform":["scale(0.3333333333333333)","scale(1)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"forwards"}] Learn more about DAG in Apache Spark Click here

A distributed computing framework that follows a master-slave architecture. It allows users to process large-scale data in parallel across a cluster of computers.

[{"selector":"#anim-ac41b6da-eb96-4052-8f32-a73827664b69 [data-leaf-element=\"true\"]","keyframes":{"transform":["translate(0%, 0%) scale(1.5)","translate(0%, 0%) scale(1)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"forwards"}] [{"selector":"#anim-b36158a0-e15a-4c17-b241-3467e4f9d7e6","keyframes":{"opacity":[0,1]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"both"}] [{"selector":"#anim-41b14e4d-4395-4db7-960f-3687df8d45e6","keyframes":{"transform":["scale(0.3333333333333333)","scale(1)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"forwards"}] [{"selector":"#anim-4fbefa4b-8b2e-4ff5-b7c5-43a3a7ddfdbf","keyframes":{"opacity":[0,1]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"both"}] [{"selector":"#anim-90241d8e-c369-40bb-98f0-56849e551dd7","keyframes":{"transform":["scale(0.3333333333333333)","scale(1)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"forwards"}] [{"selector":"#anim-602a7861-d6d4-4a4e-be35-7b1e35d2c93d","keyframes":{"opacity":[0,1]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"both"}] [{"selector":"#anim-99bb0bfd-d0da-438b-b49d-309692a3e293","keyframes":{"transform":["scale(0.3333333333333333)","scale(1)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"forwards"}] Want to see the basic Spark Architecture? Click here

1. The Spark driver 2. The Spark executors 3. Cluster Manager 4. Worker Nodes

[{"selector":"#anim-c6f4a1b8-cd62-4a48-8bb9-cc89227d0d95 [data-leaf-element=\"true\"]","keyframes":{"transform":["translate(0%, 0%) scale(1.5)","translate(0%, 0%) scale(1)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"forwards"}] [{"selector":"#anim-a3876be5-4ab3-499b-a1ec-59fc33c95407","keyframes":{"opacity":[0,1]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"both"}] [{"selector":"#anim-d594d070-d173-4dfa-adba-070d3169b69f","keyframes":{"transform":["scale(0.3333333333333333)","scale(1)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"forwards"}] [{"selector":"#anim-734a853d-128c-4ed8-a43d-12d2d6293f7a","keyframes":{"opacity":[0,1]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"both"}] [{"selector":"#anim-0e49cdc9-1b81-4bcc-a35f-4216ce5785a1","keyframes":{"transform":["scale(0.3333333333333333)","scale(1)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"forwards"}] [{"selector":"#anim-5003f539-f009-4947-b38e-080d4c70b052","keyframes":{"opacity":[0,1]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"both"}] [{"selector":"#anim-27fb9f66-152e-45d0-9b87-3c56da7fbd97","keyframes":{"transform":["scale(0.3333333333333333)","scale(1)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"forwards"}] Want to dive in detail? Click here

The modes determine where your app’s resources are physically located when you run your app. You can choose from three different execution modes - 1. Cluster mode 2. Client mode 3. Local mode

[{"selector":"#anim-a7506678-b257-434f-920f-b1e8519001cf [data-leaf-element=\"true\"]","keyframes":{"transform":["translate(0%, 0%) scale(1.5)","translate(0%, 0%) scale(1)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"forwards"}] [{"selector":"#anim-6754ec45-2cdf-4716-8228-bcf98c040f0d","keyframes":{"opacity":[0,1]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"both"}] [{"selector":"#anim-1905fb5b-c2a4-4970-acc1-ddcf07d3a7dd","keyframes":{"transform":["scale(0.3333333333333333)","scale(1)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"forwards"}] [{"selector":"#anim-7c0cd861-2d45-4644-a70c-640f8d0dbe9a","keyframes":{"opacity":[0,1]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"both"}] [{"selector":"#anim-67441d62-11de-4b1e-bf2d-5aa9fe7a5d88","keyframes":{"transform":["scale(0.3333333333333333)","scale(1)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"forwards"}] [{"selector":"#anim-c3347c44-2f3f-4b93-99f1-9352a1280834","keyframes":{"opacity":[0,1]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"both"}] [{"selector":"#anim-d4977099-ff49-42cf-93ee-5eb25e0dfea3","keyframes":{"transform":["scale(0.3333333333333333)","scale(1)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"forwards"}] Check out the blog to learn more Click here

How does Data Lake differ from Data warehouse?

[{"selector":"#anim-ff2f75c8-44c5-4ecd-a4fc-fb31211464af","keyframes":{"opacity":[0,1]},"delay":0,"duration":1000,"easing":"cubic-bezier(0.2, 0.6, 0.0, 1)","fill":"both"}] [{"selector":"#anim-4489b75f-c2ab-4bb5-b9f3-71b42402d676","keyframes":{"transform":["translate3d(-103.0137%, 0px, 0)","translate3d(0px, 0px, 0)"]},"delay":0,"duration":1000,"easing":"cubic-bezier(0.2, 0.6, 0.0, 1)","fill":"both"}] [{"selector":"#anim-66efb47f-5f4b-4aca-a3ed-a5db7c82d0a7","keyframes":{"opacity":[0,1]},"delay":0,"duration":600,"easing":"cubic-bezier(0.2, 0.6, 0.0, 1)","fill":"both"}] [{"selector":"#anim-a5555001-27b4-4fe9-9e09-4580e4353d0b","keyframes":{"transform":["translate3d(-103.02197%, 0px, 0)","translate3d(0px, 0px, 0)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.2, 0.6, 0.0, 1)","fill":"both"}] Click here

Don't miss out on the chance to upskill yourself with IntervewBit's engaging web stories.

[{"selector":"#anim-d46f1664-5d7c-4975-9243-654afa9b026e","keyframes":{"opacity":[0,1]},"delay":0,"duration":1000,"easing":"cubic-bezier(0.2, 0.6, 0.0, 1)","fill":"both"}] [{"selector":"#anim-dd5c3a9b-2f65-406e-ad43-614a9d0da61c","keyframes":{"transform":["translate3d(0px, -211.49618%, 0)","translate3d(0px, 0px, 0)"]},"delay":0,"duration":1000,"easing":"cubic-bezier(0.2, 0.6, 0.0, 1)","fill":"both"}] [{"selector":"#anim-66353520-ffac-4a78-8971-6362183b35c6","keyframes":{"opacity":[0,1]},"delay":0,"duration":600,"easing":"cubic-bezier(0.2, 0.6, 0.0, 1)","fill":"both"}] [{"selector":"#anim-1695ecc8-8dee-4ee0-a453-4f348800f336","keyframes":{"transform":["translate3d(0px, -621.00265%, 0)","translate3d(0px, 0px, 0)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.2, 0.6, 0.0, 1)","fill":"both"}] Step Up Your Game with InterviewBit Web Stories SWIPE UP

Understading Apache Spark Architecture

Transform Your Big Data Processing with Apache Spark!

Looking to process and analyze large amounts of data efficiently? Look no further than Apache Spark architecture.

What is Apache Spark?

A big data processing framework that uses RDD & DAG data storage & processing models to handle massive amounts of structured, semi-structured, & unstructured data for analytics.

1. Resilient Distributed Datasets (RDD) A data storage & processing framework that facilitates data recovery & computation in Apache Spark. It has two methods for data modification: transformations and actions.

Two Main Abstractions of Apache Spark

2. Directed Acyclic Graph (DAG) A series of connections between nodes that represent a program converted by the driver for each job. It helps optimize data processing & management in Spark apps.

A distributed computing framework that follows a master-slave architecture. It allows users to process large-scale data in parallel across a cluster of computers.

Apache Spark Architecture

1. The Spark driver 2. The Spark executors 3. Cluster Manager 4. Worker Nodes

Spark Architecture Applications

The modes determine where your app’s resources are physically located when you run your app. You can choose from three different execution modes - 1. Cluster mode 2. Client mode 3. Local mode

Modes of Execution

How does Data Lake differ from Data warehouse?

Check out our detailed guide for a deep dive into the world of Data Lakes.

Don't miss out on the chance to upskill yourself with IntervewBit's engaging web stories.

“