← All posts
Tag
Apache Spark
4 posts.
20254 posts
- The Life Cycle of a Spark Application ( Outside )You run spark-submit and, a while later, get a result. In between, four things happen and none of them are in your code: the request, the launch, the execution and the shutdown.
- Spark Execution ModesSubmit a Spark job from your laptop, close the lid, and the job dies. Change one flag and it keeps running. Cluster, client and local mode come down to where the driver runs.
- Under the hood of a Spark jobTwo DataFrames with 5 and 6 partitions, one join, and the Spark UI shows 200 tasks. Where that number comes from, and how Spark splits work into jobs, stages and tasks.
- Handling Nulls in SparkSix of ten movies have no rating, and filtering for == None returns zero rows with no error. How nulls behave in Spark, and the functions that handle them properly.