-
sparkEnv
(D:\Spark\SourceCode\spark\core\src\main\scala\org\apache\spark\SparkEnv.scala)
Holds all the runtime environment objects for a running Spark instance (either master or worker), including the serializer, Akka actor system, block manager, map output tracker, etc. Currently Spark code finds the SparkEnv through a global variable, so all the threads can access the same SparkEnv. It can be accessed by SparkEnv.get (e.g. after creating a SparkContext).
-
Block manager
(D:\Spark\SourceCode\spark\core\src\main\scala\org\apache\spark\storage\BlockManager.scala)
Manager running on every node (driver and executors) which provides interfaces for putting and retrieving blocks both locally and remotely into various stores (memory, disk, and off-heap).
Note that #initialize() must be called before the BlockManager is usable.
-
SparkConf
(D:\Spark\SourceCode\spark\core\src\main\scala\org\apache\spark\SparkConf.scala)
- Configuration for a Spark application. Used to set various Spark parameters as key-value pairs.
- Most of the time, you would create a SparkConf object with `new SparkConf()`, which will load values from any `spark.*` Java system properties set in your application as well. In this case,parameters you set directly on the `SparkConf` object take priority over system properties.
- For unit tests, you can also call `new SparkConf(false)` to skip loading external settings and get the same configuration no matter what the system properties are.
- All setter methods in this class support chaining. For example, you can write `new SparkConf().setMaster("local").setAppName("My app")`.
- Note that once a SparkConf object is passed to Spark, it is cloned and can no longer be modified by the user. Spark does not support modifying the configuration at runtime.
- setMaster(): The master URL to connect to, such as "local" to run locally with one thread, "local[4]" to run locally with 4 cores, or "spark://master:7077" to run o``n a Spark standalone cluster.
- setAppName(): Set a name for your application. Shown in the Spark web UI