matlab.compiler.mlspark.SparkContext Class

Namespace: matlab.compiler.mlspark
Superclasses:

Interface class to initialize a connection to a Spark enabled cluster

Description

A SparkContext object serves as an entry point to Spark™ by initializing a connection to a Spark cluster. It accepts a SparkConf object as an input argument and uses the parameters specified in that object to set up the internal services necessary to establish a connection to the Spark execution environment.

Construction

sc = matlab.compiler.mlspark.SparkContext(conf) creates a SparkContext object initializes a connection to a Spark cluster.

Input Arguments

conf — Variable name representing a SparkConf object

SparkConf object

Pass the SparkConf object as input to the SparkContext.

Example: sc = matlab.compiler.mlspark.SparkContext(conf);

See matlab.compiler.mlspark.SparkConf for information on how to create a SparkConf object.

Properties

The properties of this class are hidden.

Methods

addJar	Add JAR file dependency for all tasks that need to be executed in a SparkContext
broadcast	Broadcast a read-only variable to the cluster
datastoreToRDD	Convert MATLAB datastore to a Spark RDD
delete	Shutdown connection to Spark enabled cluster
getSparkConf	Get SparkConf configuration parameters
parallelize	Create an RDD from a collection of local MATLAB values
setCheckpointDir	Set the directory under which RDDs are to be checkpointed
setLogLevel	Set log level
textFile	Create an RDD from a text file

Examples

Initialize a Connection to a Spark Enabled Cluster

The SparkContext class initializes a connection to a Spark enabled cluster using Spark properties.

% Setup Spark Properties as a containers.Map object
sparkProp = containers.Map({'spark.executor.cores'}, {'1'}); 

% Create SparkConf object
conf = matlab.compiler.mlspark.SparkConf('AppName','myApp', ...
                        'Master','local[1]','SparkProperties',sparkProp);
% Create a SparkContext
sc = matlab.compiler.mlspark.SparkContext(conf);

More About

SparkContext

A SparkContext represents a connection to a Spark cluster. It is the entry point to Spark and sets up the internal services necessary to establish a connection to the Spark execution environment.

References

See the latest Spark documentation for more information.

Version History

Introduced in R2016b