parallel.cluster.Hadoop

R2026b

Hadoop cluster for mapreducer, mapreduce and tall arrays

Description

A parallel.cluster.Hadoop object provides access to a cluster for configuring mapreducer, mapreduce, and tall arrays.

Creation

Use the parallel.cluster.Hadoop function to create a Hadoop^® cluster object.

Syntax

hadoopCluster = parallel.cluster.Hadoop

hadoopCluster = parallel.cluster.Hadoop(PropertyName=Value)

Description

hadoopCluster = parallel.cluster.Hadoop creates a parallel.cluster.Hadoop object representing the Hadoop cluster.

You use the resulting object as input to the mapreduce and mapreducer functions, for specifying the Hadoop cluster as the parallel execution environment for tall arrays and mapreduce.

hadoopCluster = parallel.cluster.Hadoop(PropertyName=Value) specifies one or more property values using name-value arguments. For example, parallel.cluster.Hadoop(HadoopInstallFolder='/host/hadoop-install'); sets the Hadoop install folder.

Output Arguments

hadoopCluster — Hadoop cluster

parallel.cluster.Hadoop object

Hadoop cluster, returned as a parallel.cluster.Hadoop object.

Properties

AdditionalPaths — Folders to add to MATLAB^® search path

character vector | string | string array | cell array of character vectors

Folders to add to MATLAB search path of workers, specified as a character vector, string or string array, or cell array of character vectors.

When you offload computations to workers, any files that the client needs for computations must also be available on workers. By default, the client attempts to detect and attach these files. To turn off automatic detection, set the AutoAttachFiles property to false. If the software cannot find all the files, or if sending files from client to worker is slow, use one of these options.

• If the files are in a folder that is not accessible on the workers, set the AttachedFiles property. The cluster copies each file you specify from the client to the workers.

• If the files are in a folder that is accessible on the workers, you can set the AdditionalPaths property instead. Use the AdditionalPaths property to add paths to the MATLAB search path for each worker and avoid copying files unnecessarily from the client to the workers.

AttachedFiles — Files and folders sent to workers

character vector | string | string array | cell array of character vectors

Files and folders sent to workers during a mapreduce call, specified as a character vector, string or string array, or cell array of character vectors

When you offload computations to workers, any files that the client needs for computations must also be available on workers. By default, the client attempts to detect and attach these files. To turn off automatic detection, set the AutoAttachFiles property to false. If the software cannot find all the files, or if sending files from client to worker is slow, use one of these options.

• If the files are in a folder that is not accessible on the workers, set the AttachedFiles property. The cluster copies each file you specify from the client to the workers.

• If the files are in a folder that is accessible on the workers, you can set the AdditionalPaths property instead. Use the AdditionalPaths property to add paths to the MATLAB search path for each worker and avoid copying files unnecessarily from the client to the workers.

AutoAttachFiles — Specify whether to automatically attach files

true (default) | false

Specify whether to automatically detect and attach files on the client.

Data Types: logical

ClusterMatlabRoot — Path to MATLAB for workers

character vector

Path to MATLAB for workers, specified as a character vector. This points to the installation of MATLAB Parallel Server™ for the workers, whether local to each machine or on a network share.

HadoopConfigurationFile — Path to Hadoop application configuration file

character vector

Path to Hadoop application configuration file, specified as a character vector.

HadoopInstallFolder — Path to Hadoop installation on the local machine

character vector

Path to Hadoop installation on the local machine, specified as a character vector. If this property is not set, the default is the value specified by the environment variable HADOOP_PREFIX, or if that is not set, then HADOOP_HOME.

HadoopProperties — Map of Hadoop name-value property pairs

character vector

Map of Hadoop name-value property pairs to be given to the Hadoop cluster.

HadoopProperties allows you to override configuration properties for Hadoop. See the list of properties in the Hadoop documentation.

LicenseNumber — License number to use

integer

License number to use with online licensing.

RequiresOnlineLicensing — Specify whether cluster requires online licensing

false (default) | true

Specify whether the Hadoop cluster uses online licensing.

Data Types: logical

SparkInstallFolder — Path to Spark™ enabled Hadoop installation on worker machines

character vector

Path to Spark installation on worker machines, specified as a character vector. If this property is not set, the default is the value specified by the environment variable SPARK_PREFIX, or if that is not set, then SPARK_HOME.

The SparkInstallFolder is by default set to the SPARK_HOME environment variable. This is required for tall array evaluation on Hadoop (but not for mapreduce). For a correctly configured cluster, you only need to set the installation folder.

SparkProperties — Map of Spark name-value property pairs

character vector

Map of Spark name-value property pairs to be given to the Spark enabled Hadoop cluster.

SparkProperties allows you to override configuration properties for Spark. See the list of properties in the Spark documentation.

Object Functions

mapreduce	Programming technique for analyzing data sets that do not fit in memory
mapreducer	Define parallel execution environment for mapreduce and tall arrays

Examples

Set Hadoop Cluster as Execution Environment for mapreduce and mapreducer

Create and use a parallel.cluster.Hadoop object to set a Hadoop cluster as the mapreduce parallel execution environment.

hadoopCluster = parallel.cluster.Hadoop(HadoopInstallFolder='/host/hadoop-install');
mr = mapreducer(hadoopCluster);

Set Hadoop Cluster as Execution Environment for tall arrays

This example shows how to create and use a parallel.cluster.Hadoop object to set a Hadoop cluster as the tall array parallel execution environment.

hadoopCluster = parallel.cluster.Hadoop(...
    HadoopInstallFolder='/host/hadoop-install', ...
    SparkInstallFolder='/host/spark-install');
mr = mapreducer(hadoopCluster);

More About

Specify Memory Properties

Spark enabled Hadoop clusters place limits on how much memory is available. You must adjust these limits to support your workflow.

Size of Data to Gather

The amount of data gathered to the client is limited by the Spark properties:

• spark.driver.memory

• spark.executor.memory

The amount of data to gather from a single Spark task must fit in these properties. A single Spark task processes one block of data from HDFS, which is 128 MB of data by default. If you gather a tall array containing most of the original data, you must ensure these properties are set to fit.

If these properties are set too small, you see an error like the following.

Error using tall/gather (line 50)
Out of memory; unable to gather a partition of size 300m from Spark.
Adjust the values of the Spark properties spark.driver.memory and 
spark.executor.memory to fit this partition.

The error message also specifies the property settings you need.

Adjust the properties either in the default settings of the cluster or directly in MATLAB. To adjust the properties in MATLAB, add Spark and values to the SparkProperties property of the cluster object. For example, set the spark.driver.memory and spark.executor.memory Spark properties in the SparkProperties of the Hadoop cluster object.

cluster = parallel.cluster.Hadoop;
cluster.SparkProperties('spark.driver.memory') = '2048m';
cluster.SparkProperties('spark.executor.memory') = '2048m';
mapreducer(cluster);

Specify Working Memory Size for a MATLAB Worker

The amount of working memory for a MATLAB worker is limited by the Spark property:

• spark.yarn.executor.memoryOverhead

By default, this is set to 2.5 GB. You typically need to increase this if you use the arrayfun and cellfun functions, or custom datastores to generate large amounts of data in one go. It is advisable to increase this if you come across lost or crashed Spark Executor processes.

You can adjust these properties either in the default settings of the cluster or directly in MATLAB. To adjust the properties in MATLAB, add Spark and values to the SparkProperties property of the cluster object. For example, set the spark.yarn.executor.memoryOverhead Spark property in the SparkProperties of the Hadoop cluster object.

cluster = parallel.cluster.Hadoop; 
cluster.SparkProperties('spark.yarn.executor.memoryOverhead') = '4096m';
mapreducer(cluster);

Version History

Introduced in R2014b