aggregate

Class: matlab.compiler.mlspark.RDD
Namespace: matlab.compiler.mlspark

Aggregate the elements of each partition and subsequently the results for all partitions into a single value

Syntax

result = aggregate(obj,zeroValue,seqOp,combOp)


Description

result = aggregate(obj,zeroValue,seqOp,combOp) aggregates the elements into a single value using given combine functions specified by seqOp and combOp, and a neutral “zero value” specified by zeroValue.

Input Arguments

obj — Input RDD

RDD object

An input RDD, specified as a RDD object.

zeroValue — Neutral “zero value”

cell array of numbers

A neutral “zero value”, specified as a cell array of numbers.

Data Types: cell

seqOp — Function to aggregate the values of each key

function handle

A function to aggregate the values of each key, specified as a function handle.

Data Types: function_handle

combOp — Function to aggregate results of seqOp

function handle

A function to aggregate results of seqOp, specified as a function handle.

Data Types: function_handle

Output Arguments

result — RDD containing aggregated elements

RDD object

An RDD containing aggregated elements, returned as an RDD object.

Examples

Aggregate Elements of an RDD into a Single Value

%% Connect to Spark
sparkProp = containers.Map({'spark.executor.cores'}, {'1'});
conf = matlab.compiler.mlspark.SparkConf('AppName','myApp', ...
                        'Master','local[1]','SparkProperties',sparkProp);
sc = matlab.compiler.mlspark.SparkContext(conf);

%% Aggregate
seqOp = @(x,y)({x{1} + y, x{2} + 1});
combOp = @(x,y)({x{1} + y{1}, x{2} + y{2}});
x = sc.parallelize({1, 2, 3, 4});
y = x.aggregate({0, 0}, seqOp, combOp) % {10,4}

Version History

Introduced in R2016b