降维-基于RDD的API

降维-基于RDD的API

• Singular value decomposition (SVD)
  • Performance
  • SVD Example
• Principal component analysis (PCA)

Dimensionality reduction is the process of reducing the number of variables under consideration. It can be used to extract latent features from raw and noisy features or compress data while maintaining the structure. spark.mllib provides support for dimensionality reduction on the RowMatrix class.

• 奇异值分解（SVD）
  • 性能
  • SVD示例
• 主成分分析（PCA）

降维是减少所考虑的变量数量的过程。可用于从原始和嘈杂的特征中提取潜在特征，或者在保持结构的同时压缩数据。 spark.mllib提供对RowMatrix类降维的支持。

奇异值分解Singular value decomposition (SVD)

Singular value decomposition (SVD) factorizes a matrix into three matrices奇异值分解（SVD） 将矩阵分解为三个矩阵:  UU, ΣΣ, and VV such that

A=UΣVT,A=UΣVT,

where

• UU is an orthonormal matrix, whose columns are called left singular vectors, 正交矩阵，其列称为左奇异矢量
• ΣΣ is a diagonal matrix with non-negative diagonals in descending order, whose diagonals are called singular values, 具有非负对角线降序的对角矩阵，其对角线称为奇异值
• VV is an orthonormal matrix, whose columns are called right singular vectors. 正交矩阵，其列称为右奇异向量。

For large matrices, usually we don’t need the complete factorization but only the top singular values and its associated singular vectors. This can save storage, de-noise and recover the low-rank structure of the matrix. 对于大型矩阵，通常不需要完整的因式分解，而仅需要顶部top奇异值及其关联的奇异矢量。可以节省存储空间，降低噪声并恢复矩阵的低阶结构。

If we keep the top k singular values, then the dimensions of the resulting low-rank matrix will be如果保持领先 ķ 奇异值，则所得低秩矩阵的维将为：

• UU: m×km×k,
• ΣΣ: k×kk×k,
• VV: n×kn×k.

Performance

We assume n is smaller than m假设n小于m. The singular values and the right singular vectors are derived from the eigenvalues and the eigenvectors of the Gramian matrix 奇异值和右奇异向量是从Gramian矩阵的特征值和特征向量得出。The matrix storing the left singular vectors UU, is computed via matrix multiplication as U=A(VS−1)U=A(VS−1), if requested by the user via the computeU parameter. The actual method to use is determined automatically based on the computational cost存储左奇异向量的矩阵U" role="presentation" style="box-sizing: border-box; overflow-wrap: normal; max-width: none; max-height: none; min-width: 0px; min-height: 0px; float: none;" id="MathJax-Element-18-Frame">ü通过矩阵乘法计算为 U=A(VS−1)" role="presentation" style="box-sizing: border-box; overflow-wrap: normal; max-width: none; max-height: none; min-width: 0px; min-height: 0px; float: none;" id="MathJax-Element-19-Frame">ü= A ，如果用户通过computeU参数请求。实际使用的方法是根据计算成本自动确定的:

• If nn is small (n<100n<100) or k is large compared with nn (k>n/2), we compute the Gramian matrix first and then compute its top eigenvalues and eigenvectors locally on the driver. This requires a single pass with O(n2) storage on each executor and on the driver, and O(n2k) time on the driver.
• Otherwise, we compute (ATA)v in a distributive way and send it to ARPACK to compute (ATA)(ATA)’s top eigenvalues and eigenvectors on the driver node. This requires O(k) passes, O(n) storage on each executor, and O(nk) storage on the driver.

• 如果 ñ 是小 （n < 100） 或者 ķ 与 ñ （k > n / 2个），首先计算Gramian矩阵，然后在驱动程序上局部计算其最高特征值和特征向量。需要一次通过O （ñ2个）存储在每个执行器和驱动程序上，以及 O （ñ2个k ） 在驱动程序上的时间。
• 否则，计算 （一个ŤA ）v以分布式方式将其发送到 ARPACK以进行计算（一个ŤA ），驱动程序节点上的最大特征值和特征向量。需要O （k ）通过， O （n ）存储在每个执行器上，以及 Ø（ñķ） 存储在驱动程序上。

SVD Example

spark.mllib provides SVD functionality to row-oriented matrices, provided in the RowMatrix class. spark.mllib向RowMatrix类提供的面向行的矩阵，提供SVD功能 。

Scala

• Java
• Python

Refer to the SingularValueDecomposition Scala docs for details on the API. 有关该API的详细信息，请参考SingularValueDecompositionScala文档。

import org.apache.spark.mllib.linalg.Matrix

import org.apache.spark.mllib.linalg.SingularValueDecomposition

import org.apache.spark.mllib.linalg.Vector

import org.apache.spark.mllib.linalg.Vectors

import org.apache.spark.mllib.linalg.distributed.RowMatrix

 

val data = Array(

  Vectors.sparse(5, Seq((1, 1.0), (3, 7.0))),

  Vectors.dense(2.0, 0.0, 3.0, 4.0, 5.0),

  Vectors.dense(4.0, 0.0, 0.0, 6.0, 7.0))

 

val rows = sc.parallelize(data)

 

val mat: RowMatrix = new RowMatrix(rows)

 

// Compute the top 5 singular values and corresponding singular vectors.

val svd: SingularValueDecomposition[RowMatrix, Matrix] = mat.computeSVD(5, computeU = true)

val U: RowMatrix = svd.U  // The U factor is a RowMatrix.

val s: Vector = svd.s     // The singular values are stored in a local dense vector.

val V: Matrix = svd.V     // The V factor is a local dense matrix.

Find full example code at "examples/src/main/scala/org/apache/spark/examples/mllib/SVDExample.scala" in the Spark repo.

The same code applies to IndexedRowMatrix if U is defined as an IndexedRowMatrix.

Principal component analysis (PCA) 主成分分析

Principal component analysis (PCA) is a statistical method to find a rotation such that the first coordinate has the largest variance possible, and each succeeding coordinate, in turn, has the largest variance possible. The columns of the rotation matrix are called principal components. PCA is used widely in dimensionality reduction.

spark.mllib supports PCA for tall-and-skinny matrices stored in row-oriented format and any Vectors.

主成分分析（PCA）是一种统计方法，用于查找旋转，以使第一个坐标具有最大的方差，而每个后续坐标又具有最大的方差。旋转矩阵的列称为主成分。PCA被广泛用于降维。

spark.mllib 支持PCA用于以行格式和任何向量存储的高和稀疏矩阵。

• Scala
• Java
• Python

The following code demonstrates how to compute principal components on a RowMatrix and use them to project the vectors into a low-dimensional space.

Refer to the RowMatrix Scala docs for details on the API.

以下代码演示了如何在 RowMatrix 上计算主成分，将向量投影到低维空间中。

有关该API的详细信息，请参考RowMatrixScala文档。

import org.apache.spark.mllib.linalg.Matrix

import org.apache.spark.mllib.linalg.Vectors

import org.apache.spark.mllib.linalg.distributed.RowMatrix

 

val data = Array(

  Vectors.sparse(5, Seq((1, 1.0), (3, 7.0))),

  Vectors.dense(2.0, 0.0, 3.0, 4.0, 5.0),

  Vectors.dense(4.0, 0.0, 0.0, 6.0, 7.0))

 

val rows = sc.parallelize(data)

 

val mat: RowMatrix = new RowMatrix(rows)

 

// Compute the top 4 principal components.

// Principal components are stored in a local dense matrix.

val pc: Matrix = mat.computePrincipalComponents(4)

 

// Project the rows to the linear space spanned by the top 4 principal components.

val projected: RowMatrix = mat.multiply(pc)

Find full example code at "examples/src/main/scala/org/apache/spark/examples/mllib/PCAOnRowMatrixExample.scala" in the Spark repo.

The following code demonstrates how to compute principal components on source vectors and use them to project the vectors into a low-dimensional space while keeping associated labels:

Refer to the PCA Scala docs for details on the API.

在Spark存储库中找到完整的示例代码。

以下代码演示了如何在源向量上计算主成分，将向量投影到低维空间中，同时保留关联的标签：

有关该API的详细信息，请参考PCAScala文档。

import org.apache.spark.mllib.feature.PCA

import org.apache.spark.mllib.linalg.Vectors

import org.apache.spark.mllib.regression.LabeledPoint

import org.apache.spark.rdd.RDD

 

val data: RDD[LabeledPoint] = sc.parallelize(Seq(

  new LabeledPoint(0, Vectors.dense(1, 0, 0, 0, 1)),

  new LabeledPoint(1, Vectors.dense(1, 1, 0, 1, 0)),

  new LabeledPoint(1, Vectors.dense(1, 1, 0, 0, 0)),

  new LabeledPoint(0, Vectors.dense(1, 0, 0, 0, 0)),

  new LabeledPoint(1, Vectors.dense(1, 1, 0, 0, 0))))

 

// Compute the top 5 principal components.

val pca = new PCA(5).fit(data.map(_.features))

 

// Project vectors to the linear space spanned by the top 5 principal

// components, keeping the label

val projected = data.map(p => p.copy(features = pca.transform(p.features)))

Find full example code at "examples/src/main/scala/org/apache/spark/examples/mllib/PCAOnSourceVectorExample.scala" in the Spark repo.