Spark学习(4)----ScalaTest
一、例子:
1、一个简单例子:https://www.jianshu.com/p/ceabf3437dd7
2、Funsuite例子:https://www.programcreek.com/scala/org.scalatest.FunSuite
3、SparkFunsuite例子:https://www.programcreek.com/scala/org.apache.spark.SparkFunSuite
4、一个spark学习博客:https://liuxiaofei.com.cn/blog/category/spark/spark-core/
一个scalatest博客:https://blog.csdn.net/zhouyan8603/article/details/85560861
5、一个GBTRegressorSuite例子:https://fossies.org/linux/spark/mllib/src/test/scala/org/apache/spark/ml/regression/GBTRegressorSuite.scala
6、scalatest在线文档:http://tool.oschina.net/apidocs/apidoc?api=scalatest-1.7.2
7、scalatest官方文档:http://www.scalatest.org/getting_started_with_fun_suite
http://doc.scalatest.org/3.0.1/#org.scalatest.FunSuite
7、spark官方文档:http://spark.apache.org/docs/2.3.0/
二、 记录一个折腾了两天的错误
在本地写了ScalaTest的测试,并且可以通过,但是一放上服务器,一直报错,错误有:
【只要有dataframe.show()出现就报错,有collect() 就报错】
java.lang.UnsupportedOperationException: empty.max
java.lang.IllegalArgumentException: Can not set final [B field org.codehaus.janino.util.ClassFile$CodeAttribute.code to org.codehaus.janino.util.ClassFile$CodeAttribute
以上这些错误可能还是版本问题,因为我在maven的pom.xml中加了test的相关环境:【红字原来没加,就会报错】
<dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-sql_${scala.binary.version}</artifactId> <version>${spark.version}</version> <classifier>tests</classifier> <scope>test</scope> </dependency> <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-core_${scala.binary.version}</artifactId> <version>${spark.version}</version> <classifier>tests</classifier> <scope>test</scope> </dependency> <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-catalyst_${scala.binary.version}</artifactId> <version>${spark.version}</version> <classifier>tests</classifier> <scope>test</scope> </dependency>
所以要加上红色字体,表明这是test,就不会变异打包到服务器上跑,因为服务器上原本也有spark环境,这样两个spark环境就会有冲突。