看看spark的源码

  1. IntelliJ IDEA安装plugins,加入scala插件和SBT插件

  2. 下载spark的源代码,下载地址 https://github.com/apache/spark,可以使用git下载或者下载zip包

  3. 导入项目,开启 IntelliJ 之后选择 Import Project,而后选择 Spark 源代码,并将其导入为 SBT 项目,在之后的过程中 SBT 过程将会自动进行,下载作为依赖的 jar 包。

  4. 源代码已经准备好了。开干吧

    private def schedule(): Unit = {
        if (state != RecoveryState.ALIVE) {
          return
        }
        // Drivers take strict precedence over executors
        val shuffledAliveWorkers = Random.shuffle(workers.toSeq.filter(_.state == WorkerState.ALIVE))
        val numWorkersAlive = shuffledAliveWorkers.size
        var curPos = 0
        for (driver <- waitingDrivers.toList) { // iterate over a copy of waitingDrivers
          // We assign workers to each waiting driver in a round-robin fashion. For each driver, we
          // start from the last worker that was assigned a driver, and continue onwards until we have
          // explored all alive workers.
          var launched = false
          var numWorkersVisited = 0
          while (numWorkersVisited < numWorkersAlive && !launched) {
            val worker = shuffledAliveWorkers(curPos)
            numWorkersVisited += 1
            if (worker.memoryFree >= driver.desc.mem && worker.coresFree >= driver.desc.cores) {
              launchDriver(worker, driver)
              waitingDrivers -= driver
              launched = true
            }
            curPos = (curPos + 1) % numWorkersAlive
          }
        }
        startExecutorsOnWorkers()
      }

     

 

posted @ 2018-10-10 16:03  华格瑞沙  阅读(1429)  评论(0编辑  收藏  举报