【SparkAPI】JavaPairRDD——countByKey、countByKeyApprox

/** * Count the number of elements for each key, collecting the results to a local Map. * * @note This method should only be used if the resulting map is expected to be small, as * the whole thing is loaded into the driver's memory. * To handle very large results, consider using rdd.mapValues(_ => 1L).reduceByKey(_ + _), which * returns an RDD[T, Long] instead of a map. */

計算每個鍵的元素數，將結果放到Map中去。

注意：

只有當數據量很小時，才應使用此方法，因為整個數據都被載入內存中。

如果要處理大量數據，請考慮使用rdd.mapValues(_ => 1L).reduceByKey(_ + _)，

返回的結果是 RDD[T, Long] 而不是Map。

// java public java.util.Map countByKey() // scala def countByKey(): Map[K, Long]

public class CountByKey { public static void main(String[] args) { System.setProperty("hadoop.home.dir", "E:\hadoop-2.7.1"); SparkConf sparkConf = new SparkConf().setMaster("local").setAppName("Spark_DEMO"); JavaSparkContext sc = new JavaSparkContext(sparkConf); JavaPairRDD javaPairRDD1 = sc.parallelizePairs(Lists.newArrayList( new Tuple2("cat", "11"), new Tuple2("dog", "22"), new Tuple2("cat", "33"), new Tuple2("pig", "44"), new Tuple2("duck", "55"), new Tuple2("cat", "66")), 3); Map key = javaPairRDD1.countByKey(); for (Map.Entry entry : key.entrySet()){ System.out.println(entry.getKey()+":"+entry.getValue()); } } }

19/03/20 16:36:11 INFO DAGScheduler: ResultStage 1 (countByKey at CountByKey.java:23) finished in 0.093 s 19/03/20 16:36:11 INFO DAGScheduler: Job 0 finished: countByKey at CountByKey.java:23, took 1.229949 s duck:1 cat:3 dog:1 pig:1 19/03/20 16:36:11 INFO SparkContext: Invoking stop() from shutdown hook

/** * Approximate version of countByKey that can return a partial result if it does * not finish within a timeout. * * The confidence is the probability that the error bounds of the result will * contain the true value. That is, if countApprox were called repeatedly * with confidence 0.9, we would expect 90% of the results to contain the * true count. The confidence must be in the range [0,1] or an exception will * be thrown. * * @param timeout maximum time to wait for the job, in milliseconds * @param confidence the desired statistical confidence in the result * @return a potentially incomplete result, with error bounds */

CountByKey的近似版本，如果沒有在規定時間內完成就返回部分結果。

@參數超時等待作業的最長時間（毫秒）

@參數置信度結果中所需的統計置信度

@返回一個可能不完整的結果，帶有錯誤界限

// java public PartialResult> countByKeyApprox(long timeout) public PartialResult> countByKeyApprox(long timeout, double confidence) // scala def countByKeyApprox(timeout: Long): PartialResult[Map[K, BoundedDouble]] def countByKeyApprox(timeout: Long, confidence: Double = 0.95): PartialResult[Map[K, BoundedDouble]]

EI企業智能 Java spark 可信智能計算服務 TICS 智能數據

版權聲明：本文內容由網絡用戶投稿，版權歸原作者所有，本站不擁有其著作權，亦不承擔相應法律責任。如果您發現本站中有涉嫌抄襲或描述失實的內容，請聯系我們jiasou666@gmail.com 處理，核實后本網站將在24小時內刪除侵權內容。

版權聲明：本文內容由網絡用戶投稿，版權歸原作者所有，本站不擁有其著作權，亦不承擔相應法律責任。如果您發現本站中有涉嫌抄襲或描述失實的內容，請聯系我們jiasou666@gmail.com 處理，核實后本網站將在24小時內刪除侵權內容。