【SparkAPI Java版】JavaPairRDD——countByValue、countByValueApprox

/** * Return the count of each unique value in this RDD as a map of (value, count) pairs. The final * combine step happens locally on the master, equivalent to running a single reduce task. */

返回RDD中每個(gè)值的計(jì)數(shù)，作為（value，count）對(duì)的映射。

返回的是map

// java public static java.util.Map countByValue() // scala def countByValue(): Map[(K, V), Long]

public class CountByValue { public static void main(String[] args) { System.setProperty("hadoop.home.dir", "E:\hadoop-2.7.1"); SparkConf sparkConf = new SparkConf().setMaster("local").setAppName("Spark_DEMO"); JavaSparkContext sc = new JavaSparkContext(sparkConf); JavaPairRDD javaPairRDD1 = sc.parallelizePairs(Lists.newArrayList( new Tuple2("cat", "11"), new Tuple2("dog", "22"), new Tuple2("cat", "11"), new Tuple2("pig", "44"), new Tuple2("duck", "55"), new Tuple2("cat", "66")), 3); Map, Long> value = javaPairRDD1.countByValue(); for (Map.Entry, Long> entry : value.entrySet()){ System.out.println(entry.getKey()+"->"+entry.getValue()); } } }

19/03/20 17:15:31 INFO DAGScheduler: Job 0 finished: countByValue at CountByValue.java:23, took 1.093040 s 19/03/20 17:15:31 INFO SparkContext: Invoking stop() from shutdown hook (duck,55)->1 (dog,22)->1 (pig,44)->1 (cat,66)->1 (cat,11)->2 19/03/20 17:15:31 INFO SparkUI: Stopped Spark web UI at http://10.124.209.6:4040

/** * Approximate version of countByValue(). * * The confidence is the probability that the error bounds of the result will * contain the true value. That is, if countApprox were called repeatedly * with confidence 0.9, we would expect 90% of the results to contain the * true count. The confidence must be in the range [0,1] or an exception will * be thrown. * * @param timeout maximum time to wait for the job, in milliseconds * @param confidence the desired statistical confidence in the result * @return a potentially incomplete result, with error bounds */

CountByValue（）的近似版本。

置信度必須在[0,1]范圍內(nèi)，否則異常將被扔掉。

*@參數(shù)超時(shí)等待作業(yè)的最長(zhǎng)時(shí)間（毫秒）

*@參數(shù)置信度結(jié)果中所需的統(tǒng)計(jì)置信度

*@返回一個(gè)可能不完整的結(jié)果，帶有錯(cuò)誤界限

// java public static PartialResult> countByValueApprox(long timeout) public static PartialResult> countByValueApprox(long timeout, double confidence) // scala def countByValueApprox(timeout: Long): PartialResult[Map[(K, V), BoundedDouble]] def countByValueApprox(timeout: Long, confidence: Double): PartialResult[Map[(K, V), BoundedDouble]]

EI企業(yè)智能 Java spark 可信智能計(jì)算服務(wù) TICS 智能數(shù)據(jù)

版權(quán)聲明：本文內(nèi)容由網(wǎng)絡(luò)用戶投稿，版權(quán)歸原作者所有，本站不擁有其著作權(quán)，亦不承擔(dān)相應(yīng)法律責(zé)任。如果您發(fā)現(xiàn)本站中有涉嫌抄襲或描述失實(shí)的內(nèi)容，請(qǐng)聯(lián)系我們jiasou666@gmail.com 處理，核實(shí)后本網(wǎng)站將在24小時(shí)內(nèi)刪除侵權(quán)內(nèi)容。

版權(quán)聲明：本文內(nèi)容由網(wǎng)絡(luò)用戶投稿，版權(quán)歸原作者所有，本站不擁有其著作權(quán)，亦不承擔(dān)相應(yīng)法律責(zé)任。如果您發(fā)現(xiàn)本站中有涉嫌抄襲或描述失實(shí)的內(nèi)容，請(qǐng)聯(lián)系我們jiasou666@gmail.com 處理，核實(shí)后本網(wǎng)站將在24小時(shí)內(nèi)刪除侵權(quán)內(nèi)容。