Kylin4 對(duì)接MRS_3.0.5非安全集群

網(wǎng)友投稿 982 2022-05-30

MRS_3.0.5非安全集群對(duì)接Kylin4

一、準(zhǔn)備資源

1.準(zhǔn)備集群外ECS節(jié)點(diǎn)：

需要修改安全組，把節(jié)點(diǎn)網(wǎng)絡(luò)和集群網(wǎng)絡(luò)打通；

在節(jié)點(diǎn)上創(chuàng)建omm用戶: ?useradd -d /home/omm omm

2.準(zhǔn)備MySQL環(huán)境；

創(chuàng)建MySQL數(shù)據(jù)庫(kù)并和集群外ECS節(jié)點(diǎn)打通網(wǎng)絡(luò)；

準(zhǔn)備連接MySQL的java驅(qū)動(dòng)包；

3.下載Kylin4安裝包；

-：https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-4.0.1/apache-kylin-4.0.1-bin-spark2.tar.gz

4.下載Spark 2.4.7安裝包；

-：https://archive.apache.org/dist/spark/spark-2.4.7/spark-2.4.7-bin-without-hadoop.tgz

5.下載依賴的jar包

https://mirrors.huaweicloud.com/repository/maven/huaweicloudsdk/org/spark-project/hive/hive-beeline/1.2.1.spark_2.3.2-mrs-2.0/hive-beeline-1.2.1.spark_2.3.2-mrs-2.0.jar

https://mirrors.huaweicloud.com/repository/maven/huaweicloudsdk/org/spark-project/hive/hive-cli/1.2.1.spark_2.3.2-mrs-2.0/hive-cli-1.2.1.spark_2.3.2-mrs-2.0.jar

https://mirrors.huaweicloud.com/repository/maven/huaweicloudsdk/org/spark-project/hive/hive-common/1.2.1.spark_2.3.2-mrs-2.0/hive-common-1.2.1.spark_2.3.2-mrs-2.0.jar

https://mirrors.huaweicloud.com/repository/maven/huaweicloudsdk/org/spark-project/hive/hive-exec/1.2.1.spark_2.3.2-mrs-2.0/hive-exec-1.2.1.spark_2.3.2-mrs-2.0.jar

https://mirrors.huaweicloud.com/repository/maven/huaweicloudsdk/org/spark-project/hive/hive-jdbc/1.2.1.spark_2.3.2-mrs-2.0/hive-jdbc-1.2.1.spark_2.3.2-mrs-2.0.jar

https://mirrors.huaweicloud.com/repository/maven/huaweicloudsdk/org/spark-project/hive/hive-metastore/1.2.1.spark_2.3.2-mrs-2.0/hive-metastore-1.2.1.spark_2.3.2-mrs-2.0.jar

https://mirrors.huaweicloud.com/repository/maven/huaweicloudsdk/org/spark-project/hive/hive-shims/1.2.1.spark_2.3.2-mrs-2.0/hive-shims-1.2.1.spark_2.3.2-mrs-2.0.jar

https://mirrors.huaweicloud.com/repository/maven/huaweicloudsdk/org/spark-project/hive/shims/hive-shims-common/1.2.1.spark_2.3.2-mrs-2.0/hive-shims-common-1.2.1.spark_2.3.2-mrs-2.0.jar

https://mirrors.huaweicloud.com/repository/maven/huaweicloudsdk/org/spark-project/hive/shims/hive-shims-0.23/1.2.1.spark_2.3.2-mrs-2.0/hive-shims-0.23-1.2.1.spark_2.3.2-mrs-2.0.jar

https://mirrors.huaweicloud.com/repository/maven/huaweicloudsdk/org/spark-project/hive/shims/hive-shims-scheduler/1.2.1.spark_2.3.2-mrs-2.0/hive-shims-scheduler-1.2.1.spark_2.3.2-mrs-2.0.jar

https://mirrors.huaweicloud.com/repository/maven/huaweicloudsdk/org/spark-project/hive/hive-serde/1.2.1.spark_2.3.2-mrs-2.0/hive-serde-1.2.1.spark_2.3.2-mrs-2.0.jar

https://dlcdn.apache.org//commons/configuration/binaries/commons-configuration-1.10-bin.zip

https://search.maven.org/remotecontent?filepath=org/apache/spark/spark-hive_2.11/2.4.7/spark-hive_2.11-2.4.7.jar

https://search.maven.org/remotecontent?filepath=org/apache/spark/spark-hive-thriftserver_2.11/2.4.7/spark-hive-thriftserver_2.11-2.4.7.jar

二、安裝MRS客戶端

在集群外節(jié)點(diǎn)安裝MRS客戶端，具體安裝方法見(jiàn)MRS官網(wǎng)：https://support.huaweicloud.com/function-mrs/index.html

三、部署Kylin4對(duì)接MRS集群

1.上傳附件kylin4_adapter.tar.gz到集群外節(jié)點(diǎn)，并把前面下載的依賴jar放到解壓后的jars目錄下；

2.解壓并執(zhí)行腳本進(jìn)行適配

sh install.sh MRS客戶端路徑? kylin路徑? spark開(kāi)源解壓包路徑

例如：sh install.sh /opt/auto/mrsclient /opt/auto/apache-kylin-4.0.1-bin-spark2 /opt/auto/spark-2.4.7-bin-without-hadoop

四、配置Kylin4環(huán)境

參考上一步腳本執(zhí)行完的提示信息進(jìn)行配置；

Kylin4對(duì)接MRS_3.0.5非安全集群

五、使用Kylin4

1.執(zhí)行如下命令啟動(dòng)kylin

source /xxx/xxx/bigdata_env

$KYLIN_HOME/bin/kylin.sh start

2.訪問(wèn)kylin頁(yè)面地址

http://xxx.xxx.xxx.xxx:7070/kylin/jobs

六、注意事項(xiàng)：

1.本文是MRS 305版本非安全集群集成kylin的文檔;

2.此文檔默認(rèn)是用omm作為執(zhí)行用戶，如果更改用戶需要保證用戶有Kylin配置的HDFS路徑的權(quán)限；

附件： kylin4_adapter.rar 2.07KB 下載次數(shù)：0次

版權(quán)聲明：本文內(nèi)容由網(wǎng)絡(luò)用戶投稿，版權(quán)歸原作者所有，本站不擁有其著作權(quán)，亦不承擔(dān)相應(yīng)法律責(zé)任。如果您發(fā)現(xiàn)本站中有涉嫌抄襲或描述失實(shí)的內(nèi)容，請(qǐng)聯(lián)系我們jiasou666@gmail.com 處理，核實(shí)后本網(wǎng)站將在24小時(shí)內(nèi)刪除侵權(quán)內(nèi)容。