Windows10配置運(yùn)行Hive（非WSL模式）

背景

相關(guān)篇：Windows10配置運(yùn)行Hadoop（非WSL模式）

Hive的運(yùn)行依賴HDFS，需要先準(zhǔn)備好Hadoop部分的配置。

版本

OS：Windows 10 Pro 1903

Java：1.8.0_231

Hive：Hive-3.1.0.tar.gz（Apache官方源下載 https://archive.apache.org/dist/hive/hive-3.1.0/ ）

derby：db-derby-10.14.2.0-bin.tar.gz（可以在官方 https://db.apache.org/derby/derby_downloads.html 下載，或是在鏡像站 https://downloads.apache.org/db/derby/ 尋找下載）

前置依賴

JDK環(huán)境

Hadoop環(huán)境：參考上一篇博客

derby數(shù)據(jù)庫(kù)：

輕量級(jí)數(shù)據(jù)庫(kù)，作為Hive Metastore；如果有時(shí)間搗鼓MySQL可以自行替換；

解壓即可運(yùn)行，注意執(zhí)行腳本的目錄即為數(shù)據(jù)目錄

啟動(dòng)derby參考：

#?PowerShell $env:DERBY_HOME="D:\install\db-derby-10.14.2.0-bin" #?創(chuàng)建目錄:?D:\install\db-derby-10.14.2.0-data cd?D:\install\db-derby-10.14.2.0-data #?注意直接使用環(huán)境變量可能不成功,?用tab補(bǔ)全絕對(duì)路徑后即可 $env:DERBY_HOME\bin\startNetworkServer.bat?-h?0.0.0.0

WSL / Cygwin / Git Bash

N選一，為了擁有一個(gè)類unix的環(huán)境

不像Hadoop，Hive太多地方?jīng)]有做好類unix和Windows的兼容

hive在windows下的可執(zhí)行文件

此處演示在WSL中下載

安裝svn: sudo apt-get install subversion

下載: svn co https://svn.apache.org/repos/asf/hive/trunk/bin hive-bin

-在svn上：https://svn.apache.org/repos/asf/hive/trunk/bin

可選：手動(dòng)逐個(gè)下載

可選：通過(guò)svn下載

配置步驟

解壓Hive包

可選：使用管理員權(quán)限運(yùn)行winrar解壓;

非管理員解壓，會(huì)導(dǎo)致linux軟連接無(wú)法轉(zhuǎn)換為windows快捷方式。但是會(huì)改成復(fù)制內(nèi)容。

打開(kāi)指定的tar.gz后，選擇解壓地址并解壓

D:\install\是我常用的軟件安裝目錄，各自實(shí)操時(shí)注意更換自己的解壓/安裝目錄

環(huán)境變量配置

環(huán)境變量

#?PowerShell #?Hadoop與JDK環(huán)境變量設(shè)置,?以防沒(méi)設(shè)置 $env:JAVA_HOME="D:\install\Java\jdk1.8.0_231" $env:HADOOP_HOME="D:\install\Hadoop-3.1.1\hadoop" $env:PATH="$env:PATH;$env:JAVA_HOME\bin;$env:HADOOP_HOME\bin" #?Hive環(huán)境變量設(shè)置 $env:HIVE_HOME="D:\install\Hive-3.1.0\hive-3.1.0" $env:DERBY_HOME="D:\install\db-derby-10.14.2.0-bin" $env:HIVE_LIB="$env:HIVE_HOME\lib" $env:HIVE_BIN="$env:HIVE_HOME\bin" $env:HADOOP_USER_CLASSPATH_FIRST="true"

復(fù)制derby lib到hive lib

cp D:\install\db-derby-10.14.2.0-bin\lib\* D:\install\Hive-3.1.0\hive-3.1.0\lib

配置hive-site.xml

D:\install\Hive-3.1.0\hive-3.1.0\conf 目錄下新建文件

添加內(nèi)容：

???? ????????javax.jdo.option.ConnectionURL ????????jdbc:derby://localhost:1527/metastore_db;create=true ???? ???? ????????javax.jdo.option.ConnectionDriverName ????????org.apache.derby.jdbc.ClientDriver ???? ???? ????????hive.server2.enable.doAs ????????true ???? ???? ????????hive.server2.authentication ????????NONE ???? ???? ????????datanucleus.autoCreateTables ????????True ???? ???? ????????hadoop.bin.path ????????/D:/install/Hadoop-3.1.1/hadoop/bin/hadoop.cmd ???? ???? ????????hive.jar.path ????????/D:/install/Hive-3.1.0/hive-3.1.0/lib/hive-exec-3.1.0.jar ????

Hadoop應(yīng)當(dāng)啟動(dòng)妥當(dāng)，參考上一份文檔

替換bin目錄下Windows執(zhí)行文件

hive-bin覆蓋到bin目錄下

初始化metastore

由于腳本是bash，只得依賴Cygwin或者WSL了 (注意使用bash環(huán)境, 而不是zsh環(huán)境)

初始化過(guò)程可能遇到問(wèn)題，參考下文的 其他問(wèn)題 章節(jié)

#?bash,?注意不是zsh #?Hadoop與JDK環(huán)境變量 #?WSL下的JDK與Windows下的有所區(qū)別,?注意替換 export?JAVA_HOME=/mnt/d/install/jdk/jdk1.8.0_231 export?HADOOP_HOME='/mnt/d/install/Hadoop-3.1.1/hadoop' export?PATH=$PATH:$HADOOP_HOME/bin #?Hive環(huán)境變量 export?HIVE_HOME='/mnt/d/install/Hive-3.1.0/hive-3.1.0' export?PATH=$PATH:$HIVE_HOME/bin export?HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$HIVE_HOME/lib/*.jar #?Hive目錄下的初始化腳本 cd?$HIVE_HOME ./bin/schematool?-dbType?derby?-initSchema

啟動(dòng)hive

這里并沒(méi)有啟動(dòng)hiveserver2，因?yàn)閱?dòng)不起來(lái) (手動(dòng)微笑)

所以只是用本地模式，主要的目的是為了方便接下來(lái)的功能性開(kāi)發(fā)與測(cè)試

#?PowerShell cd?$env:HIVE_HOME .\bin\hive.cmd

可用性測(cè)試

準(zhǔn)備數(shù)據(jù)

注意替換Windows風(fēng)格的換行符, hive處理時(shí)候不能正確處理，例如行尾的整數(shù)就沒(méi)法正確轉(zhuǎn)換。

2,zhangsan,98 7,lisi,87

建表

create?table?test?( ????id?INTEGER, ????name?STRING, ????grade?INTEGER )?row?format?delimited fields?terminated?by?',' lines?terminated?by?'\n';

從HDFS中讀入表中

load?data?inpath?"/tmp/samples.csv"?into?table?'test';

查詢測(cè)試

測(cè)試過(guò)程可能遇到問(wèn)題，參考下文的 其他問(wèn)題 章節(jié)

select?id,?count(1)?as?cnt?from?test?group?by?id; insert?into?table?test select?(id?+?10)?as?id,?name,?grade?from?test; create?table?test2?stored?as?ORC?as select?(id?+?10)?as?id,?name,?grade?from?test;

后續(xù)啟動(dòng)或停止

環(huán)境變量設(shè)置

#?PowerShell #?Hadoop與JDK環(huán)境變量 $env:JAVA_HOME="D:\install\Java\jdk1.8.0_231" $env:HADOOP_HOME="D:\install\Hadoop-3.1.1\hadoop" $env:PATH="$env:PATH;$env:JAVA_HOME\bin;$env:HADOOP_HOME\bin" #?Hive環(huán)境變量 $env:HIVE_HOME="D:\install\Hive-3.1.0\hive-3.1.0" $env:DERBY_HOME="D:\install\db-derby-10.14.2.0-bin" $env:HIVE_LIB="$env:HIVE_HOME\lib" $env:HIVE_BIN="$env:HIVE_HOME\bin" $env:HADOOP_USER_CLASSPATH_FIRST="true"

啟動(dòng)hadoop

啟動(dòng)derby

啟動(dòng)Hive

#?PowerShell cd?$env:HIVE_HOME .\bin\hive.cmd

日志目錄：C:/Users/${Windows用戶名}/AppData/Local/Temp/${Windows用戶名}/beeline.log

其他問(wèn)題

\r command not found

原因: Windows的換行符不兼容WSL/cygwin

替換: 例 sed -i 's/\r$//' hadoop

NumberFormatException: For input string: "0x100"

原因: 終端配置有誤

配置: export TERM=xterm-color，可以加入.bashrc

Error: FUNCTION 'NUCLEUS_ASCII' already exists.

derby重復(fù)創(chuàng)建，到derby啟動(dòng)的目錄刪除metastore_db

Create symbolic links error 創(chuàng)建符號(hào)連接失敗

hadoop無(wú)法創(chuàng)建符號(hào)鏈接，無(wú)法寫入hdfs

開(kāi)放給當(dāng)前用戶

hadoop配置篇也有說(shuō)明

java.io.FileNotFoundException: java.io.FileNotFoundException: HADOOP_HOME and hadoop.home.dir are unset.

只有配置在環(huán)境變量了

JAVA_HOME="D:\install\Java\jdk1.8.0_231" HADOOP_HOME="D:\install\Hadoop-3.1.1\hadoop" PATH="%PATH%;%JAVA_HOME%\bin;%HADOOP_HOME%\bin"

即使配置了環(huán)境變量，依然找不到HADOOP_HOME

etc/hadoop/yarn-site.xml設(shè)置的白名單少了HADOOP_HOME (自己坑自己)

放心，上一篇博文已經(jīng)改了這里

???? ????????yarn.nodemanager.aux-services ????????mapreduce_shuffle ???? ???? ????????yarn.nodemanager.env-whitelist ???????? ????????JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME,HADOOP_HOME,PATH,LANG,TZ ????

找不到hive-log4j.properties

D:\install\Hive-3.1.0\hive-3.1.0\conf目錄下新建文件hive-log4j.properties

添加內(nèi)容：

#?Licensed?to?the?Apache?Software?Foundation?(ASF)?under?one #?or?more?contributor?license?agreements.??See?the?NOTICE?file #?distributed?with?this?work?for?additional?information #?regarding?copyright?ownership.??The?ASF?licenses?this?file #?to?you?under?the?Apache?License,?Version?2.0?(the #?"License");?you?may?not?use?this?file?except?in?compliance #?with?the?License.??You?may?obtain?a?copy?of?the?License?at # #?????http://www.apache.org/licenses/LICENSE-2.0 # #?Unless?required?by?applicable?law?or?agreed?to?in?writing,?software #?distributed?under?the?License?is?distributed?on?an?"AS?IS"?BASIS, #?WITHOUT?WARRANTIES?OR?CONDITIONS?OF?ANY?KIND,?either?express?or?implied. #?See?the?License?for?the?specific?language?governing?permissions?and #?limitations?under?the?License. status?=?INFO name?=?HiveLog4j2 packages?=?org.apache.hadoop.hive.ql.log #?list?of?properties property.hive.log.level?=?INFO property.hive.root.logger?=?DRFA property.hive.log.dir?=?${sys:java.io.tmpdir}/${sys:user.name} property.hive.log.file?=?beeline.log property.hive.perflogger.log.level?=?INFO #?list?of?all?appenders appenders?=?console,?DRFA #?console?appender appender.console.type?=?Console appender.console.name?=?console appender.console.target?=?SYSTEM_ERR appender.console.layout.type?=?PatternLayout appender.console.layout.pattern?=?%d{ISO8601}?%5p?[%t]?%c{2}:?%m%n #?daily?rolling?file?appender appender.DRFA.type?=?RollingRandomAccessFile appender.DRFA.name?=?DRFA appender.DRFA.fileName?=?${sys:hive.log.dir}/${sys:hive.log.file} #?Use?%pid?in?the?filePattern?to?append?@?to?the?filename?if?you?want?separate?log?files?for?different?CLI?session appender.DRFA.filePattern?=?${sys:hive.log.dir}/${sys:hive.log.file}.%d{yyyy-MM-dd} appender.DRFA.layout.type?=?PatternLayout appender.DRFA.layout.pattern?=?%d{ISO8601}?%5p?[%t]?%c{2}:?%m%n appender.DRFA.policies.type?=?Policies appender.DRFA.policies.time.type?=?TimeBasedTriggeringPolicy appender.DRFA.policies.time.interval?=?1 appender.DRFA.policies.time.modulate?=?true appender.DRFA.strategy.type?=?DefaultRolloverStrategy appender.DRFA.strategy.max?=?30 #?list?of?all?loggers loggers?=?NIOServerCnxn,?ClientCnxnSocketNIO,?DataNucleus,?Datastore,?JPOX,?PerfLogger,?AmazonAws,?ApacheHttp logger.NIOServerCnxn.name?=?org.apache.zookeeper.server.NIOServerCnxn logger.NIOServerCnxn.level?=?WARN logger.ClientCnxnSocketNIO.name?=?org.apache.zookeeper.ClientCnxnSocketNIO logger.ClientCnxnSocketNIO.level?=?WARN logger.DataNucleus.name?=?DataNucleus logger.DataNucleus.level?=?ERROR logger.Datastore.name?=?Datastore logger.Datastore.level?=?ERROR logger.JPOX.name?=?JPOX logger.JPOX.level?=?ERROR logger.AmazonAws.name=com.amazonaws logger.AmazonAws.level?=?INFO logger.ApacheHttp.name=org.apache.http logger.ApacheHttp.level?=?INFO logger.PerfLogger.name?=?org.apache.hadoop.hive.ql.log.PerfLogger logger.PerfLogger.level?=?${sys:hive.perflogger.log.level} #?root?logger rootLogger.level?=?${sys:hive.log.level} rootLogger.appenderRefs?=?root rootLogger.appenderRef.root.ref?=?${sys:hive.root.logger}

參考文檔

https://towardsdatascience.com/installing-apache-hive-3-1-2-on-windows-10-70669ce79c79

https://stackoverflow.com/questions/11616835/r-command-not-found-bashrc-bash-profile

https://stackoverflow.com/questions/56457685/how-to-fix-the-sbt-crash-java-lang-numberformatexception-for-input-string-0x

https://stackoverflow.com/questions/35655306/hive-installation-issues-hive-metastore-database-is-not-initialized

https://blog.csdn.net/Harvay/article/details/83024127

Windows Hive EI企業(yè)智能 智能數(shù)據(jù) 表格存儲(chǔ)服務(wù) CloudTable

版權(quán)聲明：本文內(nèi)容由網(wǎng)絡(luò)用戶投稿，版權(quán)歸原作者所有，本站不擁有其著作權(quán)，亦不承擔(dān)相應(yīng)法律責(zé)任。如果您發(fā)現(xiàn)本站中有涉嫌抄襲或描述失實(shí)的內(nèi)容，請(qǐng)聯(lián)系我們jiasou666@gmail.com 處理，核實(shí)后本網(wǎng)站將在24小時(shí)內(nèi)刪除侵權(quán)內(nèi)容。