MRS : flume實時提交日志文件到hdfs系統

MRS : flume實時提交日志文件到hdfs系統

關鍵詞:MRS? flume? hdfs? Kerberos 日志文件

摘要: 本文主要介紹了在MRS集群環境,如何使用flume客戶端從日志主機收集日志保存至hdfs指定文件夾

前期準備

1.?????? 創建集群,參考https://support.huaweicloud.com/usermanual-mrs/mrs_01_0027.html.集群選擇mrs2.1.0的混合集群, Kerberos認證開啟,組件至少包含hadoop,flume.

2.?????? 參考https://support.huaweicloud.com/usermanual-mrs/mrs_01_0091.html中”前提條件”小標題,創建集群外ECS日志主機節點.注意這臺主機的VPC和安全組和集群保持一致.

3.?????? 使用Flume搜集日志時，需要在日志主機上安裝Flume客戶端,參考鏈接https://support.huaweicloud.com/usermanual-mrs/mrs_01_0392.html

3.1?????? 鏈接文檔中沒有flume客戶端的下載步驟,參考下圖

sh /opt/MRS_Flume_ClientConfig/Flume/install.sh -d /opt/FlumeClient -f 172.16.0.135? -e 172.16.0.100

開發程序(參考https://support.huaweicloud.com/usermanual-mrs/mrs_01_0397.html)

2.?????? 在集群內的master節點將$HADOOP_HOME/etc/hadoop目錄下的hdfs-site.xml和core-site.xml文件發送到日志主機節點的flume客戶端的配置目錄"/opt/FlumeClient/fusioninsight-flume-1.6.0/conf"下

3.?????? 在MRS Manage確認flume角色所在的機器節點ip,登錄這個節點,將目錄/opt/Bigdata/MRS_x.x.x/1_x_Flume/etc/"下的jaas.conf文件發送到日志主機節點的flume客戶端的配置目錄"/opt/FlumeClient/fusioninsight-flume-1.6.0/conf"下

4.?????? 登錄到日志主機節點"/opt/FlumeClient/fusioninsight-flume-1.6.0/conf"目錄下,完成配置

4.1??? 修改jaas.conf內容,參考下圖, principal是MRS Manager創建的用戶名, keyTab是keytab認證文件的路徑.

4.2??? 保證配置文件對于程序具有訪問權限,修改keytab文件的權限,例如”chmod 777 user.keytab”

4.3??? 修改flume-env.sh,在 “-XX:+UseCMSCompactAtFullCollection”后面,增加以下內容:” -Djava.security.krb5.conf=/opt/FlumeClient/fusioninsight-flume-1.6.0/conf/krb5.conf -Djava.security.auth.login.config=/opt/FlumeClient/fusioninsight-flume-1.6.0/conf/jaas.conf -Dzookeeper.request.timeout=120000”

4.4??? 將以下內容覆蓋寫入properties.properties文件中

client.sources = r1

client.sinks = k1

client.channels = c1

client.sources.r1.type = spooldir

client.sources.r1.spoolDir = /var/log/test

client.sources.r1.trackerDir = /opt/FlumeClient/fusioninsight-flume-1.6.0/conf/trackerDir

client.sources.r1.ignorePattern = ^$

client.sources.r1.fileSuffix = .COMPLETED

client.sources.r1.maxBlobLength = 16384

client.sources.r1.batchSize = 51200

client.sources.r1.inputCharset = UTF-8

client.sources.r1.deserializer = LINE

client.sources.r1.selector.type = replicating

client.sources.r1.fileHeaderKey = file

client.sources.r1.fileHeader = false

client.sources.r1.basenameHeader = true

client.sources.r1.basenameHeaderKey = basename

client.sources.r1.deletePolicy = never

client.sources.r1.channels = c1

client.channels.c1.type = file

client.channels.c1.checkpointDir = /opt/FlumeClient/fusioninsight-flume-1.6.0/conf/checkpointDir/

client.channels.c1.dataDirs = /opt/FlumeClient/fusioninsight-flume-1.6.0/conf/dataDirs/

client.channels.c1.maxFileSize = 2146435071

client.channels.c1.minimumRequiredSpace = 524288000

client.channels.c1.capacity = 1000000

client.channels.c1.transactionCapacity = 10000

client.channels.c1.channelfullcount = 10

client.sinks.k1.type = hdfs

client.sinks.k1.channel = c1

client.sinks.k1.hdfs.path = /flume/file/%y-%m-%d/%H%M/

client.sinks.k1.hdfs.filePrefix = event

client.sinks.k1.hdfs.rollSize = 102400000

client.sinks.k1.hdfs.fileType= DataStream

client.sinks.k1.hdfs.rollInterval = 600

client.sinks.k1.hdfs.rollCount = 0

cient.sinks.k1.hdfs.inUseSuffix = .tmp

client.sinks.k1.hdfs.fileSuffix = .log

client.sinks.k1.hdfs.idleTimeout = 0

client.sinks.k1.hdfs.useLocalTimeStamp = true

client.sinks.k1.hdfs.round = true

client.sinks.k1.hdfs.roundValue = 10

client.sinks.k1.hdfs.roundUnit = minute

client.sinks.k1.hdfs.useLocalTimeStamp = true

client.sinks.k1.hdfs.kerberosPrincipal = flumeuser

client.sinks.k1.hdfs.kerberosKeytab= /opt/FlumeClient/fusioninsight-flume-1.6.0/conf/user.keytab

4.5??? 在目錄下面創建下面紅框標記的文件夾

5.?????? 在日志節點Flume客戶端,執行一下命令

cd /opt/FlumeClient/fusioninsight-flume-1.6.0/bin

./flume-manage.sh restart

測試結果

1.?????? Flume監控的日志文件是集群外日志主機節點下的 /var/log/test”目錄,可以通過向此目錄寫入文本,然后在集群內hdfs的/flume/file目錄下查看是否實時同步

附件： 集群外節點flume客戶端 source日志文件sink hdfs .docx 439.18KB 下載次數：5次

MapReduce服務

版權聲明：本文內容由網絡用戶投稿，版權歸原作者所有，本站不擁有其著作權，亦不承擔相應法律責任。如果您發現本站中有涉嫌抄襲或描述失實的內容，請聯系我們jiasou666@gmail.com 處理，核實后本網站將在24小時內刪除侵權內容。

版權聲明：本文內容由網絡用戶投稿，版權歸原作者所有，本站不擁有其著作權，亦不承擔相應法律責任。如果您發現本站中有涉嫌抄襲或描述失實的內容，請聯系我們jiasou666@gmail.com 處理，核實后本網站將在24小時內刪除侵權內容。