GaussDB T 分布式集群部署（1）

GaussDB 強大于ORACLE的一個核心在于其可以部署分布式集群,且有實際部署使用案例，理論上水平橫向支持擴展1024個DN節點，實際案例中已有500+DN的大規模分布式集群。本節簡要記錄4節點單機房分布式高可用架構部署過程。

環境說明：

4臺虛擬機：

操作系統 redhat 7.5? mini安裝

Gauss1???2C 8G 100G????????192.168.10.11/16 10.10.10.11/16

Gauss2???1C 6G 100G????????192.168.10.12/16 10.10.10.12/16

Gauss3???1C 6G 100G????????192.168.10.13/16 10.10.10.13/16

Gauss4???1C 6G 100G????????192.168.10.14/16 10.10.10.14/16

注：磁盤規劃要點：每個實例至少保證20G，比如下邊部署方案中除了4節點是3實例外，其他3個節點都是4個實例；本例網絡平面分業務平面（192.168.0.0）和維護平面（10.10.0.0），生產環境還需加入備份平面，以提升網絡獨立性和安全性；使用默認模版安裝對內存要求較高，可通過修改建庫模版文件中內存參數降低內存消耗。修改后每個節點2G內存即可。

部署方案：

注：

CN: 協調節點，負責分解任務，調度任務分片在DN上并行執行；

CM: （Cluster Manager） 集群管理模塊，管理和監控各個實例運行情況，確保系統穩定運行；

ETCD:?是一個高可用的分布式鍵值（KEY-VALUE）數據庫，收集實例狀態信息，CM從ETCD獲取實例狀態信息；

DN: (DataNode)數據節點，負責存儲業務數據，執行數據查詢結果并向CN返回結果；

AZ1：1號機房（單機房）；

1.系統初始化（所有主機均執行）：

1>?關閉防火墻

[root@Gauss1 ~]# systemctl status firewalld.service

[root@Gauss1 ~]# systemctl stop firewalld.service

[root@Gauss1 ~]# systemctl disable firewalld.service

[root@Gauss1 ~]# firewall-cmd --state

not running

[root@Gauss1 ~]#

2>?優化內核參數

/etc/sysctl.conf追加如下內容：

net.ipv4.tcp_tw_reuse = 1

net.ipv4.tcp_tw_recycle = 1

net.ipv4.tcp_keepalive_time = 30

net.ipv4.tcp_keepalive_probes = 9

net.ipv4.tcp_keepalive_intvl = 30

net.ipv4.tcp_retries2 = 80

vm.overcommit_memory = 0

net.ipv4.tcp_rmem = 8192 250000 16777216

net.core.wmem_max = 21299200

net.core.rmem_max = 21299200

net.core.wmem_default = 21299200

net.core.rmem_default = 21299200

kernel.sem = 250 6400000 1000 25600

vm.min_free_kbytes = 314572 ? ? ? ? ? ? --預留系統內存5%

net.core.somaxconn = 65535

net.ipv4.tcp_syncookies = 1

net.core.netdev_max_backlog = 65535

net.ipv4.tcp_max_syn_backlog = 65535

net.ipv4.tcp_wmem = 8192 250000 16777216

net.ipv4.tcp_max_tw_buckets = 10000

增加完成后執行sysctl -p刷新

3>?安裝依賴包

配置本地YUM源：

echo "/dev/sr0 mnt/iso iso9660 defaults,loop 0 0" >> etc/fstab

mkdir -p mnt/iso

rm -rf etc/yum.repos.d/*

cat >> etc/yum.repos.d/ios.repo <

[ios]

name=ios

baseurl=file:///mnt/iso

enabled=1

gpgcheck=0

EOF

mount -a

yum repolist

安裝依賴包

[root@Gauss1 yum.repos.d]# unset uninstall_rpm;for i in zlib readline gcc python python-devel perl-ExtUtils-Embed readline-devel zlib-devel lsof;do rpm -q $i &>/dev/null || uninstall_rpm="$uninstall_rpm $i";done ;[[ -z "$uninstall_rpm" ]] && echo -e "\nuninstall_rpm:\n\tOK.OK.OK" || ?echo -e "\nuninstall_rpm:\n\t$uninstall_rpm"

uninstall_rpm:

gcc python-devel perl-ExtUtils-Embed readline-devel zlib-devel lsof

[root@Gauss1?yum.repos.d]#?yum?-y?install?gcc?python-devel?perl-ExtUtils-Embed?readline-devel?zlib-devel?lsof

[root@Gauss1?yum.repos.d]#?unset?uninstall_rpm;for?i?in?zlib?readline?gcc?python?python-devel?perl-ExtUtils-Embed?readline-devel?zlib-devel?lsof;do?rpm?-q?$i?&>/dev/null?||?uninstall_rpm="$uninstall_rpm?$i";done?;[[?-z?"$uninstall_rpm"?]]?&&?echo?-e?"\nuninstall_rpm:\n\tOK.OK.OK"?||??echo?-e?"\nuninstall_rpm:\n\t$uninstall_rpm"

uninstall_rpm:

OK.OK.OK

[root@Gauss1?yum.repos.d]#

2.上傳安裝包（后續操作僅一個節點）

[root@Gauss1?~]#?mkdir?-p?/opt/software/gaussdb

[root@Gauss1?~]#?cd?/opt/software/gaussdb

[root@Gauss1?gaussdb]#?rz

[root@Gauss1 gaussdb]# tar -zxvf GaussDB_100_1.0.0-CLUSTER-REDHAT7.5-64bit.tar.g

[root@Gauss1 gaussdb]# ll

total 134984

-rw-r--r--. ?1 root root 86248211 Dec 14 20:18 GaussDB_100_1.0.0-CLUSTER-REDHAT7.5-64bit.tar.gz

-rw-r--r--. ?1 root root 44364163 Jul 29 19:14 GaussDB_100_1.0.0-CM-REDHAT-64bit.tar.gz

-rw-r--r--. ?1 root root ?7478115 Jul 29 19:16 GaussDB_100_1.0.0-DATABASE-REDHAT-64bit.tar.gz

-rw-r--r--. ?1 root root ? 118593 Jul 29 19:12 GaussDB_100_1.0.0-ROACH-REDHAT-64bit.tar.gz

drwx------. 16 root root ? ? 4096 Jul 29 19:18 lib

drwx------. ?6 root root ? ? 4096 Jul 29 19:11 script

drwxr-xr-x. ?5 root root ? ? ? 49 Jul 29 19:11 shardingscript

[root@Gauss1?gaussdb]#

3.編輯集群配置文件

創建配置文件/opt/software/gaussdb/clusterconfig.xml，內容如下：

4.集群初始化

執行環境初始化腳本，執行過程會自動配置安裝環境，創建用戶，環境變量，目錄等。

[root@Gauss1 script]# ./gs_preinstall -U omm -G dbgrp -X opt/software/gaussdb/clusterconfig.xml

Parsing the configuration file.

Successfully parsed the configuration file.

Installing the tools on the local node.

Successfully installed the tools on the local node.

Are you sure you want to create trust for root (yes/no)? yes

Please enter password for root.

Password:

Creating SSH trust for the root permission user.

Checking network information.

All nodes in the network are Normal.

Successfully checked network information.

Creating SSH trust.

Creating the local key file.

Successfully created the local key files.

Appending local ID to authorized_keys.

Successfully appended local ID to authorized_keys.

Updating the known_hosts file.

Successfully updated the known_hosts file.

Appending authorized_key on the remote node.

Successfully appended authorized_key on all remote node.

Checking common authentication file content.

Successfully checked common authentication content.

Distributing SSH trust file to all node.

Successfully distributed SSH trust file to all node.

Verifying SSH trust on all hosts.

Successfully verified SSH trust on all hosts.

Successfully created SSH trust.

Successfully created SSH trust for the root permission user.

[GAUSS-52406] : The package type "" is inconsistent with the Cpu type "X86".

[root@Gauss1 script]#

報錯處理：

root@Gauss1?~]#?cd?/opt/gaussdb/log/omm/om

[root@Gauss1 om]# ll

total 20

-rw-------. 1 root root 1953 Dec 14 20:44 gs_local-2019-12-14_204431.log

-rw-------. 1 root root 9723 Dec 14 20:45 gs_preinstall-2019-12-14_204425.log

-rw-------. 1 root root ? 13 Dec 14 20:44 topDirPath.dat

[root@Gauss1 om]# tail gs_preinstall-2019-12-14_204425.log

2019-12-14 20:45:08.844963][gs_sshexkey][LOG]:Successfully appended authorized_key on all remote node.

[2019-12-14 20:45:08.845034][gs_sshexkey][LOG]:Checking common authentication file content.

[2019-12-14 20:45:08.845412][gs_sshexkey][LOG]:Successfully checked common authentication content.

[2019-12-14 20:45:08.845484][gs_sshexkey][LOG]:Distributing SSH trust file to all node.

[2019-12-14 20:45:09.251582][gs_sshexkey][LOG]:Successfully distributed SSH trust file to all node.

[2019-12-14 20:45:09.251687][gs_sshexkey][LOG]:Verifying SSH trust on all hosts.

[2019-12-14 20:45:09.658347][gs_sshexkey][LOG]:Successfully verified SSH trust on all hosts.

[2019-12-14 20:45:09.658447][gs_sshexkey][LOG]:Successfully created SSH trust.

[2019-12-14 20:45:09.681885][gs_preinstall][LOG]:Successfully created SSH trust for the root permission user.

[2019-12-14 20:45:10.539785][gs_preinstall][ERROR]:[GAUSS-52406] : The package type "" is inconsistent with the Cpu type "X86".

Traceback (most recent call last)

File "./gs_preinstall", line 507, in

File "/opt/software/gaussdb/script/impl/preinstall/PreinstallImpl.py", line 1861, in run

[root@Gauss1 om]# vi opt/software/gaussdb/script/impl/preinstall/PreinstallImpl.py

1760 ? ? ? ? self.createTrustForRoot()

1761 ? ? ? ? #cpu ?Detection

1762 ? ? ? ? #self.getAllCpu() ? ? ? ? ? ? ? --注釋掉

1763 ? ? ? ? #ram ?Detection

1764 ? ? ? ? self.getAllRam()

1765 ? ? ? ? # LVM

再次執行

[root@Gauss1 script]# ./gs_preinstall -U omm -G dbgrp -X opt/software/gaussdb/clusterconfig.xml

Parsing the configuration file.

Successfully parsed the configuration file.

Installing the tools on the local node.

Successfully installed the tools on the local node.

Are you sure you want to create trust for root (yes/no)? yes

Please enter password for root.

Password:

Creating SSH trust for the root permission user.

Checking network information.

All nodes in the network are Normal.

Successfully checked network information.

Creating SSH trust.

Creating the local key file.

Successfully created the local key files.

Appending local ID to authorized_keys.

Successfully appended local ID to authorized_keys.

Updating the known_hosts file.

Successfully updated the known_hosts file.

Appending authorized_key on the remote node.

Successfully appended authorized_key on all remote node.

Checking common authentication file content.

Successfully checked common authentication content.

Distributing SSH trust file to all node.

Successfully distributed SSH trust file to all node.

Verifying SSH trust on all hosts.

Successfully verified SSH trust on all hosts.

Successfully created SSH trust.

Successfully created SSH trust for the root permission user.

All host RAM is consistent

Pass over configuring LVM

Distributing package.

Successfully distributed package.

Are you sure you want to create the user[omm] and create trust for it (yes/no)? yes

Please enter password for cluster user.

Password:

Please enter password for cluster user again.

Password:

Creating [omm] user on all nodes.

Successfully created [omm] user on all nodes.

Installing the tools in the cluster.

Successfully installed the tools in the cluster.

Checking hostname mapping.

Successfully checked hostname mapping.

Creating SSH trust for [omm] user.

Please enter password for current user[omm].

Password:

Checking network information.

All nodes in the network are Normal.

Successfully checked network information.

Creating SSH trust.

Creating the local key file.

Successfully created the local key files.

Appending local ID to authorized_keys.

Successfully appended local ID to authorized_keys.

Updating the known_hosts file.

Successfully updated the known_hosts file.

Appending authorized_key on the remote node.

Successfully appended authorized_key on all remote node.

Checking common authentication file content.

Successfully checked common authentication content.

Distributing SSH trust file to all node.

Successfully distributed SSH trust file to all node.

Verifying SSH trust on all hosts.

Successfully verified SSH trust on all hosts.

Successfully created SSH trust.

Successfully created SSH trust for [omm] user.

Checking OS version.

Successfully checked OS version.

Creating cluster's path.

Successfully created cluster's path.

Setting SCTP service.

Successfully set SCTP service.

Set and check OS parameter.

Successfully set NTP service.

Setting OS parameters.

Successfully set OS parameters.

Warning: Installation environment contains some warning messages.

Please get more details by "/opt/software/gaussdb/script/gs_checkos -i A -h Gauss1,Gauss2,Gauss3,Gauss4 -X opt/software/gaussdb/clusterconfig.xml".

Set and check OS parameter completed.

Preparing CRON service.

Successfully prepared CRON service.

Preparing SSH service.

Successfully prepared SSH service.

Setting user environmental variables.

Successfully set user environmental variables.

Configuring alarms on the cluster nodes.

Successfully configured alarms on the cluster nodes.

Setting the dynamic link library.

Successfully set the dynamic link library.

Fixing server package owner.

Successfully fixed server package owner.

Create logrotate service.

Successfully create logrotate service.

Setting finish flag.

Successfully set finish flag.

check time consistency(maximum execution time 10 minutes).

Time consistent is running(20/20)...

Preinstallation succeeded.

[root@Gauss1 script]#

執行告警的腳本

[root@Gauss1 script]# opt/software/gaussdb/script/gs_checkos -i A -h Gauss1,Gauss2,Gauss3,Gauss4 -X opt/software/gaussdb/clusterconfig.xml

Checking items:

A1. [ OS version status ] ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? : Normal

A2. [ Kernel version status ] ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? : Normal

A3. [ Unicode status ] ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?: Normal

A4. [ Time zone status ] ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?: Normal

A5. [ Swap memory status ] ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?: Normal

A6. [ System control parameters status ] ? ? ? ? ? ? ? ? ? ?: Normal

A7. [ File system configuration status ] ? ? ? ? ? ? ? ? ? ?: Normal

A8. [ Disk configuration status ] ? ? ? ? ? ? ? ? ? ? ? ? ? : Normal

A9. [ Pre-read block size status ] ? ? ? ? ? ? ? ? ? ? ? ? ?: Normal

A10.[ IO scheduler status ] ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? : Normal

A12.[ Time consistency status ] ? ? ? ? ? ? ? ? ? ? ? ? ? ? : Warning

A13.[ Firewall service status ] ? ? ? ? ? ? ? ? ? ? ? ? ? ? : Normal

A14.[ THP service status ] ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?: Normal

A15.[ Storcli tool status ] ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? : Normal

Total numbers:14. Abnormal numbers:0. Warning numbers:1.

[root@Gauss1 script]#

分布式環境對時間要求很高，測試環境配置Gauss1為時鐘服務器，其他節點同步

Gauss1:

[root@Gauss1?~]#?yum?-y?install?ntp ntpdate

[root@Gauss1?~]#?echo?server 127.127.1.0 iburst?>>?/etc/ntp.conf

[root@Gauss1 ~]# systemctl restart ntpd

[root@Gauss1 ~]# systemctl enable ntpd

[root@Gauss1?~]#?ntpq?-p

remote ? ? ? ? ? refid ? ? ?st t when poll reach ? delay ? offset ?jitter

==============================================================================

*LOCAL(0)????????.LOCL.???????????5?l???12???64????1????0.000????0.000???0.000

[root@Gauss1 ~]#

Gauss2,3,4做如下配置:

[root@Gauss2 ~]# yum -y install ntp ntpdate

[root@Gauss2 ~]# echo server 192.168.10.11 >> /etc/ntp.conf

[root@Gauss2?~]#?echo?restrict?192.168.10.11?nomodify?notrap?noquery >> /etc/ntp.conf

[root@Gauss2 ~]# ntpdate -u 192.168.10.11

[root@Gauss2?~]#?hwclock??-w

[root@Gauss2 ~]# systemctl restart ntpd

[root@Gauss2 ~]# systemctl enable ntpd

[root@Gauss2 ~]# ntpq -p

remote ? ? ? ? ? refid ? ? ?st t when poll reach ? delay ? offset ?jitter

==============================================================================

Gauss1 ? ? ? ? ?LOCAL(0) ? ? ? ? 6 u ? 10 ? 64 ? ?1 ? ?0.416 ?197.856 ? 0.000

[root@Gauss2 ~]#

重新檢測：

[root@Gauss1 ~]# /opt/software/gaussdb/script/gs_checkos -i A -h Gauss1,Gauss2,Gauss3,Gauss4 -X /opt/software/gaussdb/clusterconfig.xml

Checking items:

A1. [ OS version status ] ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? : Normal

A2. [ Kernel version status ] ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? : Normal

A3. [ Unicode status ] ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?: Normal

A4. [ Time zone status ] ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?: Normal

A5. [ Swap memory status ] ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?: Normal

A6. [ System control parameters status ] ? ? ? ? ? ? ? ? ? ?: Normal

A7. [ File system configuration status ] ? ? ? ? ? ? ? ? ? ?: Normal

A8. [ Disk configuration status ] ? ? ? ? ? ? ? ? ? ? ? ? ? : Normal

A9. [ Pre-read block size status ] ? ? ? ? ? ? ? ? ? ? ? ? ?: Normal

A10.[ IO scheduler status ] ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? : Normal

A12.[ Time consistency status ] ? ? ? ? ? ? ? ? ? ? ? ? ? ? : Normal

A13.[ Firewall service status ] ? ? ? ? ? ? ? ? ? ? ? ? ? ? : Normal

A14.[ THP service status ] ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?: Normal

A15.[ Storcli tool status ] ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? : Normal

Total numbers:14. Abnormal numbers:0. Warning numbers:0.

[root@Gauss1 ~]#

本文轉自“墨天輪”社區GaussDB頻道

<未完待續>

下一篇:? GaussDB T 分布式集群部署（2）https://bbs.huaweicloud.com/blogs/140821

GaussDB

版權聲明：本文內容由網絡用戶投稿，版權歸原作者所有，本站不擁有其著作權，亦不承擔相應法律責任。如果您發現本站中有涉嫌抄襲或描述失實的內容，請聯系我們jiasou666@gmail.com 處理，核實后本網站將在24小時內刪除侵權內容。

版權聲明：本文內容由網絡用戶投稿，版權歸原作者所有，本站不擁有其著作權，亦不承擔相應法律責任。如果您發現本站中有涉嫌抄襲或描述失實的內容，請聯系我們jiasou666@gmail.com 處理，核實后本網站將在24小時內刪除侵權內容。