Columnstore multi combined install and addmodule (2017)

OS : Centos 6.8
CS : Columnstore 1.1.2
server1 : 192.168.0.195
server2 : 192.168.0.197
server3 : 192.168.0.199

++ 추가 : binary 도 addmodule 가능

Installing and Configuring a Multi Server ColumnStore System - 1.1.X

How to install ColumnStore on a multi-server system

mariadb.com

내가 난독인지 모르겠지만 너무 모르겠더라...


멀티인스톨 / combined 설치   를 위해 여러가지 테스트를 진행.
binary는 install 시 user와 group명이 같지 않으면 진행이 불가능하다고 판단.
rpm버전으로 테스트 진행

UM으로 사용할 서버(server2) 에서 설치 진행
초기설치는 server1(pm), server2(um, pm) 및
addmodule 을 통한 server3(pm)추가

1. rpm 압축해제
     # tar zxvf mariadb-columnstore-1.1.2-1-centos6.x86_64.rpm.tar.gz 

2. 기 설치 mysql 제거
     # rpm -e mysql-libs --nodeps

3. columnstore rpm install
     # yum install -y mariadb-columnstore-1.1.2-1-*rpm

4. centos_boost 
     # tar zxvf centos6_boost_1_55.tar.gz -C /usr/local/mariadb/columnstore/lib/

5. post-install
     # cd /usr/local/mariadb/columnstore/bin
     # ./post-install 

6. hosts 설정
     # vi /etc/hosts

     192.168.0.195   server1
     192.168.0.197   server2
     192.168.0.199   server3

7. ssh-keygen
     # ssh-keygen

8. ssh-copy-id
     # ssh-copy-id -i ~/.ssh/id_rsa.pub server1
     # ssh-copy-id -i ~/.ssh/id_rsa.pub server2
     # ssh server1

9. 설치 rpm들 home directory 로 이동
     # mv /download/mariadb-columnstore-1.1.2-1*.rpm  /root
     # mv /download/mariadb-columnstore-1.1.2-1*.tar.gz /root 

10. 방화벽 해제
     # service iptables stop
     # service iptables status

11. columnstoreClusterTester를 위한 설치 
     # yum install -y rsync perl-DBD-MySQL nmap



  ( 여기까지 server1, server2, server3 모두 진행 )  

 

여기서부터 server2 에서 실행


12. columnstoreClusterTester 를 통해 모든 테스트 통과 메세지 확인
     # cd /usr/local/mariadb/columnstore/bin/
     # ./columnstoreClusterTester.sh  --ipaddr=192.168.0.195

      *** Finished Validation of the Cluster, all Test Passed ***   

13. postConfigure 
     # /usr/local/mariadb/columnstore/bin/postConfigure

     multi
     combined
     internal
     pm (2)  - 1  server2
                  - 2  server1

14. alias 등록
     # . /usr/local/mariadb/columnstore/bin/columnstoreAlias

15. 설치확인
     # mcsadmin

mcsadmin> getsystemi getsysteminfo Thu Feb 22 09:39:12 2018 System columnstore-1 System and Module statuses Component Status Last Status Change ------------ -------------------------- ------------------------ System ACTIVE Thu Feb 22 09:38:29 2018 Module pm1 ACTIVE Thu Feb 22 09:38:27 2018 Module pm2 ACTIVE Thu Feb 22 09:38:22 2018 Active Parent OAM Performance Module is 'pm1' Primary Front-End MariaDB ColumnStore Module is 'pm1' MariaDB ColumnStore Replication Feature is enabled MariaDB ColumnStore Process statuses Process Module Status Last Status Change Process ID ------------------ ------ --------------- ------------------------ ---------- ProcessMonitor pm1 ACTIVE Thu Feb 22 09:37:11 2018 4318 ProcessManager pm1 ACTIVE Thu Feb 22 09:37:17 2018 4403 DBRMControllerNode pm1 ACTIVE Thu Feb 22 09:38:00 2018 5424 ServerMonitor pm1 ACTIVE Thu Feb 22 09:38:02 2018 5458 DBRMWorkerNode pm1 ACTIVE Thu Feb 22 09:38:02 2018 5489 DecomSvr pm1 ACTIVE Thu Feb 22 09:38:06 2018 5614 PrimProc pm1 ACTIVE Thu Feb 22 09:38:08 2018 5679 ExeMgr pm1 ACTIVE Thu Feb 22 09:38:15 2018 6569 WriteEngineServer pm1 ACTIVE Thu Feb 22 09:38:20 2018 6705 DDLProc pm1 ACTIVE Thu Feb 22 09:38:24 2018 6836 DMLProc pm1 ACTIVE Thu Feb 22 09:38:28 2018 6921 mysqld pm1 ACTIVE Thu Feb 22 09:38:18 2018 5330 ProcessMonitor pm2 ACTIVE Thu Feb 22 09:37:53 2018 5497 ProcessManager pm2 HOT_STANDBY Thu Feb 22 09:37:57 2018 5564 DBRMControllerNode pm2 COLD_STANDBY Thu Feb 22 09:38:03 2018 ServerMonitor pm2 ACTIVE Thu Feb 22 09:38:06 2018 5869 DBRMWorkerNode pm2 ACTIVE Thu Feb 22 09:38:07 2018 5882 DecomSvr pm2 ACTIVE Thu Feb 22 09:38:10 2018 5896 PrimProc pm2 ACTIVE Thu Feb 22 09:38:13 2018 5928 ExeMgr pm2 ACTIVE Thu Feb 22 09:38:17 2018 5952 WriteEngineServer pm2 ACTIVE Thu Feb 22 09:38:21 2018 5969 DDLProc pm2 COLD_STANDBY Thu Feb 22 09:38:22 2018 DMLProc pm2 COLD_STANDBY Thu Feb 22 09:38:22 2018 mysqld pm2 ACTIVE Thu Feb 22 09:38:10 2018 5812 Active Alarm Counts: Critical = 0, Major = 0, Minor = 0, Warning = 0, Info = 0

16. pm 모듈 추가

# dbroot 추가 mcsadmin> adddbroot 1 adddbroot Thu Feb 22 09:44:42 2018 New DBRoot IDs added = 3 # dbroot 추가 확인 mcsadmin> getstorageconfig getstorageconfig Thu Feb 22 09:44:46 2018 System Storage Configuration Performance Module (DBRoot) Storage Type = internal System Assigned DBRoot Count = 2 DBRoot IDs assigned to 'pm1' = 1 DBRoot IDs assigned to 'pm2' = 2 DBRoot IDs unassigned = 3 # server3 addmodule mcsadmin> addmodule pm 1 server3 addmodule Thu Feb 22 09:45:43 2018 Adding Modules pm3, please wait... Add Module(s) successfully completed addModule Command Successfully completed: Modules are Disabled, run alterSystem-enableModule command to enable them mcsadmin> altersyhstem-enablemodule pm3 altersyhstem-enablemodule: Unknown Command, type help for list of commands mcsadmin> altersystem-enablemodule pm3 altersystem-enablemodule Thu Feb 22 09:47:06 2018 This command starts the processing of applications on a Module within the MariaDB ColumnStore System Do you want to proceed: (y or n) [n]: y Enabling Modules Successful enable of Modules Performance Module(s) Enabled, run movePmDbrootConfig or assignDbrootPmConfig to assign dbroots, if needed # assigndbrootpmconfig mcsadmin> assigndbrootpmconfig 3 pm3 assigndbrootpmconfig Thu Feb 22 09:47:24 2018 DBRoot IDs assigned to 'pm3' = Changes being applied DBRoot IDs assigned to 'pm3' = 3 Successfully Assigned DBRoots # pm3 확인 mcsadmin> getstorageconfig getstorageconfig Thu Feb 22 09:47:30 2018 System Storage Configuration Performance Module (DBRoot) Storage Type = internal System Assigned DBRoot Count = 3 DBRoot IDs assigned to 'pm1' = 1 DBRoot IDs assigned to 'pm2' = 2 DBRoot IDs assigned to 'pm3' = 3

17. 시스템 확인  :  pm3가 정상이 아닌걸로 나온다.   stop and start

mcsadmin> getsystemi getsysteminfo Thu Feb 22 09:47:44 2018 System columnstore-1 System and Module statuses Component Status Last Status Change ------------ -------------------------- ------------------------ System ACTIVE Thu Feb 22 09:38:29 2018 Module pm1 ACTIVE Thu Feb 22 09:38:27 2018 Module pm2 ACTIVE Thu Feb 22 09:38:22 2018 Module pm3 MAN_OFFLINE Thu Feb 22 09:47:07 2018 Active Parent OAM Performance Module is 'pm1' Primary Front-End MariaDB ColumnStore Module is 'pm1' MariaDB ColumnStore Replication Feature is enabled MariaDB ColumnStore Process statuses Process Module Status Last Status Change Process ID ------------------ ------ --------------- ------------------------ ---------- ProcessMonitor pm1 ACTIVE Thu Feb 22 09:46:53 2018 18797 ProcessManager pm1 ACTIVE Thu Feb 22 09:37:17 2018 4403 DBRMControllerNode pm1 ACTIVE Thu Feb 22 09:38:00 2018 5424 ServerMonitor pm1 ACTIVE Thu Feb 22 09:38:02 2018 5458 DBRMWorkerNode pm1 ACTIVE Thu Feb 22 09:38:02 2018 5489 DecomSvr pm1 ACTIVE Thu Feb 22 09:38:06 2018 5614 PrimProc pm1 ACTIVE Thu Feb 22 09:38:08 2018 5679 ExeMgr pm1 ACTIVE Thu Feb 22 09:38:15 2018 6569 WriteEngineServer pm1 ACTIVE Thu Feb 22 09:38:20 2018 6705 DDLProc pm1 ACTIVE Thu Feb 22 09:38:24 2018 6836 DMLProc pm1 ACTIVE Thu Feb 22 09:38:28 2018 6921 mysqld pm1 ACTIVE Thu Feb 22 09:38:18 2018 5330 ProcessMonitor pm2 ACTIVE Thu Feb 22 09:37:53 2018 5497 ProcessManager pm2 HOT_STANDBY Thu Feb 22 09:37:57 2018 5564 DBRMControllerNode pm2 COLD_STANDBY Thu Feb 22 09:38:03 2018 ServerMonitor pm2 ACTIVE Thu Feb 22 09:38:06 2018 5869 DBRMWorkerNode pm2 ACTIVE Thu Feb 22 09:38:07 2018 5882 DecomSvr pm2 ACTIVE Thu Feb 22 09:38:10 2018 5896 PrimProc pm2 ACTIVE Thu Feb 22 09:38:13 2018 5928 ExeMgr pm2 ACTIVE Thu Feb 22 09:38:17 2018 5952 WriteEngineServer pm2 ACTIVE Thu Feb 22 09:38:21 2018 5969 DDLProc pm2 COLD_STANDBY Thu Feb 22 09:38:22 2018 DMLProc pm2 COLD_STANDBY Thu Feb 22 09:38:22 2018 mysqld pm2 ACTIVE Thu Feb 22 09:38:10 2018 5812 ProcessMonitor pm3 ACTIVE Thu Feb 22 09:46:49 2018 4770 ProcessManager pm3 MAN_OFFLINE Thu Feb 22 09:45:44 2018 DBRMControllerNode pm3 MAN_OFFLINE Thu Feb 22 09:45:44 2018 ServerMonitor pm3 MAN_OFFLINE Thu Feb 22 09:45:44 2018 DBRMWorkerNode pm3 MAN_OFFLINE Thu Feb 22 09:45:44 2018 DecomSvr pm3 MAN_OFFLINE Thu Feb 22 09:45:44 2018 PrimProc pm3 MAN_OFFLINE Thu Feb 22 09:45:44 2018 ExeMgr pm3 MAN_OFFLINE Thu Feb 22 09:45:44 2018 WriteEngineServer pm3 MAN_OFFLINE Thu Feb 22 09:45:44 2018 DDLProc pm3 MAN_OFFLINE Thu Feb 22 09:45:44 2018 DMLProc pm3 MAN_OFFLINE Thu Feb 22 09:45:44 2018 mysqld pm3 MAN_OFFLINE Thu Feb 22 09:45:44 2018 Active Alarm Counts: Critical = 0, Major = 0, Minor = 0, Warning = 0, Info = 0

18. 재기동 후 확인하면 pm 3개 모두 정상작동중인 것을 확인

mcsadmin> stopsystem stopsystem Thu Feb 22 09:47:48 2018 This command stops the processing of applications on all Modules within the MariaDB ColumnStore System Checking for active transactions Do you want to proceed: (y or n) [n]: y System being stopped now... Successful stop of System mcsadmin> startsystem startsystem Thu Feb 22 09:48:17 2018 System being started, please wait... Successful start of System mcsadmin> getsystemi getsysteminfo Thu Feb 22 09:50:06 2018 System columnstore-1 System and Module statuses Component Status Last Status Change ------------ -------------------------- ------------------------ System ACTIVE Thu Feb 22 09:49:00 2018 Module pm1 ACTIVE Thu Feb 22 09:48:55 2018 Module pm2 ACTIVE Thu Feb 22 09:48:50 2018 Module pm3 ACTIVE Thu Feb 22 09:48:51 2018 Active Parent OAM Performance Module is 'pm1' Primary Front-End MariaDB ColumnStore Module is 'pm1' MariaDB ColumnStore Replication Feature is enabled MariaDB ColumnStore Process statuses Process Module Status Last Status Change Process ID ------------------ ------ --------------- ------------------------ ---------- ProcessMonitor pm1 ACTIVE Thu Feb 22 09:46:53 2018 18797 ProcessManager pm1 ACTIVE Thu Feb 22 09:37:17 2018 4403 DBRMControllerNode pm1 ACTIVE Thu Feb 22 09:48:26 2018 21889 ServerMonitor pm1 ACTIVE Thu Feb 22 09:48:28 2018 21915 DBRMWorkerNode pm1 ACTIVE Thu Feb 22 09:48:28 2018 21972 DecomSvr pm1 ACTIVE Thu Feb 22 09:48:32 2018 22122 PrimProc pm1 ACTIVE Thu Feb 22 09:48:34 2018 22260 ExeMgr pm1 ACTIVE Thu Feb 22 09:48:44 2018 23902 WriteEngineServer pm1 ACTIVE Thu Feb 22 09:48:48 2018 24066 DDLProc pm1 ACTIVE Thu Feb 22 09:48:52 2018 24245 DMLProc pm1 ACTIVE Thu Feb 22 09:48:57 2018 24413 mysqld pm1 ACTIVE Thu Feb 22 09:48:52 2018 21755 ProcessMonitor pm2 ACTIVE Thu Feb 22 09:37:53 2018 5497 ProcessManager pm2 HOT_STANDBY Thu Feb 22 09:48:20 2018 8134 DBRMControllerNode pm2 COLD_STANDBY Thu Feb 22 09:48:42 2018 ServerMonitor pm2 ACTIVE Thu Feb 22 09:48:29 2018 8445 DBRMWorkerNode pm2 ACTIVE Thu Feb 22 09:48:31 2018 8471 DecomSvr pm2 ACTIVE Thu Feb 22 09:48:34 2018 8485 PrimProc pm2 ACTIVE Thu Feb 22 09:48:37 2018 8517 ExeMgr pm2 ACTIVE Thu Feb 22 09:48:45 2018 9168 WriteEngineServer pm2 ACTIVE Thu Feb 22 09:48:49 2018 9215 DDLProc pm2 COLD_STANDBY Thu Feb 22 09:48:50 2018 DMLProc pm2 COLD_STANDBY Thu Feb 22 09:48:50 2018 mysqld pm2 ACTIVE Thu Feb 22 09:48:43 2018 8382 ProcessMonitor pm3 ACTIVE Thu Feb 22 09:46:49 2018 4770 ProcessManager pm3 COLD_STANDBY Thu Feb 22 09:48:31 2018 DBRMControllerNode pm3 COLD_STANDBY Thu Feb 22 09:48:31 2018 ServerMonitor pm3 ACTIVE Thu Feb 22 09:48:34 2018 5469 DBRMWorkerNode pm3 ACTIVE Thu Feb 22 09:48:35 2018 5495 DecomSvr pm3 ACTIVE Thu Feb 22 09:48:39 2018 5507 PrimProc pm3 ACTIVE Thu Feb 22 09:48:41 2018 5539 ExeMgr pm3 ACTIVE Thu Feb 22 09:48:45 2018 5563 WriteEngineServer pm3 ACTIVE Thu Feb 22 09:48:49 2018 5582 DDLProc pm3 COLD_STANDBY Thu Feb 22 09:48:51 2018 DMLProc pm3 COLD_STANDBY Thu Feb 22 09:48:51 2018 mysqld pm3 ACTIVE Thu Feb 22 09:48:39 2018 5407 Active Alarm Counts: Critical = 0, Major = 0, Minor = 0, Warning = 0, Info = 0 mcsadmin> getsystemn getsystemnetworkconfig Thu Feb 22 10:00:01 2018 System Network Configuration Module Name Module Description NIC ID Host Name IP Address ----------- ------------------------- ------ --------- --------------- pm1 Performance Module #1 1 server2 192.168.0.197 pm2 Performance Module #2 1 server1 192.168.0.195 pm3 Performance Module #3 1 server3 192.168.0.199

19. data directories

um & pm1(server2) 는 /usr/local/mariadb/columnstore/data1
[root@server2 columnstore]# ll data1
합계 8
drwxr-xr-x 3 root root 4096 2018-02-22 09:38 000.dir
-rw-r--r-- 1 root root    0 2018-02-22 09:48 OAMdbrootCheck
drwxr-xr-t 3 root root 4096 2018-02-22 09:13 systemFiles
[root@server2 columnstore]# ll data2
합계 0
[root@server2 columnstore]# ll data3
합계 0

pm2(server1) 는 /usr/local/mariadb/columnstore/data2
[root@server1 columnstore]# ll data1
합계 4
drwxr-xr-t 3 root root 4096 2018-02-22 09:13 systemFiles
[root@server1 columnstore]# ll data2
합계 0
-rw-r--r-- 1 root root 0 2018-02-22 09:48 OAMdbrootCheck
[root@server1 columnstore]# ll data3
합계 0



pm3(server3) 는 /usr/local/mariadb/columnstore/data3
[root@server3 columnstore]# ll data1
합계 4
drwxr-xr-t 3 root root 4096 2018-02-22 09:13 systemFiles
[root@server3 columnstore]# ll data2
합계 0
[root@server3 columnstore]# ll data3
합계 0
-rw-r--r-- 1 root root 0 2018-02-22 09:48 OAMdbrootCheck
[root@server3 columnstore]# 

 

++  추가 2018.02.22

binary'도 됨
post-install 스크립트가 $user.$user 로 잡혀있어서   user와 group이 다르면 에러뱉음. 수정해줘야됨.

sed -i 's/\$user.\$user/\$user.\$group/g' /download/maria/mariadb/columnstore/bin/post-install

수정해서 다시 tar로 말고 그 파일을 /home/user 디렉토리에 옮겨놔야   
addmodule 시 scp로 그 파일들 넘겨받아서 진행함.


그리고
um1 + pm1 설치 후 addmodule pm2는 에러나는데
um1 + pm1 + pm2 설치 후 addmodule pm3는 에러가 안남

 

++  추가 2018.02.23

1. pm 설치서버는 post-install 후  export 까지 진행 --> library 못찾는현상 해결
2. 모듈 추가 후 추가한 pm이 getsysteminfo 에서 active 상태 안될경우.  
    일단 pm 노드 종료

maria@server2 ]$ /sw/maria/mariadb/columnstore/bin/columnstore stop

     그리고 mcsadmin 에서 shutdown 후 start

mcsadmin> shutdownSystem y mcsadmin> startSystem

  pm node에서 종료된 pm node 실행여부 확인

watch -n 2 '/sw/maria/mariadb/columnstore/bin/columnstore status'

  로 보면 is not running --> is running 으로 바뀜

--> pm이 addmodule시 um에 잘 안붙는듯. pm node stop 후 mcsadmin shutdown and start







모듈 추가 후 데이터 재분산 처리를 위한 명령어.


An important point is that the operation works at Partition granularity, so a minimal data set is 64M rows in a a table for this to run.

소량데이터는 분산되지않음

mcsadmin> redistributeData start

 

https://mariadb.com/kb/en/columnstore-redistribute-data/

ColumnStore Redistribute Data