Heartbeat, LVS 를 이용한 고가용성 서버 구축하기

LVS : www.google.com ( 66.249.89.99 )

가상 IP : 66.249.89.100 ( device : eth0:0 )

실행스크립트 : /root/bin/ipvsadm_tool


RS1 : 66.249.89.101, RS2 : 66.249.89.102, RS3 : 66.249.89.103

가상 IP : 66.249.89.100 ( device : tunl0 )

변경적용사항

/etc/sysctl.conf

net.ipv4.ip_forward = 1

net.ipv4.conf.lo.arp_ignore = 1
net.ipv4.conf.lo.arp_announce = 2
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2


route add -host 66.249.89.100 dev tunl0

현재 wget 을 사용하여 LVS 로 접근시 weight round robin 방식으로

잘 접속되고 있음. wget 을 사용한 LVS 로의 접근테스트는 다음과 같이 할 수 있음


wget -O - http://66.249.89.100:80/


위 명령어를 여러 번 실행시 RS* 의 서버들이 로테이션 되면서 출력됨을 확인할 수 있음


Setting up Heartbeat 2

how to make active, standby cluser


   1.

      heart 비트 관련 패키지를 아래와 같이 모두 설치함

      yum -y install heartbeat*

      /etc/sysctl.conf 에 다음과 같은 내용을 추가

      net.ipv4.conf.all.forwarding = 1
      net.ipv4.conf.all.arp_ignore = 1
      net.ipv4.conf.all.arp_announce = 2

      추가 후 sysctl -p 명령으로 적용함

   2.

      아래와 같이 각 파일을 셋팅해야 함

      /etc/ha.d/authkeys

      auth 1

      1 sha1 ekftjd


/etc/ha.d/authkeys 셋팅 후 반드시 퍼미션을 600 으로 변경해줘야 함!!!


/etc/ha.d/ha.cf

logfile /var/log/ha-log ( 로그파일 지정 )
logfacility daemon 
keepalive 2
warntime 10
deadtime 30
auto_failback off ( 접속실패시 클러스터로 접속을 시도할 것인가 )
crm on ( 모니터링 툴을 사용할 것인가 )
udpport 694 ( 통신에 사용할 포트 )
bcast eth1 (통신에 사용할 디바이스 )
node www.google.com code.google.com ( node는 uname -n 으로 확인한 각 클러스터의 서버의 정보로 대체함 )


/etc/ha.d/haresources

www.google.com    IPaddr::66.249.89.100   ldirectord::ldirectord.cf

(스크립트::인수, IPaddr 는 가상아이피로 사용할 아이피를 지정, ldirectord 는 direct routing 용 데몬)


/etc/ha.d/ldirectord.cf

checktimeout=2
checkinterval=2
logfile="/var/log/ldirectord.log"
quiescent=yes
autoreload=yes

# Virtual Server for HTTP
virtual=66.249.89.100:80
        real=66.249.89.102:80 gate
        real=66.249.89.103:80 gate
        protocol=tcp
        service=http
        request="test.html"
        receive="connected."
        scheduler=wrr
        checktype=negotiate

위와 같이 다이렉트 라우팅을 지원할 클러스터 노드 밑에 존재하는 리얼서버들의 정보를 입력함


주의할 것은 heartbeat 2 부터는 haresources를 직접 로딩하지 않고 /var/lib/heartbeat/crm 에 있는 cib.xml 파일을 사용함

새로운 cib.xml 을 만드는 법은 ec34 서버의 /root/bin 디렉토리에 간단한 스크립트를 만들어뒀으며 현재는 반자동으로 실행하여야 함

추후 모든 테스트 완료 후 완전하 자동화되는 스크립트를 만들어둘 예정

cib.xml 을 새로 만들어서 업데이트할 때 반드시 admin_epoch 와 epoch 를 +1 설정하여 업데이트하여야 하며

업데이트 명령은 다음과 같음

cibadmin -R -x ./cib.xml.new

 /var/lib/heartbeat/crm/cib.xml 의 내용을 변경하여야 함

primitive 태그 중 ldirectord 에 관한 태그를 찾아서 ldirectord 의 class 를 ocf 로 변경해준다.

그리고  /usr/lib/ocf/resource.d/heartbeat/ldirectord 의 내용이 수정이 필요함.


/usr/lib/ocf/resource.d/heartbeat/ldirectord

#!/bin/sh
#
#   ldirectord OCF RA. Wrapper around /usr/sbin/ldirectord to
#   be OCF RA compliant and therefore to get the possibility
#   to monitor ldirectord by HAv2.
#   Tested on SuSE Linux Enterprise Server 10.
#
#   Should conform to the specification found at
#    http://www.linux-ha.org/OCFResourceAgent
#   and
#   http://www.opencf.org/cgi-bin/viewcvs.cgi/specs/ra/resource-agent-api.txt?rev=HEAD
#
#   ToDo: Add parameter to start several instances of ldirectord
#   with different config files.
#
# Copyright (c) 2007 Andreas Mock (andreas.mock@web.de)
#                    All Rights Reserved.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of version 2 of the GNU General Public License as
# published by the Free Software Foundation.
#
# This program is distributed in the hope that it would be useful, but
# WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
#
# Further, this software is distributed without any warranty that it is
# free of the rightful claim of any third person regarding infringement
# or the like.  Any license provided herein, whether implied or
# otherwise, applies only to this software file.  Patent licenses, if
# any, provided herein do not apply to combinations of this program with
# other software, or any other product whatsoever.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write the Free Software Foundation,
# Inc., 59 Temple Place - Suite 330, Boston MA 02111-1307, USA.
#
#######################################################################
#
#       OCF Parameters
#       OCF_RESKEY_configfile
#       OCF_RESKEY_ldirectord
#
#######################################################################
# Initialization:

HA_VARRUNDIR=${HA_VARRUN}

. ${OCF_ROOT}/resource.d/heartbeat/.ocf-shellfuncs

#######################################################################

LDIRECTORD=/usr/sbin/ldirectord
LDIRCONF=/etc/ha.d/ldirectord.cf

meta_data() {
        cat <<END
<?xml version="1.0"?>
<!DOCTYPE resource-agent SYSTEM "ra-api-1.dtd">
<resource-agent name="Ldirectord" version="0.9">
<version>1.0</version>

<longdesc lang="en">
It's a simple OCF RA wrapper for ldirectord and uses the ldirectord interface
to create the OCF compliant interface. You win monitoring of ldirectord.
Be warned: Asking ldirectord status is an expensive action.
</longdesc>
<shortdesc lang="en">Wrapper OCF Resource Agent for ldirectord</shortdesc>

<parameters>

<parameter name="configfile" unique="1">
<longdesc lang="en">
The full pathname of the ldirectord configuration file.
</longdesc>
<shortdesc lang="en">configuration file path</shortdesc>
<content type="string" default="/etc/ha.d/ldirectord.cf" />
</parameter>

<parameter name="ldirectord">
<longdesc lang="en">
The full pathname of the ldirectord.
</longdesc>
<shortdesc lang="en">ldirectord binary path</shortdesc>
<content type="string" default="/usr/sbin/ldirectord" />
</parameter>

</parameters>

<actions>
<action name="start"   timeout="15" />
<action name="stop"    timeout="15" />
<action name="monitor" depth="0"  timeout="10" interval="20" />
<action name="meta-data"  timeout="10" />
<action name="verify-all"  timeout="10" />
</actions>
</resource-agent>
END
}

#######################################################################

ldir_init() {

  COMMAND=$1

  LDIRCONF=$OCF_RESKEY_configfile
  if [ x"${LDIRCONF}" = "x" ]; then
    LDIRCONF=/etc/ha.d/ldirectord.cf
  fi

  if [ ! -f $LDIRCONF ]; then
    case $COMMAND in
      stop)     ocf_log warn "$LDIRCONF not found. ldirectord considered stopped"
                exit $OCF_SUCCESS;;
      monitior) exit $OCF_NOT_RUNNING;;
      status)   exit $LSB_STATUS_STOPPED;;
      start)    ocf_log warn "$LDIRCONF not found."
                exit $OCF_NOT_RUNNING;;
    esac
  fi

  LDIRECTORD=$OCF_RESKEY_ldirectord
  if [ x"${LDIRECTORD}" = x ]; then
    LDIRECTORD="/usr/sbin/ldirectord"
  fi

  if [ ! -x $LDIRECTORD ]; then
    case $COMMAND in
      stop)     ocf_log warn "$LDIRECTORD not found."
                exit $OCF_SUCCESS;;
      monitor)  exit $OCF_NOT_RUNNING;;
      status)   exit $LSB_STATUS_STOPPED;;
      start)    ocf_log warn "$LDIRECTORD not found."
                exit $OCF_NOT_RUNNING;;
    esac
  fi

}

ldirectord_usage() {
        cat <<END
usage: $0 {start|stop|monitor|validate-all|meta-data}

Expects to have a fully populated OCF RA-compliant environment set.
END
}

ldirectord_exit() {
    exit $1
}

ldirectord_start() {
    $LDIRECTORD $LDIRCONF start
}

ldirectord_stop() {
    $LDIRECTORD $LDIRCONF stop
}

ldirectord_status() {
    OUTPUT=`$LDIRECTORD $LDIRCONF status 2>&1`
    case $? in
        1)  echo $OUTPUT
            return $OCF_ERR_GENERIC
            ;;
        0)  echo running
            return $OCF_SUCCESS
            ;;
        3)  echo stopped
            return $OCF_SUCCESS
            ;;
        *)  echo $OUTPUT
            return $OCF_ERR_GENERIC
            ;;
    esac
}

ldirectord_monitor() {
    OUTPUT=`$LDIRECTORD $LDIRCONF status 2>&1`
    case $? in
        0)  return $OCF_SUCCESS
            ;;
        3)  return $OCF_NOT_RUNNING
            ;;
        *)  echo $OUTPUT
            return $OCF_ERR_GENERIC
            ;;
    esac
}

ldirectord_validate() {
        if [ ! -f $LDIRCONF ]; then
                ocf_log err $LDIRCONF
                ocf_log err "Configuration file $LDIRCONF not found!"
                exit $OCF_ERR_CONFIGURED
        fi

        if [ ! -x $LDIRECTORD ]; then
                ocf_log err "Binary file $LDIRECTORD not found."
                exit $OCF_ERR_CONFIGURED
        fi

#        exit $OC_ERR_UNIMPLEMENTED

}

ldir_init

case $__OCF_ACTION in
meta-data)      meta_data
                exit $OCF_SUCCESS
                ;;
start)  ldirectord_start
                ldirectord_exit $?
                ;;
stop)   ldirectord_stop
                ldirectord_exit $?
                ;;
status) ldirectord_status
                ldirectord_exit $?
                ;;
monitor)    ldirectord_monitor
                ldirectord_exit $?
                ;;
validate-all)   ldirectord_validate
                ldirectord_exit $?
                ;;
usage|help)     ldirectord_usage
                exit $OCF_SUCCESS
                ;;
*)              ldirectord_usage
                exit $OCF_ERR_UNIMPLEMENTED
                ;;
esac


위와 같이 수정/적용하여 재실행하면 정상적으로 실행됨.