概述
Tengine是由淘宝网发起的Web服务器项目。它在Nginx的基础上,针对大访问量网站的需求,添加了很多高级功能和特性。Tengine的性能和稳定性已经在大型的网站如淘宝、天猫商城等得到了很好的检验。它的最终目标是打造一个高效、稳定、安全、易用的Web平台。它有很多良好的特性:支持百万级高并发,动态模块加载(DSO),强大的负载均衡能力,会话保持模块,主动健康检查,根据服务器状态自动上线下线,以及动态解析upstream中出现的域名等一系列强大的功能;
Keepalived是一个免费开源的,用C编写的类似于layer3, 4 & 7交换机制软件,具备我们平时说的第3层、第4层和第7层交换机的功能。主要提供loadbalancing(负载均衡)和 high-availability(高可用)功能,负载均衡实现需要依赖Linux的虚拟服务内核模块(ipvs),而高可用是通过VRRP协议实现多台机器之间的故障转移服务。
所以Tengine很适合用来做七层的负载均衡,而用Keepalived来解决单点故障实现高可用;阿里云的七层负载便是用Keepalived+Tengine实现的,这两个会是很好的组合;Keepalived有主备、主主模式;这里使用主备模式:即一台出现故障,VIP实现漂移到另外一台自动接管服务。
部署实践
拓扑图
1、在前端234、233主机上分别安装配置Tengine实现负载均衡。
编译安装 yum install -y gcc gcc-c++ autoconf automake pcre pcre-devel openssl openssl-devel wget http://tengine.taobao.org/download/tengine-2.2.1.tar.gz tar -zxvf tengine-2.2.1.tar.gz && cd tengine-2.2.1.tar.gz ./configure --with-http_sub_module --with-http_stub_status_module --with-http_gzip_static_module make && make install
1.1关于动态模块
- 如果你想要编译官方模块为动态模块,你需要在configure的时候加上类似这样的指令(–with-http_xxx_module),./configure –help可以看到更多的细节.
- 如果只想要安装官方模块为动态模块(不安装Nginx),那么就只需要configure之后,执行 make dso_install命令.
- 动态加载模块的个数限制为128个.
- 如果已经加载的动态模块有修改,那么必须重起Tengine才会生效.
- 只支持HTTP模块
Tengine默认将安装在/usr/local/nginx目录,nginx -m 可以查看模块列表。
1.2、配置Tengine,实现对后端web集群的负载:
编辑 vim /usr/local/nginx/conf/nginx.conf配置文件插入以下参数
http{ upstream master { # simple round-robin server 192.168.1.230:80; server 192.168.1.231:80; session_sticky; #保持会话连接 check interval=3000 rise=2 fall=5 timeout=1000 type=http; #后端健康检查 check_http_send "HEAD / HTTP/1.0\r\n\r\n"; check_http_expect_alive http_2xx http_3xx; } server{ listen 80; server_name localhost; location / { proxy_pass http://master; proxy_set_header Host $host; #匹配请求头对应后其端服务器 proxy_setheader X-Forwarded-For $remote_addr; #获取用户真实IP } location /status { #状态监控 check_status; } } }
1.3、编写Tengine启动脚本 :vim /etc/init.d/tengine
#!/bin/bash # tengine - this script start and stop the tengine daemon # # chkconfig: 2345 55 25 # description: Startup script for tengine # processname: nginx # config: /usr/local/nginx/conf/nginx.conf # pidfile: /usr/local/nginx/logs/nginx.pid # PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin DAEMON=/usr/local/nginx/sbin/nginx CONFIGFILE=/usr/local/nginx/conf/nginx.conf PIDFILE=/usr/local/nginx/logs/nginx.pid SCRIPTNAME=/etc/init.d/tengine LOCKFILE=/var/lock/nginx.lock set -e [ -x "$DAEMON" ] || exit 0 start() { echo "Startting Tengine......" [ -x $DAEMON ] || exit 5 [ -f $CONFIGFILE ] || exit 6 $DAEMON -c $CONFIGFILE || echo -n "Tengine already running!" [ $? -eq 0 ] && touch $LOCKFILE } stop() { echo "Stopping Tengine......" MPID=`ps aux | grep nginx | awk '/master/{print $2}'` if [ "${MPID}X" != "X" ]; then kill -QUIT $MPID [ $? -eq 0 ] && rm -f $LOCKFILE else echo "Tengine server is not running!" fi } reload() { echo "Reloading Tengine......" MPID=`ps aux | grep nginx | awk '/master/{print $2}'` if [ "${MPID}X" != "X" ]; then kill -HUP $MPID else echo "Tengine can't reload!" fi } case "$1" in start) start ;; stop) stop ;; reload) reload ;; restart) stop sleep 1 start ;; *) echo "Usage: $SCRIPTNAME {start|stop|reload|restart}" exit 3 ;; esac exit 0
1.4、保存后退出,并为文件添加可执行权限,启动服务
chmod +x tengine #添加执行权限 chkconfig tengine on #设置开机启动 /etc/init.d/tengine start #启动服务
1.5、查看web服务是否启动
[[email protected] ~]# netstat -ntlp Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 43742/nginx: master
2、分别在后端230、231 web服务器上安装nginx。
可以参考https://www.anonym0x1.com/lnmp/146.html这篇文章。
yum install -y nginx systemctl start nginx && systemctl enable nginx echo "192.168.1.230" >> /usr/share/nginx/html/index.html echo "192.168.1.231" >> /usr/share/nginx/html/index.html
3、在前端234、233主机上上查看后端web服务状态
清除Cookie再次访问:
4、分别在前端234、233主机上安装部署Keepalived服务
yum install -y Keepalived systemctl start Keepalived && systemctl enable Keepalived
4.1、在234/MASTER主服务器上编辑配置文件
[[email protected] ~]# vim /etc/keepalived/keepalived.conf ! Configuration File for keepalived global_defs { notification_email { [email protected] #通知收件人地址,可以写换行写多个 } notification_email_from [email protected] #发件人地址 smtp_server 192.168.200.1 #邮件smtp服务器地址 smtp_connect_timeout 30 #邮件smtp连接超时时间 router_id LVS_DEVEL #运行Keepalived服务器的标识,自定义; } vrrp_script chk_nginx { #定义一个外部脚本 script "/etc/keepalived/chk_nginx.sh" #脚本的路径 interval 1 weight 2 } vrrp_instance VI_1 { #实例名称为VI_1,相同实例的备节点名字要和这个相同 state MASTER #状态为MASTER,备节点状态需要为BACKUP interface ens33 #通信接口为ens33 此参数备节点设置和主节点相同 virtual_router_id 51 #虚拟路由ID;一组集群ID号必须一样 priority 150 #权重,BACKUP不能高于MASTER advert_int 1 #检测时间间隔 authentication { auth_type PASS #认证类型 auth_pass 1111 #认证密码,同一集群密码要一样 } virtual_ipaddress { 192.168.1.90/24 dev ens33 label ens33:2 } #配置的虚拟ip,掩码24,并绑定网卡ens33接口,别名为ens33:2 #此参数备节点设置和主节点设置相同 } track_script { #定义状态跟踪,名称为vrrp_script中定义的 chk_nginx } }
在主服务器上除了可以让keepalived挂掉后自动漂移外,还可以自定义一个检测nginx服务是否存活的脚本,若是nginx无法访问那么自动关闭Keepalived让其自动漂移到备用服务器,在主配置文件的脚本如下:
[[email protected] ~]# vim /etc/keepalived/chk_nginx.sh #!/bin/bash killall -0 nginx if [[ $? -ne 0 ]];then systemctl stop keepalived fi
添加执行权限:chmod +x chk_nginx.sh;
查看Keepalived服务和ip信息:可以看到VIP已经自动配置好了。
4.2、在233/BACKUP 备用服务器配置文件
[[email protected] ~]# vim /etc/keepalived/keepalived.conf ! Configuration File for keepalived global_defs { notification_email { [email protected] } notification_email_from [email protected] smtp_server 192.168.200.1 smtp_connect_timeout 30 router_id LVS_DEVEL } vrrp_instance VI_1 { state MASTER interface ens33 virtual_router_id 51 priority 150 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 192.168.1.90/24 dev ens33 label ens33:2 } }
测试Keepalived+Tengine高可用
1、分别重启MASTER和BACKUP的Keepalived和Tengine服务
systemctl restart keepalived /etc/init.d/tengine restart
2、分别查看MASTER和BACKUP的IP地址
[[email protected] ~]# ip a
ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:61:81:c5 brd ff:ff:ff:ff:ff:ff
inet 192.168.1.234/24 brd 192.168.1.255 scope global ens33
valid_lft forever preferred_lft forever
inet 192.168.1.90/24 scope global secondary ens33:2
valid_lft forever preferred_lft forever
inet6 fe80::20c:29ff:fe61:81c5/64 scope link
valid_lft forever preferred_lft forever
[[email protected] ~]# ip a
ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:fb:c2:92 brd ff:ff:ff:ff:ff:ff
inet 192.168.1.233/24 brd 192.168.1.255 scope global ens33
valid_lft forever preferred_lft forever
inet6 fe80::20c:29ff:fefb:c292/64 scope link
valid_lft forever preferred_lft forever
3、测试访问192.168.1.90,换浏览器或者清除cookie再次访问会变成230
4、测试关闭MASTER服务器上的Keepalived或者nginx出现故障不能正常提供服务
[[email protected] ~]# systemctl stop keepalived [[email protected] ~]# ip a ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:61:81:c5 brd ff:ff:ff:ff:ff:ff inet 192.168.1.234/24 brd 192.168.1.255 scope global ens33 valid_lft forever preferred_lft forever inet6 fe80::20c:29ff:fe61:81c5/64 scope link valid_lft forever preferred_lft forever
5、在BACKUP服务器上查看ip地址并再次测试访问
[[email protected] ~]# ip a ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:fb:c2:92 brd ff:ff:ff:ff:ff:ff inet 192.168.1.233/24 brd 192.168.1.255 scope global ens33 valid_lft forever preferred_lft forever inet 192.168.1.90/24 scope global secondary ens33:2 valid_lft forever preferred_lft forever inet6 fe80::20c:29ff:fefb:c292/64 scope link valid_lft forever preferred_lft forever
由上图可见,依然可以正常访问;Keepalived+Tengine高可用已实现