BoyChai's Blog - nginx https://blog.boychai.xyz/index.php/tag/nginx/ [Nginx]ngx_lua模块 https://blog.boychai.xyz/index.php/archives/71/ 2024-04-25T00:51:00+00:00 概述淘宝开发的ngx_lua模块通过将lua解释器解释器集成Nginx，可以采用lua脚本实现业务逻辑，由于lua的紧凑、快速以及内建协程，所以在保证高并发服务能力的同时极大降低了业务逻辑实现成本。安装方式1(已弃用)lua-nginx-moduleLuaJIT是采用C语言编写的Lua代表的解释器。官网: http://luajit.org在官网找到对应下载地址: https://github.com/LuaJIT/LuaJIT/tags[root@work env]# wget https://github.com/LuaJIT/LuaJIT/archive/refs/tags/v2.0.5.tar.gz [root@work env]# tar xvf v2.0.5.tar.gz [root@work env]# cd LuaJIT-2.0.5/ [root@work LuaJIT-2.0.5]# make && make install make[1]: Leaving directory '/opt/env/LuaJIT-2.0.5/src' ==== Successfully built LuaJIT 2.0.5 ==== ==== Installing LuaJIT 2.0.5 to /usr/local ==== mkdir -p /usr/local/bin /usr/local/lib /usr/local/include/luajit-2.0 /usr/local/share/man/man1 /usr/local/lib/pkgconfig /usr/local/share/luajit-2.0.5/jit /usr/local/share/lua/5.1 /usr/local/lib/lua/5.1 cd src && install -m 0755 luajit /usr/local/bin/luajit-2.0.5 cd src && test -f libluajit.a && install -m 0644 libluajit.a /usr/local/lib/libluajit-5.1.a || : rm -f /usr/local/bin/luajit /usr/local/lib/libluajit-5.1.so.2.0.5 /usr/local/lib/libluajit-5.1.so /usr/local/lib/libluajit-5.1.so.2 cd src && test -f libluajit.so && \ install -m 0755 libluajit.so /usr/local/lib/libluajit-5.1.so.2.0.5 && \ ldconfig -n /usr/local/lib && \ ln -sf libluajit-5.1.so.2.0.5 /usr/local/lib/libluajit-5.1.so && \ ln -sf libluajit-5.1.so.2.0.5 /usr/local/lib/libluajit-5.1.so.2 || : cd etc && install -m 0644 luajit.1 /usr/local/share/man/man1 cd etc && sed -e "s|^prefix=.*|prefix=/usr/local|" -e "s|^multilib=.*|multilib=lib|" luajit.pc > luajit.pc.tmp && \ install -m 0644 luajit.pc.tmp /usr/local/lib/pkgconfig/luajit.pc && \ rm -f luajit.pc.tmp cd src && install -m 0644 lua.h lualib.h lauxlib.h luaconf.h lua.hpp luajit.h /usr/local/include/luajit-2.0 cd src/jit && install -m 0644 bc.lua v.lua dump.lua dis_x86.lua dis_x64.lua dis_arm.lua dis_ppc.lua dis_mips.lua dis_mipsel.lua bcsave.lua vmdef.lua /usr/local/share/luajit-2.0.5/jit ln -sf luajit-2.0.5 /usr/local/bin/luajit ==== Successfully installed LuaJIT 2.0.5 to /usr/local ====lua-nginx-modulenginx第三方模块lua-nginx-module官网: https://github.com/openresty/lua-nginx-module[root@work env]# wget https://github.com/openresty/lua-nginx-module/archive/refs/tags/v0.10.26.tar.gz [root@work env]# tar xvf v0.10.26.tar.gz [root@work env]# ln -s lua-nginx-module-0.10.26 lua-nginx-module环境变量设置[root@work ~]# tail -n2 /etc/profile export LUAJIT_LIB=/usr/local/lib export LUAJIT_INC=/usr/local/include/luajit-2.0 [root@work ~]# source /etc/profile扩展nginx模块打开nginx编译安装的位置 进行重新编译安装[root@work nginx-1.24.0]# ./configure --prefix=/usr/local/nginx --sbin-path=/usr/local/nginx/sbin/nginx --conf-path=/usr/local/nginx/conf/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx/nginx.pid --lock-path=/var/lock/nginx.lock --user=nginx --group=nginx --with-http_ssl_module --with-http_stub_status_module --with-http_gzip_static_module --http-client-body-temp-path=/var/tmp/nginx/client/ --http-proxy-temp-path=/var/tmp/nginx/proxy/ --http-fastcgi-temp-path=/var/tmp/nginx/fcgi/ --http-uwsgi-temp-path=/var/tmp/nginx/uwsgi --http-scgi-temp-path=/var/tmp/nginx/scgi --with-pcre --add-module=/opt/package/nginx/lua-nginx-module [root@work nginx-1.24.0]# make && make install扩展的重点是--with-pcre --add-module=/opt/package/nginx/lua-nginx-module这里就相当于重新安装了，之前安装的模块还需要再这里再添加一遍错误libluajit-5.1.so.2当在扩展号nginx模块后执行nginx相关命令出现以下错误[root@work ~]# nginx -V nginx: error while loading shared libraries: libluajit-5.1.so.2: cannot open shared object file: No such file or directory这个错误表明 Nginx 在启动时无法找到名为 libluajit-5.1.so.2 的共享库文件。这很可能是由于 Nginx 模块依赖 LuaJIT 库，但系统中缺少了该库所致。解决办法如下[root@work ~]# ln -s /usr/local/lib/libluajit-5.1.so.2 /lib64/liblua-5.1.so.2reason: module 'resty.core' not found[root@work conf]# nginx nginx: [alert] detected a LuaJIT version which is not OpenResty's; many optimizations will be disabled and performance will be compromised (see https://github.com/openresty/luajit2 for OpenResty's LuaJIT or, even better, consider using the OpenResty releases from https://openresty.org/en/download.html) nginx: [alert] failed to load the 'resty.core' module (https://github.com/openresty/lua-resty-core); ensure you are using an OpenResty release from https://openresty.org/en/download.html (reason: module 'resty.core' not found: no field package.preload['resty.core'] no file './resty/core.lua' no file '/usr/local/share/luajit-2.0.5/resty/core.lua' no file '/usr/local/share/lua/5.1/resty/core.lua' no file '/usr/local/share/lua/5.1/resty/core/init.lua' no file './resty/core.so' no file '/usr/local/lib/lua/5.1/resty/core.so' no file '/usr/local/lib/lua/5.1/loadall.so' no file './resty.so' no file '/usr/local/lib/lua/5.1/resty.so' no file '/usr/local/lib/lua/5.1/loadall.so') in /usr/local/nginx/conf/nginx.conf:117原因似乎是缺少lua-resty-core模块，这里手动编译安装一下项目地址： https://github.com/openresty/lua-resty-core[root@work nginx]# tar xvf v0.1.28.tar.gz tar xvf make install安装方式2概述直接使用OpenRestry，它是由淘宝工程师开发的，它是基于Nginx与Lua的高性能Web平台，其内部集成了大量精良的Lua库，第三方模块以及大多数的依赖项，用于方便搭建能够处理高并发、扩展性极高的动态Web应用、Web服务和动态网关。所以本身OpenResty内部就已经集成了Nginx和Lua，我们用起来会更加方便安装参考： https://openresty.org/cn/linux-packages.html配置：/usr/local/openrestry/nginx/conf关于OpenRestryOpenRestry，它是由淘宝工程师开发的，它是基于Nginx与Lua的高性能Web平台，其内部集成了大量精良的Lua库，第三方模块以及大多数的依赖项，用于方便搭建能够处理高并发、扩展性极高的动态Web应用、Web服务和动态网关。所以本身OpenResty内部就已经集成了Nginx和Lua，我们用起来会更加方便。PS：本文只讲ngx_lua的使用，其他的基本和nginx配置无区别。ngx_lua相关指令块使用Lua编写Nginx脚本的基本构建块是指令。指令用于指定何时运行用户Lua代码以及如何使用结果。下图显示了执行指令的顺序。先来解释一下*的作用*：无 ， 即 xxx_by_lua ,指令后面跟的是 lua指令 *:_file，即 xxx_by_lua_file 指令后面跟的是 lua文件 *:_block,即 xxx_by_lua_block 在0.9.17版后替换init_by_lua_fileinit_by_lua*该指令在每次Nginx重新加载配置时执行，可以用来完成一些耗时模块的加载，或者初始化一些全局配置。init_worker_by_lua*该指令用于启动一些定时任务，如心跳检查、定时拉取服务器配置等。set_by_lua*该指令只要用来做变量赋值，这个指令一次只能返回一个值，并将结果赋值给Nginx中指定的变量。rewrite_by_lua*该指令用于执行内部URL重写或者外部重定向，典型的如伪静态化URL重写，本阶段在rewrite处理阶段的最后默认执行。access_by_lua*该指令用于访问控制。例如，如果只允许内网IP访问。content_by_lua*该指令是应用最多的指令，大部分任务是在这个阶段完成的，其他的过程往往为这个阶段准备数据，正式处理基本都在本阶段。header_filter_by_lua*该指令用于设置应答消息的头部信息。body_filter_by_lua*该指令是对响应数据进行过滤，如截断、替换。log_by_lua*该指令用于在log请求处理阶段，用Lua代码处理日志，但并不替换原有log处理。balancer_by_lua*该指令主要的作用是用来实现上游服务器的负载均衡器算法ssl_certificate_by_*该指令作用在Nginx和下游服务开始一个SSL握手操作时将允许本配置项的Lua代码。案例1需求输出内容配置 location /lua { default_type 'text/html'; content_by_lua 'ngx.say("<h1>HELLO,OpenResty</h1>")'; }案例2需求http://xxx/?name=张三&gender=1 Nginx接收到请求后根据gender传入的值，如果是gender传入的是1，则展示张三先生，如果是0则展示张三女士，如果都不是则展示张三。配置 location /getByGender { default_type 'text/html'; set_by_lua $param " -- 获取请求URL上的参数对应的值 local uri_args = ngx.req.get_uri_args() local name = uri_args['name'] local gender = uri_args['gender'] -- 条件判断 if gender 1 先生 0 女士 if gender == '1' then return name..'先生' elseif gender == '0' then return name..'女士' else return name end "; # 解决中文乱码 charset utf-8; # 返回数据 return 200 $param; }ngx.req.get_uri_args()返回的是一个table类型案例3需求动态获取docker容器ip，做代理配置server{ listen 80; server_name code.boychai.xyz; client_max_body_size 4096M; set_by_lua $param ' local name = "gitea" local port = "3000" local command = string.format("echo -n `docker inspect --format=\'{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}\' %s`", name) local handle = io.popen(command) local result = handle:read("*a") handle:close() return "http://"..result..":"..port '; location / { if ( $param = 'http://:3000' ) { return 500 "Error in obtaining site IP"; } proxy_pass $param; proxy_set_header Host $proxy_host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; } } Kubernetes-容器编排引擎(Ingress-nginx) https://blog.boychai.xyz/index.php/archives/29/ 2022-09-06T09:19:00+00:00 Ingress概述Service对集群之外暴露服务的主要方式有两种：NotePort和LoadBalancer，但是这两种方式，都有一定的缺点：NodePort方式的缺点是会占用很多集群机器的端口，那么当集群服务变多的时候，这个缺点就愈发明显LB方式的缺点是每个service需要一个LB，浪费、麻烦，并且需要kubernetes之外设备的支持基于这种现状，kubernetes提供了Ingress资源对象，Ingress只需要一个NodePort或者一个LB就可以满足暴露多个Service的需求。工作机制大致如下图表示：实际上，Ingress相当于一个7层的负载均衡器，是kubernetes对反向代理的一个抽象，它的工作原理类似于Nginx，可以理解成在Ingress里建立诸多映射规则，Ingress Controller通过监听这些配置规则并转化成Nginx的反向代理配置 , 然后对外部提供服务。在这里有两个核心概念：ingress：kubernetes中的一个对象，作用是定义请求如何转发到service的规则ingress controller：具体实现反向代理及负载均衡的程序，对ingress定义的规则进行解析，根据配置的规则来实现请求转发，实现方式有很多，比如Nginx, Contour, Haproxy等等Ingress（以Nginx为例）的工作原理如下：用户编写Ingress规则，说明哪个域名对应kubernetes集群中的哪个ServiceIngress控制器动态感知Ingress服务规则的变化，然后生成一段对应的Nginx反向代理配置Ingress控制器会将生成的Nginx配置写入到一个运行着的Nginx服务中，并动态更新到此为止，其实真正在工作的就是一个Nginx了，内部配置了用户定义的请求转发规则Ingress使用环境配置环境为3个deploy，分别部署3个pod，镜像依次为nginx，apache，tomcat，并对应部署了三个service创建文件Ingress-Env.yaml，内容如下--- apiVersion: apps/v1 kind: Deployment metadata: name: ingress-env-nginx namespace: default spec: replicas: 3 selector: matchLabels: ingress-env: nginx-pod template: metadata: labels: ingress-env: nginx-pod spec: containers: - name: nginx image: docker.io/library/nginx:1.23.1 --- apiVersion: apps/v1 kind: Deployment metadata: name: ingress-env-httpd namespace: default spec: replicas: 3 selector: matchLabels: ingress-env: httpd-pod template: metadata: labels: ingress-env: httpd-pod spec: containers: - name: httpd image: docker.io/library/httpd:2.4.54 --- apiVersion: apps/v1 kind: Deployment metadata: name: ingress-env-tomcat namespace: default spec: replicas: 3 selector: matchLabels: ingress-env: tomcat-pod template: metadata: labels: ingress-env: tomcat-pod spec: containers: - name: tomcat image: docker.io/library/tomcat:8.5-jre10-slim --- apiVersion: v1 kind: Service metadata: name: ingress-env-nginx-svc namespace: default spec: selector: ingress-env: nginx-pod clusterIP: 10.97.2.1 # service的ip地址，如果不写，默认会生成一个 type: ClusterIP ports: - port: 80 # Service端口 targetPort: 80 # pod端口 --- apiVersion: v1 kind: Service metadata: name: ingress-env-httpd-svc namespace: default spec: selector: ingress-env: httpd-pod clusterIP: 10.97.2.2 # service的ip地址，如果不写，默认会生成一个 type: ClusterIP ports: - port: 80 # Service端口 targetPort: 80 # pod端口 --- apiVersion: v1 kind: Service metadata: name: ingress-env-tomcat-svc namespace: default spec: selector: ingress-env: tomcat-pod clusterIP: 10.97.2.3 # service的ip地址，如果不写，默认会生成一个 type: ClusterIP ports: - port: 80 # Service端口 targetPort: 8080 # pod端口 # 创建环境 [root@master yaml]# kubectl create -f Ingress-Env.yaml deployment.apps/ingress-env-nginx created deployment.apps/ingress-env-httpd created deployment.apps/ingress-env-tomcat created service/ingress-env-nginx-svc created service/ingress-env-httpd-svc created service/ingress-env-tomcat-svc created # 查看Pod [root@master yaml]# kubectl get pod -n default NAME READY STATUS RESTARTS AGE ingress-env-httpd-59b9f557c4-gjvhn 1/1 Running 0 15s ingress-env-httpd-59b9f557c4-j6q9b 1/1 Running 0 15s ingress-env-httpd-59b9f557c4-zp9fv 1/1 Running 0 15s ingress-env-nginx-7d899c7648-4r7gx 1/1 Running 0 15s ingress-env-nginx-7d899c7648-6fzq9 1/1 Running 0 15s ingress-env-nginx-7d899c7648-stv77 1/1 Running 0 15s ingress-env-tomcat-679896868f-27zhq 1/1 Running 0 15s ingress-env-tomcat-679896868f-w9gd6 1/1 Running 0 15s ingress-env-tomcat-679896868f-wwwnn 1/1 Running 0 15s # 查看deploy [root@master yaml]# kubectl get deploy -n default NAME READY UP-TO-DATE AVAILABLE AGE ingress-env-httpd 3/3 3 3 31s ingress-env-nginx 3/3 3 3 31s ingress-env-tomcat 3/3 3 3 31s # 查看svc [root@master yaml]# kubectl get svc -n default NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE ingress-env-httpd-svc ClusterIP 10.97.2.2 <none> 80/TCP 44s ingress-env-nginx-svc ClusterIP 10.97.2.1 <none> 80/TCP 44s ingress-env-tomcat-svc ClusterIP 10.97.2.3 <none> 80/TCP 44s # 测试svc # 都有页面返回即可 [root@master yaml]# curl 10.97.2.1 ...... <h1>Welcome to nginx!</h1> ...... [root@master yaml]# curl 10.97.2.2 <html><body><h1>It works!</h1></body></html> [root@master yaml]# curl 10.97.2.3 ...... <h2>If you're seeing this, you've successfully installed Tomcat. Congratulations!</h2> ......Ingress-nginx安装# 下载Ingress的部署资源清单 # 我这里的版本是v1.3.0 [root@master yaml]# curl -o Ingress-Deploy.yaml https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.3.0/deploy/static/provider/cloud/deploy.yaml % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 15490 100 15490 0 0 15694 0 --:--:-- --:--:-- --:--:-- 15678 # 替换镜像 # Ingress-Deploy里面会用到两个镜像 # 一个是ingress-nginx/controller:1.3.0还有一个是ingress-nginx-kube-webhook-certgen:v1.1.1 # 默认都是从k8s镜像仓库下载的,都被墙了,需要把这两个修改为其他的，这里我自己科学上网pull下来放到仓库了修改内容如下 # 注意一共有三个镜像配置，有两个kube-webhook-certgen，三个都要改 image: registry.k8s.io/ingress-nginx/controller:v1.3.0... > image: docker.io/boychai/ingress-nginx-controlle:v1.3.0 image image: registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.1.1... > image: docker.io/boychai/ingress-nginx-kube-webhook-certgen:v1.1.1 # 添加hostNetwork配置 # 在Ingress-Deploy的里面会有段Deployment的配置大概在388行 # 在Deployment.spec.template.spec添加hostNetwork:true ...... 412 spec: 413 hostNetwork: true 414 containers: 415 - args: 416 - /nginx-ingress-controller 417 - --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller ...... # 创建Ingress-Nginx [root@master yaml]# kubectl create -f Ingress-Deploy.yaml namespace/ingress-nginx created serviceaccount/ingress-nginx created serviceaccount/ingress-nginx-admission created role.rbac.authorization.k8s.io/ingress-nginx created role.rbac.authorization.k8s.io/ingress-nginx-admission created clusterrole.rbac.authorization.k8s.io/ingress-nginx created clusterrole.rbac.authorization.k8s.io/ingress-nginx-admission created rolebinding.rbac.authorization.k8s.io/ingress-nginx created rolebinding.rbac.authorization.k8s.io/ingress-nginx-admission created clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx created clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx-admission created configmap/ingress-nginx-controller created service/ingress-nginx-controller created service/ingress-nginx-controller-admission created deployment.apps/ingress-nginx-controller created job.batch/ingress-nginx-admission-create created job.batch/ingress-nginx-admission-patch created ingressclass.networking.k8s.io/nginx created validatingwebhookconfiguration.admissionregistration.k8s.io/ingress-nginx-admission created # 查看pod [root@master yaml]# kubectl get pod -n ingress-nginx NAME READY STATUS RESTARTS AGE ingress-nginx-admission-create-69hcz 0/1 Completed 0 50s ingress-nginx-admission-patch-pwm7c 0/1 Completed 0 49s ingress-nginx-controller-7fc79df64f-tcx85 1/1 Running 0 50s # 查看service [root@master yaml]# kubectl get svc -n ingress-nginx NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE ingress-nginx-controller LoadBalancer 10.104.11.86 <pending> 80:32637/TCP,443:31430/TCP 80s ingress-nginx-controller-admission ClusterIP 10.99.43.141 <none> 443/TCP 80s Ingress-nginx使用ingress可以代理http和https，如果要使用https需要导入证书相关文件到secret，操作如下# 创建tls目录并生成私钥和证书 [root@master yaml]# mkdir tls [root@master tls]# openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 -keyout tls.key -out tls.crt -subj "/C=CN/ST=QD/L=QD/O=nginx/CN=host.com" Generating a RSA private key .....+++++ ................................+++++ writing new private key to 'tls.key' ----- [root@master tls]# ls tls.crt tls.key # 导入到secret [root@master tls]# kubectl create secret tls tls-secret --key tls.key --cert tls.crt secret/tls-secret created # 查看secret [root@master tls]# kubectl get secret NAME TYPE DATA AGE tls-secret kubernetes.io/tls 2 75s创建Ingress-Basic.yaml，内容如下apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: ingress-basic spec: tls: - hosts: - nginx.host.com - apache.host.com - tomcat.host.com secretName: tls.secret # 指定证书配 ingressClassName: nginx rules: - host: "nginx.host.com" http: paths: - pathType: Prefix path: / backend: service: name: ingress-env-nginx-svc port: number: 80 - host: "apache.host.com" http: paths: - pathType: Prefix path: / backend: service: name: ingress-env-httpd-svc port: number: 80 - host: "tomcat.host.com" http: paths: - pathType: Prefix path: / backend: service: name: ingress-env-tomcat-svc port: number: 80# 创建Ingres [root@master yaml]# kubectl create -f Ingress-Basic.yaml ingress.networking.k8s.io/ingress-basic created # 查看Ingress [root@master yaml]# kubectl get Ingress -n default NAME CLASS HOSTS ADDRESS PORTS AGE ingress-basic nginx nginx.host.com,apache.host.com,tomcat.host.com 80, 443 12s # 查看Ingress详情 [root@master yaml]# kubectl describe Ingress ingress-basic -n default ...... TLS: tls.secret terminates apache.host.com,tomcat.host.com Rules: Host Path Backends ---- ---- -------- nginx.host.com / ingress-env-nginx-svc:80 (10.244.52.248:80,10.244.67.124:80,10.244.67.80:80) apache.host.com / ingress-env-httpd-svc:80 (10.244.52.238:80,10.244.67.114:80,10.244.67.75:80) tomcat.host.com / ingress-env-tomcat-svc:80 (10.244.52.229:8080,10.244.67.103:8080,10.244.67.70:8080) ...... # 访问测试 # 当域名[nginx,apache,tomcat].host.com解析到work节点之后去访问即可 # master我没有消除污点也没有配置ingres的容忍所以只能访问work节点 [root@master yaml]# curl -k --tlsv1 https://nginx.host.com ...... <h1>Welcome to nginx!</h1> ...... [root@master yaml]# curl -k --tlsv1 https://apache.host.com <html><body><h1>It works!</h1></body></html> [root@master yaml]# curl -k --tlsv1 https://tomcat.host.com ...... <h2>If you're seeing this, you've successfully installed Tomcat. Congratulations!</h2> ......http代理上面的配置是https的代理，如果要用http的话把tls段全部删掉就可以了