https://blog.boychai.xyz/index.php/tag/ingress-nginx/ https://blog.boychai.xyz/index.php/archives/62/ 2023-08-25T13:53:00+00:00 引入问题我的K8s环境是宿主机的hyper-v虚拟出来的，如果要映射到外面则还需要再我的宿主机上面再做一层反代，我采用的是nginx，当ingress整好之后，我从我从我腾讯云上复制了一段nginx配置放到了我的宿主机，主要配置如下：location /test/ { proxy_pass http://kubernetes.boychai.xyz/test/; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header REMOTE-HOST $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; }为了防止传入的ip是代理主机的ip我这里设置了Host、X-Real-IP、REMOTE-HOST、X-Forwarded-For。经过测试之后发现使用宿主机配置的代理访问时返回404，在宿主机上直接却没问题。访问问题日志我去查看了宿主机的nginx日志、ingress-nginx-controller日志、应用程序的日志，发现除了宿主机的nginx均没有日志记录，宿主机日志信息如下111.180.204.54 - - [25/Aug/2023:22:12:21 +0800] "GET /test/ HTTP/1.1" 404 548 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36"除了这一条之外其他的日志均无404。解决看到日志之后有点懵，因为我的宿主机是可以直接访问ingress暴露出来的服务的，而且没有报错正常访问，我是用反代之后就报错404，我最开始以为这个404就是我宿主机报的，但是宿主机的404默认页面会返回nginx的版本，如下图而我反代返回的404页面则是这样的一想就是我ingress返回的页面，但是我去查看ingress-nginx-controller的日志并无404的报错，日志查看命令如下，访问时并无产生记录[root@kubernetes ~]# kubectl -n ingress-nginx logs -f ingress-nginx-controller-kc5np我这里去尝试修改宿主机的反代配置，配置如下location /test/ { proxy_pass http://kubernetes.boychai.xyz/test/; }发现这样是可以正常访问程序的，这就奇怪了，难不成还能是因为我设置了这几个header的问题？我挨个注释这些header的配置发现问题出在下面这段配置proxy_set_header Host $host;具体原因也没搞清楚但是取消使用这条配置就好了...原因在Kubernetes的Ingress中,Host 头部用于根据不同的域名(或主机名)将请求路由到不同的服务。每个Ingress规则可以基于请求的 Host 头部将流量路由到不同的后端服务。我宿主机代理的域名和Ingress设置的域名不同，所以导致了这个问题，我外部代理的域名是tools.boychai.xyz而我k8s设置Ingress的域名则是kubernetes.boychai.xyz,当我在宿主机的代理设置了proxy_set_header Host $host;这段配置之后，请求发到Ingress之后,Ingress拿到的路由请求域名则是tools.boychai.xyz，而我又没有设置这个资源则就返回了404。IP传递问题日志能够访问之后发现最终的应用拿不到真是访问的ip，这里通过nginx直接返回X-Forwarded-For头信息来查看问题出在什么位置，宿主机Nginx配置如下location /aaa { default_type text/html; proxy_set_header X-Real-IP $remote_addr; proxy_set_header REMOTE-HOST $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; return 200 "proxy_add_x_forwarded_for:$proxy_add_x_forwarded_for"; } location /test/ { # ingress暴露的地址`http://kubernetes.boychai.xyz/test/` proxy_pass http://kubernetes.boychai.xyz/test/; proxy_set_header X-Real-IP $remote_addr; proxy_set_header REMOTE-HOST $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; }后端Nginx配置如下location / { default_type text/html; return 200 "proxy_add_x_forwarded_for:$proxy_add_x_forwarded_for"; }这样访问反代的/aaa就是访问反代主机的ip，访问反代的/test就会返回访问nginx的后端访问的ip，访问结果如下访问/aaa返回proxy_add_x_forwarded_for:111.180.204.54访问/test返回proxy_add_x_forwarded_for:192.16.1.1, 192.16.1.2解决查看访问返回的信息发现直接访问代理的ip是没问题的，那就是Ingress的锅了，这里的1.1和1.2依次是反代的ip和k8s主机的ip，到ingress这层没有把x_forwarded_for头加进来，这里我去官方翻了翻文档发现了三条和x_forwarded_for有关系的配置，如下data: ... compute-full-forwarded-for: "true" # 这一条可以不加也需要知道 # forwarded-for-header: "X-Forwarded-For" use-forwarded-headers: "true" 给ingress的cm加上这两条配置即可解决问题，最终/test返回的内容如下proxy_add_x_forwarded_for:111.180.204.54, 192.16.1.1, 192.16.1.2原因Ingress默认是没有配置传递真实IP功能的，需要配置，这三条配置和官网文档如下：use-forwarded-headers文档位置: https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#use-forwarded-headers如果为true，则ingress-nginx会将传入的x-forward-*传递到上游，如果是Ingress上层还有一层ingress则需要配置这一条。如果他直接暴露在公网中或者它基于L3的网络负载后门则不需要管，因为它默认就是false。forwarded-for-header文档位置: https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#forwarded-for-header这个用来设置客户端来源的真实IP，默认就是X-Forwarded-For。这里不需要额外配置。compute-full-forwarded-for文档位置: https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#compute-full-forwarded-for如果开启了use-forwarded-headers的话，会发现还是没能获取到客户端的真实IP，原因是当前X-Forwaded-Fox变量是从remote_addr获取的，每次都是拿上一层的代理ip，这段配置的作用是将客户端用户访问所经过的代理ip都追加到X-Forwaded-Fox. https://blog.boychai.xyz/index.php/archives/29/ 2022-09-06T09:19:00+00:00 Ingress概述Service对集群之外暴露服务的主要方式有两种：NotePort和LoadBalancer，但是这两种方式，都有一定的缺点：NodePort方式的缺点是会占用很多集群机器的端口，那么当集群服务变多的时候，这个缺点就愈发明显LB方式的缺点是每个service需要一个LB，浪费、麻烦，并且需要kubernetes之外设备的支持基于这种现状，kubernetes提供了Ingress资源对象，Ingress只需要一个NodePort或者一个LB就可以满足暴露多个Service的需求。工作机制大致如下图表示：实际上，Ingress相当于一个7层的负载均衡器，是kubernetes对反向代理的一个抽象，它的工作原理类似于Nginx，可以理解成在Ingress里建立诸多映射规则，Ingress Controller通过监听这些配置规则并转化成Nginx的反向代理配置 , 然后对外部提供服务。在这里有两个核心概念：ingress：kubernetes中的一个对象，作用是定义请求如何转发到service的规则ingress controller：具体实现反向代理及负载均衡的程序，对ingress定义的规则进行解析，根据配置的规则来实现请求转发，实现方式有很多，比如Nginx, Contour, Haproxy等等Ingress（以Nginx为例）的工作原理如下：用户编写Ingress规则，说明哪个域名对应kubernetes集群中的哪个ServiceIngress控制器动态感知Ingress服务规则的变化，然后生成一段对应的Nginx反向代理配置Ingress控制器会将生成的Nginx配置写入到一个运行着的Nginx服务中，并动态更新到此为止，其实真正在工作的就是一个Nginx了，内部配置了用户定义的请求转发规则Ingress使用环境配置环境为3个deploy，分别部署3个pod，镜像依次为nginx，apache，tomcat，并对应部署了三个service创建文件Ingress-Env.yaml，内容如下--- apiVersion: apps/v1 kind: Deployment metadata: name: ingress-env-nginx namespace: default spec: replicas: 3 selector: matchLabels: ingress-env: nginx-pod template: metadata: labels: ingress-env: nginx-pod spec: containers: - name: nginx image: docker.io/library/nginx:1.23.1 --- apiVersion: apps/v1 kind: Deployment metadata: name: ingress-env-httpd namespace: default spec: replicas: 3 selector: matchLabels: ingress-env: httpd-pod template: metadata: labels: ingress-env: httpd-pod spec: containers: - name: httpd image: docker.io/library/httpd:2.4.54 --- apiVersion: apps/v1 kind: Deployment metadata: name: ingress-env-tomcat namespace: default spec: replicas: 3 selector: matchLabels: ingress-env: tomcat-pod template: metadata: labels: ingress-env: tomcat-pod spec: containers: - name: tomcat image: docker.io/library/tomcat:8.5-jre10-slim --- apiVersion: v1 kind: Service metadata: name: ingress-env-nginx-svc namespace: default spec: selector: ingress-env: nginx-pod clusterIP: 10.97.2.1 # service的ip地址，如果不写，默认会生成一个 type: ClusterIP ports: - port: 80 # Service端口 targetPort: 80 # pod端口 --- apiVersion: v1 kind: Service metadata: name: ingress-env-httpd-svc namespace: default spec: selector: ingress-env: httpd-pod clusterIP: 10.97.2.2 # service的ip地址，如果不写，默认会生成一个 type: ClusterIP ports: - port: 80 # Service端口 targetPort: 80 # pod端口 --- apiVersion: v1 kind: Service metadata: name: ingress-env-tomcat-svc namespace: default spec: selector: ingress-env: tomcat-pod clusterIP: 10.97.2.3 # service的ip地址，如果不写，默认会生成一个 type: ClusterIP ports: - port: 80 # Service端口 targetPort: 8080 # pod端口 # 创建环境 [root@master yaml]# kubectl create -f Ingress-Env.yaml deployment.apps/ingress-env-nginx created deployment.apps/ingress-env-httpd created deployment.apps/ingress-env-tomcat created service/ingress-env-nginx-svc created service/ingress-env-httpd-svc created service/ingress-env-tomcat-svc created # 查看Pod [root@master yaml]# kubectl get pod -n default NAME READY STATUS RESTARTS AGE ingress-env-httpd-59b9f557c4-gjvhn 1/1 Running 0 15s ingress-env-httpd-59b9f557c4-j6q9b 1/1 Running 0 15s ingress-env-httpd-59b9f557c4-zp9fv 1/1 Running 0 15s ingress-env-nginx-7d899c7648-4r7gx 1/1 Running 0 15s ingress-env-nginx-7d899c7648-6fzq9 1/1 Running 0 15s ingress-env-nginx-7d899c7648-stv77 1/1 Running 0 15s ingress-env-tomcat-679896868f-27zhq 1/1 Running 0 15s ingress-env-tomcat-679896868f-w9gd6 1/1 Running 0 15s ingress-env-tomcat-679896868f-wwwnn 1/1 Running 0 15s # 查看deploy [root@master yaml]# kubectl get deploy -n default NAME READY UP-TO-DATE AVAILABLE AGE ingress-env-httpd 3/3 3 3 31s ingress-env-nginx 3/3 3 3 31s ingress-env-tomcat 3/3 3 3 31s # 查看svc [root@master yaml]# kubectl get svc -n default NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE ingress-env-httpd-svc ClusterIP 10.97.2.2 <none> 80/TCP 44s ingress-env-nginx-svc ClusterIP 10.97.2.1 <none> 80/TCP 44s ingress-env-tomcat-svc ClusterIP 10.97.2.3 <none> 80/TCP 44s # 测试svc # 都有页面返回即可 [root@master yaml]# curl 10.97.2.1 ...... <h1>Welcome to nginx!</h1> ...... [root@master yaml]# curl 10.97.2.2 <html><body><h1>It works!</h1></body></html> [root@master yaml]# curl 10.97.2.3 ...... <h2>If you're seeing this, you've successfully installed Tomcat. Congratulations!</h2> ......Ingress-nginx安装# 下载Ingress的部署资源清单 # 我这里的版本是v1.3.0 [root@master yaml]# curl -o Ingress-Deploy.yaml https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.3.0/deploy/static/provider/cloud/deploy.yaml % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 15490 100 15490 0 0 15694 0 --:--:-- --:--:-- --:--:-- 15678 # 替换镜像 # Ingress-Deploy里面会用到两个镜像 # 一个是ingress-nginx/controller:1.3.0还有一个是ingress-nginx-kube-webhook-certgen:v1.1.1 # 默认都是从k8s镜像仓库下载的,都被墙了,需要把这两个修改为其他的，这里我自己科学上网pull下来放到仓库了修改内容如下 # 注意一共有三个镜像配置，有两个kube-webhook-certgen，三个都要改 image: registry.k8s.io/ingress-nginx/controller:v1.3.0... > image: docker.io/boychai/ingress-nginx-controlle:v1.3.0 image image: registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.1.1... > image: docker.io/boychai/ingress-nginx-kube-webhook-certgen:v1.1.1 # 添加hostNetwork配置 # 在Ingress-Deploy的里面会有段Deployment的配置大概在388行 # 在Deployment.spec.template.spec添加hostNetwork:true ...... 412 spec: 413 hostNetwork: true 414 containers: 415 - args: 416 - /nginx-ingress-controller 417 - --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller ...... # 创建Ingress-Nginx [root@master yaml]# kubectl create -f Ingress-Deploy.yaml namespace/ingress-nginx created serviceaccount/ingress-nginx created serviceaccount/ingress-nginx-admission created role.rbac.authorization.k8s.io/ingress-nginx created role.rbac.authorization.k8s.io/ingress-nginx-admission created clusterrole.rbac.authorization.k8s.io/ingress-nginx created clusterrole.rbac.authorization.k8s.io/ingress-nginx-admission created rolebinding.rbac.authorization.k8s.io/ingress-nginx created rolebinding.rbac.authorization.k8s.io/ingress-nginx-admission created clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx created clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx-admission created configmap/ingress-nginx-controller created service/ingress-nginx-controller created service/ingress-nginx-controller-admission created deployment.apps/ingress-nginx-controller created job.batch/ingress-nginx-admission-create created job.batch/ingress-nginx-admission-patch created ingressclass.networking.k8s.io/nginx created validatingwebhookconfiguration.admissionregistration.k8s.io/ingress-nginx-admission created # 查看pod [root@master yaml]# kubectl get pod -n ingress-nginx NAME READY STATUS RESTARTS AGE ingress-nginx-admission-create-69hcz 0/1 Completed 0 50s ingress-nginx-admission-patch-pwm7c 0/1 Completed 0 49s ingress-nginx-controller-7fc79df64f-tcx85 1/1 Running 0 50s # 查看service [root@master yaml]# kubectl get svc -n ingress-nginx NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE ingress-nginx-controller LoadBalancer 10.104.11.86 <pending> 80:32637/TCP,443:31430/TCP 80s ingress-nginx-controller-admission ClusterIP 10.99.43.141 <none> 443/TCP 80s Ingress-nginx使用ingress可以代理http和https，如果要使用https需要导入证书相关文件到secret，操作如下# 创建tls目录并生成私钥和证书 [root@master yaml]# mkdir tls [root@master tls]# openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 -keyout tls.key -out tls.crt -subj "/C=CN/ST=QD/L=QD/O=nginx/CN=host.com" Generating a RSA private key .....+++++ ................................+++++ writing new private key to 'tls.key' ----- [root@master tls]# ls tls.crt tls.key # 导入到secret [root@master tls]# kubectl create secret tls tls-secret --key tls.key --cert tls.crt secret/tls-secret created # 查看secret [root@master tls]# kubectl get secret NAME TYPE DATA AGE tls-secret kubernetes.io/tls 2 75s创建Ingress-Basic.yaml，内容如下apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: ingress-basic spec: tls: - hosts: - nginx.host.com - apache.host.com - tomcat.host.com secretName: tls.secret # 指定证书配 ingressClassName: nginx rules: - host: "nginx.host.com" http: paths: - pathType: Prefix path: / backend: service: name: ingress-env-nginx-svc port: number: 80 - host: "apache.host.com" http: paths: - pathType: Prefix path: / backend: service: name: ingress-env-httpd-svc port: number: 80 - host: "tomcat.host.com" http: paths: - pathType: Prefix path: / backend: service: name: ingress-env-tomcat-svc port: number: 80# 创建Ingres [root@master yaml]# kubectl create -f Ingress-Basic.yaml ingress.networking.k8s.io/ingress-basic created # 查看Ingress [root@master yaml]# kubectl get Ingress -n default NAME CLASS HOSTS ADDRESS PORTS AGE ingress-basic nginx nginx.host.com,apache.host.com,tomcat.host.com 80, 443 12s # 查看Ingress详情 [root@master yaml]# kubectl describe Ingress ingress-basic -n default ...... TLS: tls.secret terminates apache.host.com,tomcat.host.com Rules: Host Path Backends ---- ---- -------- nginx.host.com / ingress-env-nginx-svc:80 (10.244.52.248:80,10.244.67.124:80,10.244.67.80:80) apache.host.com / ingress-env-httpd-svc:80 (10.244.52.238:80,10.244.67.114:80,10.244.67.75:80) tomcat.host.com / ingress-env-tomcat-svc:80 (10.244.52.229:8080,10.244.67.103:8080,10.244.67.70:8080) ...... # 访问测试 # 当域名[nginx,apache,tomcat].host.com解析到work节点之后去访问即可 # master我没有消除污点也没有配置ingres的容忍所以只能访问work节点 [root@master yaml]# curl -k --tlsv1 https://nginx.host.com ...... <h1>Welcome to nginx!</h1> ...... [root@master yaml]# curl -k --tlsv1 https://apache.host.com <html><body><h1>It works!</h1></body></html> [root@master yaml]# curl -k --tlsv1 https://tomcat.host.com ...... <h2>If you're seeing this, you've successfully installed Tomcat. Congratulations!</h2> ......http代理上面的配置是https的代理，如果要用http的话把tls段全部删掉就可以了