云原生链路观测平台 openobserve + fluent-bit,日志收集

grpc-opentracing
https://github.com/grpc-ecosystem/grpc-opentracing

openobserve + fluent-bit
为啥会选择这个组合一个 rust 写的一个是c写的，性能和内存方面不用担心，比java 那套好太多了

openobserve 文档：https://openobserve.ai/docs/user-guide/users/

fluent-bit 文档: https://docs.fluentbit.io/manual/installation/linux/ubuntu

fluent-bit linux 服务器red hat 8 安装,安装地址
https://docs.fluentbit.io/manual/installation/linux/redhat-centos

[ec2-user@master yum.repos.d]$ cat fluent-bit.repo
[fluent-bit]
name=Fluent Bit
baseurl=https://packages.fluentbit.io/centos/8/$basearch/
gpgcheck=1
gpgkey=https://packages.fluentbit.io/fluentbit.key
repo_gpgcheck=1
enabled=1

[ec2-user@master yum.repos.d]$ pwd
/etc/yum.repos.dsudo find / -name fluent-bit
/opt/fluent-bit/bin/fluent-bit -v

openobserve 和 fluent-bit 都是二进制文件启动很简单，fluent-bit 负责传输，目前给出fluent-bit 配置
配置文件

# fluent-bit -c config_pro.conf@SET MY_DIR=/home/ec2-user/data/docker/services/api/logs
@SET MY_APP=news_api
@SET MY_DIR1=/home/ec2-user/data/docker/services/yst/logs
@SET MY_APP1=news_yst
@SET MY_DIR2=/home/ec2-user/data/docker/services/task/logs
@SET MY_APP2=news_task[SERVICE]Flush        1Daemon       OffLog_Level    infoParsers_File parsers.conf
[INPUT]Name        tailPath        /Users/admin/go/src/goBoss/my_colly/my_github/zlog/logs/*.logTag         logsParser      json_parserDB          ./flb_logs.dbMem_Buf_Limit 5MBSkip_Long_Lines OnRotate_Wait 10Path_Key    filenameExclude_Path /Users/admin/go/src/goBoss/my_colly/my_github/zlog/logs/*[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]*.log
# [FILTER]
#     Name        lua
#     Match       logs
#     Script      process_time.lua
#     Call        extract_time[OUTPUT]Name        httpMatch       *Host        127.0.0.1Port        5080URI         /api/default/test4/_jsontls         OffFormat      json
#     Json_date_key    _timestamp
#     Json_date_format iso8601HTTP_User   admin@example.comHTTP_Passwd 2rlpVuHNxjDkceZdcompress    gzip# [OUTPUT]
#     Name        stdout
#     Match       *
#     Format      json_lines

还可以直接脚本过滤数据，或者添加字段
process_time.lua

function extract_time(tag, timestamp, record)return 1, timestamp, record
--     if record.time then
--         -- Parse the time string into a Unix timestamp
--         local year, month, day, hour, min, sec = record.time:match("(%d+)-(%d+)-(%d+) (%d+):(%d+):(%d+)")
--         if year then
--             local time = os.time({year=year, month=month, day=day, hour=hour, min=min, sec=sec})
--             record.timestamp = os.date("!%Y-%m-%dT%H:%M:%SZ", time)
--         end
--     end
--     return 1, timestamp, record
end

启动命令：
fluent-bit -c config.conf

systemd 启动

[Unit]
Description=Fluent Bit
After=network.target[Service]
Type=simple
WorkingDirectory=/data/fluent_bit
#Environment="MY_DIR=/data/widgets/logs"
#Environment="MY_APP=widgets_app"
ExecStart=/opt/fluent-bit/bin/fluent-bit -c /data/fluent_bit/config_pro.conf
Restart=on-failure
ExecReload=/bin/kill -HUP $MAINPID[Install]
WantedBy=multi-user.target

openobserve 启动更简单，直接设置一个启动或者登录直接可以启动了
例如我docker 启动

version: '3.8'services:otel-collector:image: otel/opentelemetry-collector-contrib:0.109.0command: ["--config=/etc/otel-collector-config.yaml"]volumes:- ./otel-collector-config.yaml:/etc/otel-collector-config.yamlports:- "4317:4317"   # OTLP gRPC receiver- "4318:4318"   # OTLP HTTP receiverdepends_on:- openobserveopenobserve:image: openobserve/openobserve:latestcontainer_name: openobserveports:- "5080:5080"  # OpenObserve Web UI- "9200:9200"  # OpenObserve Ingest API- "5081:5081"  # OpenObserve gRPC APIenvironment:ZO_ROOT_USER_EMAIL: "admin@example.com"  # 设置 root 用户的电子邮件ZO_ROOT_USER_PASSWORD: "123456"    # 设置 root 用户的密码