现象
gitlab部分操作报错500。通过Rails日志发现以下报错:
报错:
RedisClient::CommandError
ERR unknown command 'HELLO'
{"severity": "ERROR","time": "2024-04-22T02:50:16.906Z","correlation_id": "01HW1VC67EJEGKQ1A1CW9TMVPV","meta.caller_id": "UserSettings::PersonalAccessTokensController#create","meta.remote_ip": "10.1.11.11","meta.feature_category": "system_access","meta.user": "root","meta.user_id": 1,"meta.client_id": "user/1","exception.class": "RedisClient::CommandError","exception.message": "ERR unknown command 'HELLO'","exception.backtrace": ["lib/gitlab/instrumentation/redis_client_middleware.rb:26:in `block in call_pipelined'","lib/gitlab/instrumentation/redis_helper.rb:17:in `instrument_call'","lib/gitlab/instrumentation/redis_client_middleware.rb:25:in `call_pipelined'","lib/gitlab/patch/redis_client.rb:12:in `ensure_connected'","config/initializers/forbid_sidekiq_in_transactions.rb:82:in `block (2 levels) in \u003cmodule:NoEnqueueingFromTransactions\u003e'","app/services/notification_service.rb:95:in `access_token_created'","app/services/personal_access_tokens/create_service.rb:20:in `execute'","app/controllers/user_settings/personal_access_tokens_controller.rb:33:in `create'","app/controllers/application_controller.rb:468:in `set_current_admin'","lib/gitlab/session.rb:11:in `with_session'","app/controllers/application_controller.rb:459:in `set_session_storage'","lib/gitlab/i18n.rb:114:in `with_locale'","lib/gitlab/i18n.rb:120:in `with_user_locale'","app/controllers/application_controller.rb:450:in `set_locale'","app/controllers/application_controller.rb:443:in `set_current_context'","lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call'","lib/gitlab/middleware/memory_report.rb:13:in `call'","lib/gitlab/middleware/speedscope.rb:13:in `call'","lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call'","lib/gitlab/middleware/rails_queue_duration.rb:33:in `call'","lib/gitlab/etag_caching/middleware.rb:21:in `call'","lib/gitlab/metrics/rack_middleware.rb:16:in `block in call'","lib/gitlab/metrics/web_transaction.rb:46:in `run'","lib/gitlab/metrics/rack_middleware.rb:16:in `call'","lib/gitlab/middleware/go.rb:20:in `call'","lib/gitlab/middleware/query_analyzer.rb:11:in `block in call'","lib/gitlab/database/query_analyzer.rb:40:in `within'","lib/gitlab/middleware/query_analyzer.rb:11:in `call'","lib/gitlab/middleware/multipart.rb:173:in `call'","lib/gitlab/middleware/read_only/controller.rb:50:in `call'","lib/gitlab/middleware/read_only.rb:18:in `call'","lib/gitlab/middleware/unauthenticated_session_expiry.rb:18:in `call'","lib/gitlab/middleware/same_site_cookies.rb:27:in `call'","lib/gitlab/middleware/path_traversal_check.rb:35:in `call'","lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'","lib/gitlab/middleware/basic_health_check.rb:25:in `call'","lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'","lib/gitlab/middleware/request_context.rb:15:in `call'","lib/gitlab/middleware/webhook_recursion_detection.rb:15:in `call'","config/initializers/fix_local_cache_middleware.rb:11:in `call'","lib/gitlab/middleware/compressed_json.rb:44:in `call'","lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'","lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'","lib/gitlab/metrics/requests_rack_middleware.rb:79:in `call'","lib/gitlab/middleware/release_env.rb:13:in `call'"],"user.username": "root","tags.program": "web","tags.locale": "en","tags.feature_category": "system_access","tags.correlation_id": "01HW1VC67EJEGKQ1A1CW9TMVPV","extra.storage": "queues"
}
排查
日志报错是Redis执行HELLO命令报错,这个命令适用于指定RESP版本。
- Gitlab官方文档要求16.0以上版本,需要使用Redis 6.x or 7.x
- Redis 6.x 开始支持RESP3
- 连接到Redis,执行Hello 2成功;执行Hello 3报错:ERR unknown command 'HELLO'
结合以上3点,怀疑Gitlab 16.x版本之前使用RESP2,之后使用RESP3。
进入Redis,执行monitor,复现问题查看操作:
再次看到与HELLO 3命令相关。
结论&后续处理
结论(踩坑)
- 部分公有云Redis 6.x版本,不支持RESP3协议。
- Gitlab官方虽然明确说明16.x要使用Redis 6.x or 7.x,但是没有阐述原因。我想其中一个原因就是新的ruby代码对于RESP3协议的依赖。如果官网明确说明,并且附上前置检查步骤会更好。
后续处理
- 可以自己部署6.x版本redis高可用节点。
- 可以选择支持RESP3协议的公有云Redis实例使用。